www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.experimental Timeline

reply Jack Stouffer <jack jackstouffer.com> writes:
Currently, we have two modules in std.experimental with a third 
on the way and potentially a single function as well: 
https://github.com/D-Programming-Language/phobos/pull/2945/files

Yet it's a bit concerning that there's no plan on when any of 
these will be moved into the main namespace.

std.logger has been in std.experimental for eight releases and 
eight months. Robert Schadek believes it's blocked by the non 
existant RCString: 
https://github.com/D-Programming-Language/phobos/pull/1500issuecomment-155457980

Is that actually the case? How long can we wait for something 
that might not happen and would take months of work (if that was 
one person's sole focus, longer if they had other things to do)?

Also, knowing when std.allocator will be moved is important, 
because certain functions should integrate std.allocator as soon 
as it's available and have them usable in the same release. 
People who would contribute those changes need to know when that 
is going to happen in order to get the necessary work done before 
hand.
Dec 31 2015
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/31/15 12:45 PM, Jack Stouffer wrote:

 Currently, we have two modules in std.experimental with a third on the
 way and potentially a single function as well:
 https://github.com/D-Programming-Language/phobos/pull/2945/files

 Yet it's a bit concerning that there's no plan on when any of these will
 be moved into the main namespace.


I'm not concerned. Code can live in experimental as long as it's needed 
to flesh out the API. The idea behind std.experimental is to give us a 
disclaimer that the API may change at any release despite usage.

So the timeline is "when it's ready"


 std.logger has been in std.experimental for eight releases and eight
 months. Robert Schadek believes it's blocked by the non existant
 RCString:
 https://github.com/D-Programming-Language/phobos/pull/1500issuecomment-155457980


I think this is probably something that needs revisiting. I don't 
remember the rationale behind delaying the movement of logger into std.


 Also, knowing when std.allocator will be moved is important, because
 certain functions should integrate std.allocator as soon as it's
 available and have them usable in the same release. People who would
 contribute those changes need to know when that is going to happen in
 order to get the necessary work done before hand.


This is somewhat tricky. New functions can just be put in 
std.experimental. Changed functions are not as easy.

In the case of allocator, it's quite possible we could start including 
parameters to functions that are allocators in std, since the behavior 
of the allocator is implementation, not API. In other words, a function 
that takes an allocator may not have to be in std.experimental, as long 
as you know that the parameter is an allocator and the implementation 
knows how to properly use it (and existing code compiles).

-Steve
Dec 31 2015
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Thursday, 31 December 2015 at 18:26:08 UTC, Steven 
Schveighoffer wrote:

 I'm not concerned. Code can live in experimental as long as 
 it's needed to flesh out the API. The idea behind 
 std.experimental is to give us a disclaimer that the API may 
 change at any release despite usage.

 So the timeline is "when it's ready"


I concerns me because without deadlines things get endlessly 
delayed. Also, keeping things in experimental for too long 
encourages a overly cautious mentality IMO.


 In the case of allocator, it's quite possible we could start 
 including parameters to functions that are allocators in std, 
 since the behavior of the allocator is implementation, not API. 
 In other words, a function that takes an allocator may not have 
 to be in std.experimental, as long as you know that the 
 parameter is an allocator and the implementation knows how to 
 properly use it (and existing code compiles).


I was under the impression that any Phobos code importing from 
experimental was taboo.
Dec 31 2015
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/31/15 1:54 PM, Jack Stouffer wrote:

 On Thursday, 31 December 2015 at 18:26:08 UTC, Steven Schveighoffer wrote:

 I'm not concerned. Code can live in experimental as long as it's
 needed to flesh out the API. The idea behind std.experimental is to
 give us a disclaimer that the API may change at any release despite
 usage.

 So the timeline is "when it's ready"


 I concerns me because without deadlines things get endlessly delayed.
 Also, keeping things in experimental for too long encourages a overly
 cautious mentality IMO.


Yes, but the alternative is to have a bad API we must maintain. For 
example, object.destroy used to be called object.clear. It's just now 
coming out of the requirement to be supported as clear, and I can 
finally add a method to AAs that clear all elements (I implemented it 
last year, but put on hold because it was determined that we should wait 
until we could use the "obvious" name).

The idea of having it in std.experimental instead of just on its own 
repository is so that it goes through the same rigors of testing that 
all of phobos does, and so that it gets used more. For all intents and 
purposes, it's "official", just that the API isn't completely baked.

I'd say that if people are using it for a good period of time and nobody 
has any serious objections to it, then we can consider moving it to std.


 In the case of allocator, it's quite possible we could start including
 parameters to functions that are allocators in std, since the behavior
 of the allocator is implementation, not API. In other words, a
 function that takes an allocator may not have to be in
 std.experimental, as long as you know that the parameter is an
 allocator and the implementation knows how to properly use it (and
 existing code compiles).


 I was under the impression that any Phobos code importing from
 experimental was taboo.


I wouldn't say so. If it's an implementation detail, so what? The key is 
to avoid breaking changes with user code, not breaking changes between 
std.experimental and std.

But I'm not the end-all rule maker. Perhaps there is a rule that it's taboo.

-Steve
Dec 31 2015
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, 31 December 2015 at 18:54:05 UTC, Jack Stouffer 
wrote:

 On Thursday, 31 December 2015 at 18:26:08 UTC, Steven 
 Schveighoffer wrote:

 I'm not concerned. Code can live in experimental as long as 
 it's needed to flesh out the API. The idea behind 
 std.experimental is to give us a disclaimer that the API may 
 change at any release despite usage.

 So the timeline is "when it's ready"


 I concerns me because without deadlines things get endlessly 
 delayed. Also, keeping things in experimental for too long 
 encourages a overly cautious mentality IMO.


We have no actual policy on the matter, and until we do, I expect 
that anything that's in std.experimental will indeed sit there 
indefinitely.

Putting stuff in std.experimental to begin with in order to let 
it be banged on a bit by the community at large before being 
finalized in std makes sense, but at what point is it actually 
ready to be put into std? How do we decide that? And do we need 
to vote again, or have any kind of further review in the 
newsgroup? Or do we just wait and see if we get bug reports on it 
and otherwise leave it as-is until we move it over into std due 
to age or because some of the Phobos devs feel like it? AFAIK, 
there's no consensus whatsoever on any of that. AFAIK, we don't 
really even have any proposals on that. We just sort of decided 
that having stuff put into experimental first would be a good 
idea so that we can minimize breaking stuff in std or be forced 
to leave it as-is permanently when a problem is found in the API 
within a release or two of it being put in Phobos.

Probably, we either need someone to propose a process so that we 
can discuss it and possibly adopt it (either by agreeing to it as 
a group or by Walter and Andrei approving it). Certainly, until 
someone pushes the issue, I expect that nothing is going to 
escape std.experimental.


 In the case of allocator, it's quite possible we could start 
 including parameters to functions that are allocators in std, 
 since the behavior of the allocator is implementation, not 
 API. In other words, a function that takes an allocator may 
 not have to be in std.experimental, as long as you know that 
 the parameter is an allocator and the implementation knows how 
 to properly use it (and existing code compiles).


 I was under the impression that any Phobos code importing from 
 experimental was taboo.


I confess that I was assuming that that was the case, but I'm not 
sure that it's ever been discussed. If there's any risk of the 
code from std.experimental being thrown away from Phobos rather 
than moved into std, or if using the stuff from std.experimental 
would affect the API in std, then it would probably be a bad idea 
to use it in std. But if it's definitely going to stay in one 
form or another, and changes to it aren't going to affect the API 
in std, then maybe it would be okay. I don't know. I was assuming 
that std.experimental was an island unto itself, and it hadn't 
occurred to me that anyone would be using it in std before it go 
into std itself, but that doesn't mean that that's necessarily 
how it should be.

- Jonathan M Davis
Jan 01 2016
parent Jack Stouffer <jack jackstouffer.com> writes:
On Friday, 1 January 2016 at 17:26:17 UTC, Jonathan M Davis wrote:

 On Thursday, 31 December 2015 at 18:54:05 UTC, Jack Stouffer 
 wrote:

 I concerns me because without deadlines things get endlessly 
 delayed. Also, keeping things in experimental for too long 
 encourages a overly cautious mentality IMO.


 We have no actual policy on the matter, and until we do, I 
 expect that anything that's in std.experimental will indeed sit 
 there indefinitely.

 Putting stuff in std.experimental to begin with in order to let 
 it be banged on a bit by the community at large before being 
 finalized in std makes sense, but at what point is it actually 
 ready to be put into std? How do we decide that? And do we need 
 to vote again, or have any kind of further review in the 
 newsgroup? Or do we just wait and see if we get bug reports on 
 it and otherwise leave it as-is until we move it over into std 
 due to age or because some of the Phobos devs feel like it? 
 AFAIK, there's no consensus whatsoever on any of that. AFAIK, 
 we don't really even have any proposals on that. We just sort 
 of decided that having stuff put into experimental first would 
 be a good idea so that we can minimize breaking stuff in std or 
 be forced to leave it as-is permanently when a problem is found 
 in the API within a release or two of it being put in Phobos.

 Probably, we either need someone to propose a process so that 
 we can discuss it and possibly adopt it (either by agreeing to 
 it as a group or by Walter and Andrei approving it). Certainly, 
 until someone pushes the issue, I expect that nothing is going 
 to escape std.experimental.


I propose this simple policy:

Three major releases after the module is included, the author has 
the right to ask that it be included in std by opening a new PR 
on Phobos. Voting at this point would be redundant, so if no 
Phobos contributors have any objections, based on open bugs or 
concerns in the API, then it MUST be merged.

Simple and effective.
Jan 01 2016
prev sibling parent reply Dicebot <public dicebot.lv> writes:
This topic is a bit complicated because idea of introducing 
std.experimental came from Andrei but without any specifics. At 
that time I was doing most of Phobos review queue bookkeeping and 
I had to propose some formal process to respect Andrei desire (I 
personally was against the concept).

Thus the original formal process was looking like this in my head:
     - initial voting only decides if proposed module is needed in 
Phobos at all and if author looks competent enough to get it 
finished
     - any time release beta starts std.experimental module author 
can ask review manager to start voting for final inclusion
     - this voting becomes true quality and stability control 
point - reviewers/voters have to decide if API and implementation 
are good enough to get frozen
     - if voting fails, proposal is kept in std.experimental until 
all issues are addressed and process repeats again

Idea of any "automatic" inclusion is very harmful because by 
current standard std.experimental modules are not required to 
have Phobos quality and can be very far from being good enough.

What makes things most complicated is that I don't do review 
bookkeeping anymore and no one else (including Andrei himself) 
have been willing to take control of the process since than.
Jan 01 2016
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Friday, 1 January 2016 at 20:10:22 UTC, Dicebot wrote:

 Thus the original formal process was looking like this in my 
 head:
     - initial voting only decides if proposed module is needed 
 in Phobos at all and if author looks competent enough to get it 
 finished
     - any time release beta starts std.experimental module 
 author can ask review manager to start voting for final 
 inclusion
     - this voting becomes true quality and stability control 
 point - reviewers/voters have to decide if API and 
 implementation are good enough to get frozen
     - if voting fails, proposal is kept in std.experimental 
 until all issues are addressed and process repeats again


Can this be adopted formally?


 Idea of any "automatic" inclusion is very harmful because by 
 current standard std.experimental modules are not required to 
 have Phobos quality and can be very far from being good enough.


My proposal was based on the quality of std.logger and 
std.allocator when they were merged. I wasn't aware that modules 
that were unfinished could be merged into std.experimental.


 What makes things most complicated is that I don't do review 
 bookkeeping anymore and no one else (including Andrei himself) 
 have been willing to take control of the process since than.


This can be supplemented by people asking for review managers 
like they do for initial inclusion into std.experimental, no?
Jan 01 2016
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, 2 January 2016 at 04:27:02 UTC, Jack Stouffer wrote:

 On Friday, 1 January 2016 at 20:10:22 UTC, Dicebot wrote:

 Idea of any "automatic" inclusion is very harmful because by 
 current standard std.experimental modules are not required to 
 have Phobos quality and can be very far from being good enough.


 My proposal was based on the quality of std.logger and 
 std.allocator when they were merged. I wasn't aware that 
 modules that were unfinished could be merged into 
 std.experimental.


IIRC, the review for std.experimental.logger specifically left 
some stuff hanging with the idea that it would be sorted out in 
experimental, but I could be remembering wrong. Regardless, a 
certain level of quality is required for something to get into 
std.experimental, because it has to pass the review and vote, but 
that review and vote is no longer done with the expectation that 
what's there is necessarily set in stone when it gets merged 
(otherwise, there's no point to std.experimental). It should be 
solid, but there is now the potential for redesign on some level 
before it gets into std, whereas before, it would have just gone 
into std, and we basically had to get it right up front.

Having another review for something like std.experimental.logger 
before it gets merged into Phobos makes sense, because it would 
allow us to have an open discussion on anything that needs to be 
changed now that folks have been using it. And if it's ready to 
go (potentially decided by a vote), then we could merge it. But 
without that review, the only stuff that will have changed is 
stuff that gets reported in bugzilla or that a Phobos dev 
specifically decided to change. If we want to actually find out 
how well it's been working for folks, then we need a way to get 
feedback based on actual usage and discuss it. Another review 
doesn't necessarily have to be the way that we do that, but it's 
probably the best mechanism that we have.

- Jonathan M Davis
Jan 01 2016
parent Jack Stouffer <jack jackstouffer.com> writes:
On Saturday, 2 January 2016 at 04:47:55 UTC, Jonathan M Davis 
wrote:

 Having another review for something like 
 std.experimental.logger before it gets merged into Phobos makes 
 sense, because it would allow us to have an open discussion on 
 anything that needs to be changed now that folks have been 
 using it. And if it's ready to go (potentially decided by a 
 vote), then we could merge it. But without that review, the 
 only stuff that will have changed is stuff that gets reported 
 in bugzilla or that a Phobos dev specifically decided to 
 change. If we want to actually find out how well it's been 
 working for folks, then we need a way to get feedback based on 
 actual usage and discuss it. Another review doesn't necessarily 
 have to be the way that we do that, but it's probably the best 
 mechanism that we have.


Yeah, unfortunately there is very little communication with a 
large portion of end users besides comments on reddit and 
bugzilla reports. I don't count people who post in general 
because they are largely a self selected group of people who like 
D enough to participate in the forum, and so we all have a bias. 
After doing a search on GitHub for repos that were already using 
std.logger, I found a lot of them were people who post here. Not 
that good ideas for fixing things can't come out of this forum, 
but IMO there are just not enough people on here for some of the 
more obscure of the bad ideas to come to the surface that only 
show themselves after a lot of stress is put on something, plus 
the bias mentioned earlier.

A good example of this problem is the issue that Sociomantic has 
with Phobos (I'm going off their 2013 and 2014 Dconf talks, so I 
apologize if some info is wrong). People were always complaining 
about Phobos use of the GC, so the Phobos' maintainers solution 
was to change things to stack allocation with lazy ranges and the 
like. But it turns out that that was completely missing the mark. 
Sociomantic uses a lot of fibers so their stacks are rather 
small, and so they still couldn't use Phobos. The problem wasn't 
necessarily the GC allocation, it was a lack of choice in 
allocation, which is a very different problem. Thankfully Andrei 
then wrote std.allocator which could solve the problem, but that 
just goes to show how an entire community of people can miss what 
the root cause of an issue is.

Going back to what I said about a lot of the repos being from 
people who post here, I think this shows an unwillingness from 
the normal user to import from std. experimental. Paraphrasing 
someone else's comment on here (I forget who) "importing from 
std.experimental is asking for trouble because you're essentially 
importing a ticking time bomb". And really, I can't argue with 
this mentality. So this is another block on potential feedback 
from users about bad ideas.

As it see it, there are three things that can be done to mitigate 
these issues:

1. Do the review for final inclusion, as mentioned before

2. Make the inclusion of modules into std.experimental more of an 
event

I knew about PEP 484 way before it was accepted, because I am a 
professional Python programmer and I browse /r/Python and it was 
posted there. The review threads should have the same media blast 
as a D release.

3. Gratuitous use of the deprecation -> undocument -> remove cycle

This is the more effective of the two options, because the first 
idea still has the bias and the lack of interest in experimental 
problems.

Bad ideas are going to slip through the cracks. The best thing to 
do IMO is to get rid of them ASAP as to minimize the damage done. 
When very compelling evidence is given that an idea is bad and 
should be replaced with something else, as soon as the 
replacement is ready, the cycle should start. I believe this is 
the best way to mitigate the damage from lack of eyes on a 
problem.
Jan 02 2016
prev sibling parent reply Dicebot <public dicebot.lv> writes:
On Saturday, 2 January 2016 at 04:27:02 UTC, Jack Stouffer wrote:

 On Friday, 1 January 2016 at 20:10:22 UTC, Dicebot wrote:

 Thus the original formal process was looking like this in my 
 head:
 ...


 Can this be adopted formally?


Depends on if you want to volunteer to manage the process and 
communicate with all interested parties to make it reality - and 
keep maintaining process in future. Or convince someone else to 
do the same.


 Idea of any "automatic" inclusion is very harmful because by 
 current standard std.experimental modules are not required to 
 have Phobos quality and can be very far from being good enough.


 My proposal was based on the quality of std.logger and 
 std.allocator when they were merged. I wasn't aware that 
 modules that were unfinished could be merged into 
 std.experimental.


Haven't found any issues with std.allocator so far but std.logger 
definitely is not Phobos ready per my requirements. I have been 
recently re-evaluating it as possible replacement for old Tango 
logger we use and found that in several places it forces 
unnecessary GC allocations and/or immutability (i.e. 
https://github.com/D-Programming-Language/phobos/blob/master/std/experimenta
/logger/core.d#L742 requires that log message must be formatted into immutable
string before it can be passed to another logger). This is a deal breaker that
can possibly require major API change to fix.


 What makes things most complicated is that I don't do review 
 bookkeeping anymore and no one else (including Andrei himself) 
 have been willing to take control of the process since than.


 This can be supplemented by people asking for review managers 
 like they do for initial inclusion into std.experimental, no?


As far as I understand, there is no semi-persistent review 
manager at all right now. Some people volunteered to handle 
specific voting threads but to make sure new formal process is 
accepted one needs much more than that - provide formal spec, 
ensure approval of language authors, keep poking all involved 
developers to actually follow the process etc. It doesn't take 
that much time but it must be dedicated long-term commitment to 
work.
Jan 03 2016
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/3/16 4:37 PM, Dicebot wrote:

 Haven't found any issues with std.allocator so far but std.logger
 definitely is not Phobos ready per my requirements. I have been recently
 re-evaluating it as possible replacement for old Tango logger we use and
 found that in several places it forces unnecessary GC allocations and/or
 immutability (i.e.
 https://github.com/D-Programming-Language/phobos/blob/master/std/experimental/logger/core.d#L742
 requires that log message must be formatted into immutable string before
 it can be passed to another logger). This is a deal breaker that can
 possibly require major API change to fix.


I guess that's a good example of the benefits of std.experimental. Now 
you can file bug reports and the module could undergo significant 
changes before being moved to std. Have you filed requests yet? -- Andrei
Jan 03 2016
parent Dicebot <public dicebot.lv> writes:
On Sunday, 3 January 2016 at 22:37:34 UTC, Andrei Alexandrescu 
wrote:

 I guess that's a good example of the benefits of 
 std.experimental. Now you can file bug reports and the module 
 could undergo significant changes before being moved to std. 
 Have you filed requests yet? -- Andrei


Not yet - I am hoping to come out with some suitable modification 
proposals before  to make bug report less abstract. It didn't 
look like std.experimental modules are going to be stabilized any 
time soon thus the issue wasn't on my priority list either.
Jan 03 2016
prev sibling parent reply Robert burner Schadek <rburners gmail.com> writes:
On Sunday, 3 January 2016 at 21:37:28 UTC, Dicebot wrote:

 Haven't found any issues with std.allocator so far but 
 std.logger definitely is not Phobos ready per my requirements. 
 I have been recently re-evaluating it as possible replacement 
 for old Tango logger we use and found that in several places it 
 forces unnecessary GC allocations and/or immutability (i.e. 
 https://github.com/D-Programming-Language/phobos/blob/master/std/experimenta
/logger/core.d#L742 requires that log message must be formatted into immutable
string before it can be passed to another logger). This is a deal breaker that
can possibly require major API change to fix.


I do not follow.

1. Tango passes a const(char)[] around [1]. What is the 
difference?
2. Have you looked at the doc and the impl. of FileLogger? They 
both show an easy how to implement a Logger without the GC. With 
that it is also trivial to make a MultiLogger that does not 
allocate. BTW, the default implementation is a direct result of 
the "by default multi-threading safe" requirement brought up 
multiple times during reviews.


[1] 
https://github.com/SiegeLord/Tango-D2/blob/d2port/tango/util/log/Log.d#L1297
Jan 04 2016
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, 4 January 2016 at 09:24:44 UTC, Robert burner Schadek 
wrote:

 On Sunday, 3 January 2016 at 21:37:28 UTC, Dicebot wrote:

 Haven't found any issues with std.allocator so far but 
 std.logger definitely is not Phobos ready per my requirements. 
 I have been recently re-evaluating it as possible replacement 
 for old Tango logger we use and found that in several places 
 it forces unnecessary GC allocations and/or immutability (i.e. 
 https://github.com/D-Programming-Language/phobos/blob/master/std/experimenta
/logger/core.d#L742 requires that log message must be formatted into immutable
string before it can be passed to another logger). This is a deal breaker that
can possibly require major API change to fix.


 I do not follow.

 1. Tango passes a const(char)[] around [1]. What is the 
 difference?
 2. Have you looked at the doc and the impl. of FileLogger? They 
 both show an easy how to implement a Logger without the GC. 
 With that it is also trivial to make a MultiLogger that does 
 not allocate. BTW, the default implementation is a direct 
 result of the "by default multi-threading safe" requirement 
 brought up multiple times during reviews.


 [1] 
 https://github.com/SiegeLord/Tango-D2/blob/d2port/tango/util/log/Log.d#L1297


If I understand correctly (based on previous statements by 
Dicebot), the problem is that at Sociomantic, they reuse buffers 
heavily. So, they basically don't use string much of anywhere and 
instead use some combination of char[] and const(char)[] (so that 
the array elements can be given new values without reallocating), 
and std.experimental.logger - like a lot of typical D code - uses 
string, which means that when they log a message with 
std.experimental.logger, their buffer has to be copied into a 
string, which means that an allocation occurs, which in their 
environment is unacceptable.

So, to work for them, they would need std.experimental.logger to 
accept something like const(char)[] or an arbitrary ranges of 
characters and to not allocate inside of any of its logging 
functions. Any requirement to convert to string (be it by the 
user of the logger or inside of the logger itself) doesn't work 
with their requirements.

I'm not particularly familiar with std.experimental.logger as it 
stands, by I would guess that to fit Sociomantic's requirements, 
it would need to either manage to log without actually 
constructing any array of characters (e.g. by printing each of 
its elements in turn without actually putting them together 
first), or it would need to reuse a buffer every time it created 
the line to log (which would incur the cost of copying the 
characters but wouldn't normally have to allocate). But I don't 
know how well that fits in with the logger being hierarchical via 
classes (which solves other design requirements) or how close the 
current implementation is to that. Certainly, being forced to use 
classes does prevent the logger from using some of the typical 
techniques that we use (particularly with regards to ranges). So, 
the logger definitely presents some challenges that most other 
Phobos code doesn't have to deal with. :| Though obviously, you'd 
be more aware of that than anyone. :)

- Jonathan M Davis
Jan 04 2016
next sibling parent reply Robert burner Schadek <rburners gmail.com> writes:
On Monday, 4 January 2016 at 10:58:09 UTC, Jonathan M Davis wrote:

 If I understand correctly (based on previous statements by 
 Dicebot), the problem is that at Sociomantic, they reuse 
 buffers heavily. So, they basically don't use string much of 
 anywhere and instead use some combination of char[] and 
 const(char)[] (so that the array elements can be given new 
 values without reallocating), and std.experimental.logger - 
 like a lot of typical D code - uses string, which means that 
 when they log a message with std.experimental.logger, their 
 buffer has to be copied into a string, which means that an 
 allocation occurs, which in their environment is unacceptable.


FileLogger is a Logger that can be used after only one allocation.

new FileLogger;

Everything else (logging, ...) does not require a single 
allocation.
If you look at the documentation above [1] you will understand 
why.

Building a MultiLogger (Logger of Loggers) that forwards the 
calls to the log functions to the contained Loggers without 
allocation is also trivial.
Logging into a passed in buffer is also trivial (as long as you 
buffer is big enough for the data to log, but that is design 
problem).


 So, to work for them, they would need std.experimental.logger 
 to accept something like const(char)[] or an arbitrary ranges 
 of characters and to not allocate inside of any of its logging 
 functions. Any requirement to convert to string (be it by the 
 user of the logger or inside of the logger itself) doesn't work 
 with their requirements.


as said above, no Logger has to allocate for logging, and no 
string has to be created.

An allocation schema was selected however, to make multi 
threading work by default.
Creating a user specific Logger that fits the Logger interface 
and that uses a static buffer to write the data to is properly 
less than 20 lines of code.


 I'm not particularly familiar with std.experimental.logger as 
 it stands, by I would guess that to fit Sociomantic's 
 requirements, it would need to either manage to log without 
 actually constructing any array of characters (e.g. by printing 
 each of its elements in turn without actually putting them 
 together first), or it would need to reuse a buffer every time 
 it created the line to log (which would incur the cost of 
 copying the characters but wouldn't normally have to allocate). 
 But I don't know how well that fits in with the logger being 
 hierarchical via classes (which solves other design 
 requirements) or how close the current implementation is to 
 that. Certainly, being forced to use classes does prevent the 
 logger from using some of the typical techniques that we use 
 (particularly with regards to ranges). So, the logger 
 definitely presents some challenges that most other Phobos code 
 doesn't have to deal with. :| Though obviously, you'd be more 
 aware of that than anyone. :)

 - Jonathan M Davis


If you guess of their requirements is correct, 
std.experimental.logger fulfills the requirements.

[1] 
https://github.com/D-Programming-language/phobos/blob/master/std/experimental/logger/core.d#L812
Jan 04 2016
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, 4 January 2016 at 13:48:16 UTC, Robert burner Schadek 
wrote:

 On Monday, 4 January 2016 at 10:58:09 UTC, Jonathan M Davis 
 wrote:

 So, to work for them, they would need std.experimental.logger 
 to accept something like const(char)[] or an arbitrary ranges 
 of characters and to not allocate inside of any of its logging 
 functions. Any requirement to convert to string (be it by the 
 user of the logger or inside of the logger itself) doesn't 
 work with their requirements.


 as said above, no Logger has to allocate for logging, and no 
 string has to be created.

 An allocation schema was selected however, to make multi 
 threading work by default.
 Creating a user specific Logger that fits the Logger interface 
 and that uses a static buffer to write the data to is properly 
 less than 20 lines of code.


Well, looking at your implementation in your other post, you're 
able to make it work by casting to string when it's not actually 
a string. So, you're circumventing the type system to make it 
work. Depending on what the base class does with the resulting 
string, it may work, but there's no guarantee that it will, 
especially if you end up deriving your logger class from a 3rd 
party one. For this code to be safe, msg in LogEntry really needs 
to be const(char)[] and not string.


 I'm not particularly familiar with std.experimental.logger as 
 it stands, by I would guess that to fit Sociomantic's 
 requirements, it would need to either manage to log without 
 actually constructing any array of characters (e.g. by 
 printing each of its elements in turn without actually putting 
 them together first), or it would need to reuse a buffer every 
 time it created the line to log (which would incur the cost of 
 copying the characters but wouldn't normally have to 
 allocate). But I don't know how well that fits in with the 
 logger being hierarchical via classes (which solves other 
 design requirements) or how close the current implementation 
 is to that. Certainly, being forced to use classes does 
 prevent the logger from using some of the typical techniques 
 that we use (particularly with regards to ranges). So, the 
 logger definitely presents some challenges that most other 
 Phobos code doesn't have to deal with. :| Though obviously, 
 you'd be more aware of that than anyone. :)

 - Jonathan M Davis


 If you guess of their requirements is correct, 
 std.experimental.logger fulfills the requirements.


Only by doing a risky cast that happens to work in a reduced 
example but does not work in the general case. Now, the changes 
required to avoid the cast should be minimal and straightforward, 
but as long as a cast to string is required in order to avoid 
allocating a string, the logger doesn't safely support avoiding 
allocations.

My guess is that it didn't even occur to Dicebot to make the 
cast, but I doubt that he would be happy with that solution 
anyway, particularly given the recent PR and related discussion 
on adding message to Throwable (which returns const(char)[]): 
https://github.com/D-Programming-Language/druntime/pull/1445

Regardless, Dicebot will obviously have to talk with you about 
their requirements and why he thinks that std.logger doesn't fit 
them.

- Jonathan M Davis
Jan 04 2016
parent reply Robert burner Schadek <rburners gmail.com> writes:
Yes the cast is ugly, but then I'm writing dynamic content into 
stack arrays.

I actually have a patch in the pipeline for this,. That patch 
require to change the protection for beginLogMsg, logMsgPart, 
finishLogMsg from protected to public and bye bye cast. 
logMsgPart already takes a const(char)[]. I actually thought the 
protection was public to begin with.

Anyway, if people have issues with std.(experimental.)logger, 
please write to the forum, bugzilla, github, or directly to me.
Jan 04 2016
parent reply Dicebot <public dicebot.lv> writes:
Sorry for late reponse, I don't read NG that often now.

On Monday, 4 January 2016 at 17:06:16 UTC, Robert burner Schadek 
wrote:

 I actually have a patch in the pipeline for this,. That patch 
 require to change the protection for beginLogMsg, logMsgPart, 
 finishLogMsg from protected to public and bye bye cast. 
 logMsgPart already takes a const(char)[]. I actually thought 
 the protection was public to begin with.


Very good, this will get rid of one of API problems I was going 
to mention :)


 Yes the cast is ugly, but then I'm writing dynamic content into 
 stack arrays.


It is not just ugly, it is undefined behavior by D specification. 
You are casting to immutable data that will inevitably going to 
change. Most likely it will work on all existing compilers but 
such code won't pass any decent code review.

With protection changes it becomes possible to avoid LogEntry 
though so this becomes sort of fixed. However ..


 BTW, the default implementation is a direct result of the "by 
 default multi-threading safe" requirement brought up multiple 
 times during reviews.


.. this remains a concern. I know that with some tweaks and lot 
of custom overriding I can build logging system that works with 
reusable buffers. However, it would be incompatible with any 3d 
party logger which is defined in terms of `LogEntry` and that 
kills the main benefit of having standard API in the first place.

I'd like (if it is even possible) to have a fast thread-safe 
no-allocation logger with thread-local proxies as a default one 
available because it is both very common demand from a logging 
system and will influence how derived 3d-party loggers will be 
designed.

But that can be a more problematic task because `const(char)[]` 
is not implicitly shared and we don't have any good guidelines 
for sharing mutable data in Phobos :) Right now I have no idea 
what should be done - it is simply a problem that is likely to 
result in at least some API changes and a good example why just 
being in std.experimental for a long period doesn't safeguard 
against finding unexpected API issues (I have only realized the 
issue several weeks ago, despite being the review manager of 
original proposal and reading it in great details).


 Anyway, if people have issues with std.(experimental.)logger, 
 please write to the forum, bugzilla, github, or directly to me.


As I have mentioned in the e-mail, I will do so as soon as I will 
have any half-decent proposal and will be able to help :)
Jan 05 2016
parent Robert burner Schadek <rburners gmail.com> writes:
On Tuesday, 5 January 2016 at 16:34:27 UTC, Dicebot wrote:

 BTW, the default implementation is a direct result of the "by 
 default multi-threading safe" requirement brought up multiple 
 times during reviews.


 .. this remains a concern. I know that with some tweaks and lot 
 of custom overriding I can build logging system that works with 
 reusable buffers. However, it would be incompatible with any 3d 
 party logger which is defined in terms of `LogEntry` and that 
 kills the main benefit of having standard API in the first 
 place.

 I'd like (if it is even possible) to have a fast thread-safe 
 no-allocation logger with thread-local proxies as a default one 
 available because it is both very common demand from a logging 
 system and will influence how derived 3d-party loggers will be 
 designed.


If you could write this up as a callgraph with some (ascii) art 
showing thread boundaries, you would help me a lot understanding 
your requirement.


 But that can be a more problematic task because `const(char)[]` 
 ....
 original proposal and reading it in great details).


 Anyway, if people have issues with std.(experimental.)logger, 
 please write to the forum, bugzilla, github, or directly to me.


 As I have mentioned in the e-mail, I will do so as soon as I 
 will have any half-decent proposal and will be able to help :)


You don't need to do my work, it is nice of you to do that, but 
all I need is a problem description I can work with.
Jan 05 2016
prev sibling parent Robert burner Schadek <rburners gmail.com> writes:
As talk is cheap, here is the code: 
http://dpaste.dzfl.pl/ba4df05339f6

not 20 lines but for 99 lines you get two Loggers and some tests.
Jan 04 2016