www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Vote for std.process

reply "Jesse Phillips" <Jessekphillips+d gmail.com> writes:
It is that time, If you would like to see the proposed 
std.process include into Phobos please vote yes. If one condition 
must be met specify under what condition, otherwise vote no.

std.process by Lars Kyllingstad and Steven Schveighoffer is a 
suggested improvement to the existing std.process and is a major 
change to the API. The original API remains but these will be 
going through deprecation.

In summary of the discussion there was concern of the use of 
Error and Exception. Lars is very interested in getting the 
standard hierarchy http://wiki.dlang.org/DIP33 thus leaving these 
at the safest level to allow for change without breaking code.

Please place any further comments in the official review thread 
leaving only your vote and a short comment (there should be no 
need to reply to anyone).

Docs: 
http://www.kyllingen.net/code/std-process2/phobos-prerelease/std_process.html

Source: 
https://github.com/kyllingstad/phobos/blob/std-process2/std/process.d

Friday April 19 PST will be the last day of voting.
Apr 11 2013
next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:

 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes.


Yes!
Apr 11 2013
prev sibling next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Fri, 12 Apr 2013 06:46:51 +0200
"Jesse Phillips" <Jessekphillips+d gmail.com> wrote:


 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes. If one condition 
 must be met specify under what condition, otherwise vote no.
 


Yes
Apr 11 2013
parent reply Manu <turkeyman gmail.com> writes:
Sorry to derail the topic, but I'd just like to raise a major gripe I've
had with introducing anything into phobos recently.

I see this pattern where something is designed, discussed, and then voted
into phobos. At this time the design looks good on paper, but there is very
little practical experience using the library.
The problem then is, once accepted, people start using it, and at some
point some issues are found, or ideas for improvement are made based on
user experience, but the module can no longer be touched due to the general
phobia of making breaking changes...

Can I suggest that ALL new modules should be added to exp. rather than
std.? Here they will stay for at least 1 year, and while they live in exp,
it is understood that they are still in an experimental introductory phase.
Users who choose to use modules in exp are accepting that the API may
receive changes, and consequently, accepting the responsibility to update
their code (and not complain about it breaking their program), should the
library be amended in its introductory phase.
At some time later when the module has been used in a decent amount of
software, and the API has stabilised, it can then be moved to std. This
move is obviously a breaking change its self, but anyone who has agrees to
import exp modules has already accepted the responsibility to update their
code as such.

Thoughts?


On 12 April 2013 16:12, Nick Sabalausky <SeeWebsiteToContactMe semitwist.com

 wrote:



 On Fri, 12 Apr 2013 06:46:51 +0200
 "Jesse Phillips" <Jessekphillips+d gmail.com> wrote:


 It is that time, If you would like to see the proposed
 std.process include into Phobos please vote yes. If one condition
 must be met specify under what condition, otherwise vote no.


 Yes
Apr 11 2013
next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:

 Sorry to derail the topic, but I'd just like to raise a major 
 gripe I've
 had with introducing anything into phobos recently.

 I see this pattern where something is designed, discussed, and 
 then voted
 into phobos. At this time the design looks good on paper, but 
 there is very
 little practical experience using the library.
 The problem then is, once accepted, people start using it, and 
 at some
 point some issues are found, or ideas for improvement are made 
 based on
 user experience, but the module can no longer be touched due to 
 the general
 phobia of making breaking changes...

 Can I suggest that ALL new modules should be added to exp. 
 rather than
 std.? Here they will stay for at least 1 year, and while they 
 live in exp,
 it is understood that they are still in an experimental 
 introductory phase.
 Users who choose to use modules in exp are accepting that the 
 API may
 receive changes, and consequently, accepting the responsibility 
 to update
 their code (and not complain about it breaking their program), 
 should the
 library be amended in its introductory phase.
 At some time later when the module has been used in a decent 
 amount of
 software, and the API has stabilised, it can then be moved to 
 std. This
 move is obviously a breaking change its self, but anyone who 
 has agrees to
 import exp modules has already accepted the responsibility to 
 update their
 code as such.

 Thoughts?


 On 12 April 2013 16:12, Nick Sabalausky 
 <SeeWebsiteToContactMe semitwist.com

 wrote:



 On Fri, 12 Apr 2013 06:46:51 +0200
 "Jesse Phillips" <Jessekphillips+d gmail.com> wrote:


 It is that time, If you would like to see the proposed
 std.process include into Phobos please vote yes. If one 
 condition
 must be met specify under what condition, otherwise vote no.


 Yes





Fully agree
Apr 11 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-04-12 08:24, Manu wrote:


 Thoughts?


I think it's a good idea.

-- 
/Jacob Carlborg
Apr 11 2013
prev sibling next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:

 Can I suggest that ALL new modules should be added to exp. 
 rather than
 std.?


Java tried this. As a result, they are now stuck with "javax" 
because they realized breaking backwards compatibility wasn't 
worth the benefit of "graduating" packages from "javax" to "java".

If the goal is to include new or newly-overhauled modules in 
official releases, I think simply putting a big red 
"EXPERIMENTAL" warning at the top of the documentation would be 
simpler. Then, the warning can be removed when there is a D 
release which has no breaking changes in said module since its 
last released version.

However, another direction would be to make it easier to test 
such fresh modules. For example, it would be nice to have 
"nightly" builds of D, both for the "master" version, and a "mob" 
version, which includes some work-in-progress developments.
Apr 12 2013
parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Friday, 12 April 2013 at 07:00:25 UTC, Vladimir Panteleev 
wrote:

 On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:

 Can I suggest that ALL new modules should be added to exp. 
 rather than
 std.?


 Java tried this. As a result, they are now stuck with "javax" 
 because they realized breaking backwards compatibility wasn't 
 worth the benefit of "graduating" packages from "javax" to 
 "java".


Actually they also have another issue.

Java supported since the begging something similar to a System
package for dirty tricks, officially it was a Sun only API.

http://www.docjar.com/docs/api/sun/misc/Unsafe.html

So actually all Java implementations provide a compatible API, 
because many do use it.

Younger languages seem not to have any issue breaking backwards 
compatibility for experimental code.

--
Paulo
Apr 12 2013
prev sibling next sibling parent reply "angel" <andrey.gelman gmail.com> writes:
 ...
 Thoughts?


Right suggestion.

As it was already discussed once, D community could follow Linux 
in this regard: each new module enters 'staging' folder. At some 
point in time it goes into 'mainline'.
This could also streamline the process of deprecating the old 
library.
Apr 12 2013
parent "James Wirth" <jwirth1 suddenlink.net> writes:
All progress requires risk. My vote is:
YES
Apr 12 2013
prev sibling parent reply "Jesse Phillips" <Jessekphillips+d gmail.com> writes:
On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:

 I see this pattern where something is designed, discussed, and 
 then voted
 into phobos. At this time the design looks good on paper, but 
 there is very
 little practical experience using the library.
 The problem then is, once accepted, people start using it, and 
 at some
 point some issues are found, or ideas for improvement are made 
 based on
 user experience, but the module can no longer be touched due to 
 the general
 phobia of making breaking changes...


I think this needs to happen prior to the formal review/voting. I 
would say it should be a precursor to starting the official 
review, however this would raise the bar too high for things like 
Jacob's Serialization library; he has a working library, but it 
isn't ready for Phobos and it would be silly to require the 
translation prior to approving it for Phobos.

How we choose to add to the exp module would need some 
consideration.
Apr 12 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-04-12 16:04, Jesse Phillips wrote:


 I think this needs to happen prior to the formal review/voting. I would
 say it should be a precursor to starting the official review, however
 this would raise the bar too high for things like Jacob's Serialization
 library; he has a working library, but it isn't ready for Phobos and it
 would be silly to require the translation prior to approving it for Phobos.


What's left is basically just to put it under the std namespace and 
convert tabs to spaces.

-- 
/Jacob Carlborg
Apr 13 2013
prev sibling next sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Yes.
Apr 11 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 4/12/13, Jesse Phillips <Jessekphillips+d gmail.com> wrote:

 It is that time, If you would like to see the proposed
 std.process include into Phobos please vote yes. If one condition
 must be met specify under what condition, otherwise vote no.


Yes.
Apr 11 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On 12 April 2013 14:46, Jesse Phillips <Jessekphillips+d gmail.com> wrote:


 It is that time, If you would like to see the proposed std.process include
 into Phobos please vote yes. If one condition must be met specify under
 what condition, otherwise vote no.


I didn't think I had much of an opinion on std.process, but I just gave it
a quick once over and noticed, relating to our recent discussions about
flagrant GC usage throughout phobos, that this is riddled with the exact
sorts of issues I've been talking about:

string[string] is used in the main API to receive environment variables;
perhaps kinda convenient, but it's impossible to supply environment
variables with loads of allocations.

toStringz is used liberally; alternatively, alloca() could allocate the
c-string's on the stack and zero terminate them there, passing a pointer to
the stack string to the OS functions.

String concatenation is rampant! Look at this code to parse the env
variables (which are already an AA):

    foreach (var, val; childEnv)
        envz[pos++] = (var~'='~val~'\0').ptr;


And many many more of the same...

This is a lib that spawns a process, I can't imagine why this library
should need to allocate a single byte, but it allocates wildly just to
perform a trivial pass-through to the OS.
This module is no exception either, basically all of phobos is like this,
and I'd like to see careful consideration made to these details in the
future.

I've said before, I sadly have to avoid phobos like the plague. Some
modules (like this one) that provide fundamental functionality - not just
helper functions - can't be avoided. Requirements for those should be extra
strict in my opinion.
Apr 12 2013
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:

 And many many more of the same...


So how many of these will produce a measurable performance 
increase when optimized, considering the relative cost of process 
creation?


 This is a lib that spawns a process, I can't imagine why this 
 library should need to allocate a single byte


So that the library fits in 3500 lines instead of many more, and 
has an API that can be invoked using one line, instead of ten.
Apr 12 2013
parent reply Manu <turkeyman gmail.com> writes:
On 12 April 2013 17:13, Vladimir Panteleev <vladimir thecybershadow.net>wrote:


 On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:


 And many many more of the same...


 So how many of these will produce a measurable performance increase when
 optimized, considering the relative cost of process creation?



I agree that spawning processes is a low-frequency operation, but it's a
principle I'm trying to illustrate here.


 This is a lib that spawns a process, I can't imagine why this library

 should need to allocate a single byte


 So that the library fits in 3500 lines instead of many more, and has an
 API that can be invoked using one line, instead of ten.


You can argue whatever you like. I've said my part, and I wherever it lands
is not my call.
I'm just illustrating the point that phobos, like STL, and to a lesser
extent the CRT, are to continue to be avoided like the plague in some
fields unless these details are considered in future.

Most of the required changes would be self-contained, and not affect the
API at all.
Apr 12 2013
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 07:22:42 UTC, Manu wrote:

 I agree that spawning processes is a low-frequency operation, 
 but it's a
 principle I'm trying to illustrate here.


My point was that it is not that it's low-frequency, it's that 
the OS process creation operation is so expensive, that a few 
memory allocations will not make much of a difference in 
comparison. It's the same as optimizing memory allocations in a 
program which is intrinsically disk- or network-bound.


 You can argue whatever you like. I've said my part, and I 
 wherever it lands is not my call.


Well, that's just not very constructive.

Your complaint in valid in general, but I was pointing out that 
it is not much so when specifically aimed at std.process.
Apr 12 2013
parent reply Manu <turkeyman gmail.com> writes:
On 12 April 2013 17:30, Vladimir Panteleev <vladimir thecybershadow.net>wrote:


 On Friday, 12 April 2013 at 07:22:42 UTC, Manu wrote:


 I agree that spawning processes is a low-frequency operation, but it's a
 principle I'm trying to illustrate here.


 My point was that it is not that it's low-frequency, it's that the OS
 process creation operation is so expensive, that a few memory allocations
 will not make much of a difference in comparison. It's the same as
 optimizing memory allocations in a program which is intrinsically disk- or
 network-bound.



Which OS are we talking about?
What OS runs on an a Nintendo Wii? There's only 24mb of system memory in
that machine, can we afford to allocate it frivolously?

Will I avoid phobos as a policy? Yes.


 You can argue whatever you like. I've said my part, and I wherever it

 lands is not my call.


 Well, that's just not very constructive.

 Your complaint in valid in general, but I was pointing out that it is not
 much so when specifically aimed at std.process.


I don't necessarily disagree with your point (although I don't agree
either). Perhaps in the context of std.process it's not so important... but
it's still an opportunity to set a precedent.
Honestly, any new module could have appeared for approval at this
particular moment and I would have made the same criticisms. Not
necessarily picking on std.process in particular, I'm making a point about
phobos, and what is considered acceptable.
Apr 12 2013
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:

 Which OS are we talking about?
 What OS runs on an a Nintendo Wii? There's only 24mb of system 
 memory in
 that machine, can we afford to allocate it frivolously?


I wasn't aware that writing games for the Wii involves creating 
processes in the thousands.


 I don't necessarily disagree with your point (although I don't 
 agree
 either). Perhaps in the context of std.process it's not so 
 important... but
 it's still an opportunity to set a precedent.
 Honestly, any new module could have appeared for approval at 
 this
 particular moment and I would have made the same criticisms. Not
 necessarily picking on std.process in particular, I'm making a 
 point about
 phobos, and what is considered acceptable.


Don't forget that shifting focus when adjusting goals is always a 
question of tradeoffs. Sacrificing tersity, maintainability, 
usability, safety etc. for the sake of performance would be a 
poor move in the case of std.process.
Apr 12 2013
parent reply Manu <turkeyman gmail.com> writes:
On 12 April 2013 18:18, Vladimir Panteleev <vladimir thecybershadow.net>wrote:


 On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:


 Which OS are we talking about?
 What OS runs on an a Nintendo Wii? There's only 24mb of system memory in
 that machine, can we afford to allocate it frivolously?


 I wasn't aware that writing games for the Wii involves creating processes
 in the thousands.



Again, I'm just suggesting possibilities, and trying to illustrate that
it's a STANDARD library, you can never predict where users will want to use
it.


 I don't necessarily disagree with your point (although I don't agree

 either). Perhaps in the context of std.process it's not so important...
 but
 it's still an opportunity to set a precedent.
 Honestly, any new module could have appeared for approval at this
 particular moment and I would have made the same criticisms. Not
 necessarily picking on std.process in particular, I'm making a point about
 phobos, and what is considered acceptable.


 Don't forget that shifting focus when adjusting goals is always a question
 of tradeoffs. Sacrificing tersity, maintainability, usability, safety etc.
 for the sake of performance would be a poor move in the case of std.process.


Can you perhaps quantify how any of those things would be reduced by
addressing at least the details I highlight?
I'm basically advocating making a habit of using the stack where possible.
It's not exactly hard, or cryptic.
Apr 12 2013
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 08:26:48 UTC, Manu wrote:

 Again, I'm just suggesting possibilities, and trying to 
 illustrate that
 it's a STANDARD library, you can never predict where users will 
 want to use
 it.


So you are talking about an imaginary system where memory 
allocation is expensive but process creation is cheap.

It's impossible to design and optimize software for some 
intangible goals. None of the systems that D targets, or aims to 
target, meet that criteria.


 Can you perhaps quantify how any of those things would be 
 reduced by
 addressing at least the details I highlight?
 I'm basically advocating making a habit of using the stack 
 where possible.
 It's not exactly hard, or cryptic.


Yes. However, I suggest the following instead:

Please rewrite some part of std.process with performance in mind, 
and post it here for review. This way, we can analyze the 
benefits and drawbacks based on a concrete example, instead of 
vapor and hot air.
Apr 12 2013
parent reply "Regan Heath" <regan netmail.co.nz> writes:
On Fri, 12 Apr 2013 09:38:53 +0100, Vladimir Panteleev  
<vladimir thecybershadow.net> wrote:


 On Friday, 12 April 2013 at 08:26:48 UTC, Manu wrote:

 Again, I'm just suggesting possibilities, and trying to illustrate that
 it's a STANDARD library, you can never predict where users will want to  
 use
 it.


 So you are talking about an imaginary system where memory allocation is  
 expensive but process creation is cheap.

 It's impossible to design and optimize software for some intangible  
 goals. None of the systems that D targets, or aims to target, meet that  
 criteria.


I don't understand why you're so "against" these comments.  There is no  
real "argument" here, you both want std.process to be as good as it can be  
and it will clearly be better if it performs faster and allocates less.

Yes, premature optimisation is generally frowned upon in "user" code, but  
library code should ideally be optimised.  Should this percieved lack of  
optimisation prevent it's inclusion now, no because optimisation wont  
change the API (it may extend it however).


 Can you perhaps quantify how any of those things would be reduced by
 addressing at least the details I highlight?
 I'm basically advocating making a habit of using the stack where  
 possible.
 It's not exactly hard, or cryptic.


 Yes. However, I suggest the following instead:

 Please rewrite some part of std.process with performance in mind, and  
 post it here for review. This way, we can analyze the benefits and  
 drawbacks based on a concrete example, instead of vapor and hot air.


Fair point, after all someone has to do the work and it seems logical that  
the person with the most understand of the issue should so it.  Whether he  
has the time however is the Q/problem.

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/
Apr 12 2013
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 08:57:19 UTC, Regan Heath wrote:

 I don't understand why you're so "against" these comments.  
 There is no real "argument" here, you both want std.process to 
 be as good as it can be and it will clearly be better if it 
 performs faster and allocates less.


This is a fallacy.

Performance is not free.

Somebody must do the work.
D contributors do not have an infinity of time and motivation.

Optimizing code often implies making it more complicated.

Straight-forward code is self-documenting.

More complicated code is harder to read and review.

More complicated code more easily hides bugs - including security 
bugs, such as buffer overflows.

Maintaining optimized code requires understanding not only the 
high-level logic, but also the low-level optimization details.

The benefits of optimizing std.process are likely to be so small, 
as to be difficult to measure.

Would you still say that the above costs are worth the 
nearly-intangible gain?
Apr 12 2013
parent reply "Regan Heath" <regan netmail.co.nz> writes:
On Fri, 12 Apr 2013 10:17:54 +0100, Vladimir Panteleev  
<vladimir thecybershadow.net> wrote:


 On Friday, 12 April 2013 at 08:57:19 UTC, Regan Heath wrote:

 I don't understand why you're so "against" these comments.  There is no  
 real "argument" here, you both want std.process to be as good as it can  
 be and it will clearly be better if it performs faster and allocates  
 less.


 This is a fallacy.


Nope, you're attacking a Strawman here.  At most my statement is  
incomplete because it doesn't address the costs (which I left out for  
brevity).


 Performance is not free.


I never claimed it was (thus the Strawman).  I simply claimed std.process  
would be "better" if it performed faster and allocated less - which is  
true.


 Somebody must do the work.
 D contributors do not have an infinity of time and motivation.


I know this as well as anyone, my own free time is in short supply.


 Optimizing code often implies making it more complicated.

 Straight-forward code is self-documenting.

 More complicated code is harder to read and review.

 More complicated code more easily hides bugs - including security bugs,  
 such as buffer overflows.

 Maintaining optimized code requires understanding not only the  
 high-level logic, but also the low-level optimization details.

 The benefits of optimizing std.process are likely to be so small, as to  
 be difficult to measure.


All true.  However, complexity can and should be packaged in such a way as  
to localise, and this localised complex code should be tested to death and  
maintained by someone who understands it.  It should be bracketed by  
sufficient comments and warnings about how/why it does what it does.  The  
resulting packaged complexity, with it's associated cost can be re-used  
many times over for all the benefit it gives.

Stack allocating the environment variables need not be a localised  
improvement but could be a standard library function which can be reused,  
for example.


 Would you still say that the above costs are worth the nearly-intangible  
 gain?


"nearly-intangible" is wrong. Library code is code which is used by  
(hopefully) millions of people, writing millions of applications, running  
for millions of hours, on millions of systems, creating thousands of  
processes, etc..  In short, a little effort now pays massive dividends  
over it's lifetime.

So, yes, IMO the costs shown above are worth the resulting gains.

D is a systems programming language, there is hope that it will penetrate  
a wide range of systems and environments - sure in many cases a little bit  
of memory use or performance loss is unimportant, but for many it will be  
the decisive factor which makes D unusable there.

D is constantly being compared to other languages on the basis of  
performance, so it's clearly an important aspect of D's success.

Library code needs to work first time and work well or people will roll  
their own wasting time, energy and in many cases getting some aspect of it  
just plain wrong.

Again, I'm not suggesting (nor was Manu I think) that this in any way  
blocks the inclusion of std.process.  But I do share his desire for  
performance to sometimes be a little higher up the priority list.

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/
Apr 12 2013
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 10:14:35 UTC, Regan Heath wrote:

 All true.  However, complexity can and should be packaged in 
 such a way as to localise, and this localised complex code 
 should be tested to death and maintained by someone who 
 understands it.  It should be bracketed by sufficient comments 
 and warnings about how/why it does what it does.  The resulting 
 packaged complexity, with it's associated cost can be re-used 
 many times over for all the benefit it gives.

 Stack allocating the environment variables need not be a 
 localised improvement but could be a standard library function 
 which can be reused, for example.


Performant abstractions. Like the much-awaited allocator design?

Either way, they can only diminuate, not remove the costs.


 Would you still say that the above costs are worth the 
 nearly-intangible gain?


 "nearly-intangible" is wrong. Library code is code which is 
 used by (hopefully) millions of people, writing millions of 
 applications, running for millions of hours, on millions of 
 systems, creating thousands of processes, etc..  In short, a 
 little effort now pays massive dividends over it's lifetime.

 So, yes, IMO the costs shown above are worth the resulting 
 gains.

 D is constantly being compared to other languages on the basis 
 of performance, so it's clearly an important aspect of D's 
 success.

 Library code needs to work first time and work well or people 
 will roll their own wasting time, energy and in many cases 
 getting some aspect of it just plain wrong.


But once again, you speak in vague terms.

Consider the following hypothetical decisions and outcomes:

1. std.process is left at is. One user is angry / turned away 
because it performs 0.1% slower than it can be.

2. std.process is rewritten to minimize allocations. Code 
complexity goes up, new improvements are challenging to add; bugs 
pop up and go unfixed for a while because fewer programmers are 
qualified or willing to commit the effort of making correct 
fixes. More people are angry / turned away from D because its 
standard library is buggy.

Of course, the above is an exaggerated illustration. But would 
optimizing all code left and right really make more D users 
happier?

There's also the question of priorities. Would you rather than 
effort is spent on optimizing std.process (and dealing with all 
the fallout from any such optimizations), or working on something 
that is acutely missing and hurting D?


 D is a systems programming language, there is hope that it will 
 penetrate a wide range of systems and environments - sure in 
 many cases a little bit of memory use or performance loss is 
 unimportant, but for many it will be the decisive factor which 
 makes D unusable there.


This is surely an exaggeration.

D does not attempt to please everyone out there who is choosing a 
programming language for their next project. There is no such 
language, nor can one exist. One has to accept that D has a 
number of goals, none of which are absolute, but merely point 
towards a certain, but not overly specific, point in the 
multidimensional matrix of trade-offs. D never was about 
achieving maximum performance in all possible cases.
Apr 12 2013
parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Fri, 12 Apr 2013 13:00:29 +0200
"Vladimir Panteleev" <vladimir thecybershadow.net> wrote:

 
 There's also the question of priorities. Would you rather than 
 effort is spent on optimizing std.process (and dealing with all 
 the fallout from any such optimizations), or working on something 
 that is acutely missing and hurting D?
 


While I'm not necessarily disagreeing with the other points you raise,
Phobos's excess allocations *are* verifiability hurting D. A couple of
the biggest potential user bases for D are videogames and embedded. I
feel confident in saying that no other language has as much potential
in these areas as D has. But these groups have *already*, vocally, been
facing the problem of avoiding/rewriting potentially-large parts of
Phobos, or sticking with C/C++.

Now, you could argue that many of these people are simply being overly
fearful of a merely imagined problem, but even if that's true, the
problem is still real in it's effects: Hindering D adoption and causing
people to (perhaps needlessly) avoid/rewrite parts of Phobos.

Now, I'm not suggesting that we do *or* don't start a big effort to
minimize allocations throughout Phobos, I'm simply objecting to the
implication that unnecessary allocations in Phobos aren't hurting D in
any way.
Apr 12 2013
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 16:42:18 UTC, Nick Sabalausky wrote:

 While I'm not necessarily disagreeing with the other points you 
 raise,
 Phobos's excess allocations *are* verifiability hurting D. A 
 couple of
 the biggest potential user bases for D are videogames and 
 embedded. I
 feel confident in saying that no other language has as much 
 potential
 in these areas as D has. But these groups have *already*, 
 vocally, been
 facing the problem of avoiding/rewriting potentially-large 
 parts of
 Phobos, or sticking with C/C++.

 Now, you could argue that many of these people are simply being 
 overly
 fearful of a merely imagined problem, but even if that's true, 
 the
 problem is still real in it's effects: Hindering D adoption and 
 causing
 people to (perhaps needlessly) avoid/rewrite parts of Phobos.

 Now, I'm not suggesting that we do *or* don't start a big 
 effort to
 minimize allocations throughout Phobos, I'm simply objecting to 
 the
 implication that unnecessary allocations in Phobos aren't 
 hurting D in
 any way.


No, that's not what I was trying to say. I agree with your post 
completely. My point was specifically about std.process in 
particular, since I was misunderstanding why Manu was using 
std.process as an example.
Apr 12 2013
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 06:14:35 -0400, Regan Heath <regan netmail.co.nz>  
wrote:


 On Fri, 12 Apr 2013 10:17:54 +0100, Vladimir Panteleev  
 <vladimir thecybershadow.net> wrote:



 Optimizing code often implies making it more complicated.

 Straight-forward code is self-documenting.

 More complicated code is harder to read and review.

 More complicated code more easily hides bugs - including security bugs,  
 such as buffer overflows.

 Maintaining optimized code requires understanding not only the  
 high-level logic, but also the low-level optimization details.

 The benefits of optimizing std.process are likely to be so small, as to  
 be difficult to measure.


 All true.  However, complexity can and should be packaged in such a way  
 as to localise, and this localised complex code should be tested to  
 death and maintained by someone who understands it.  It should be  
 bracketed by sufficient comments and warnings about how/why it does what  
 it does.  The resulting packaged complexity, with it's associated cost  
 can be re-used many times over for all the benefit it gives.


I would say I agree with both of you.  In this particular instance,  
maintainability trumps performance, since creating processes is  
fundamentally not cheap.  But if someone is willing to step up an give us  
a good alternative, there is no reason not to use it.  It should be  
readable and well documented.

I think in order to achieve good performance, we should be focused on  
custom allocators.  That really is the roadblock to all of this.

-Steve
Apr 12 2013
prev sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Friday, 12 April 2013 at 08:57:19 UTC, Regan Heath wrote:

 ...


It is not something to discuss in this topic. Yes, phobos has 
allocations issues. Yes, it is worth discussing. Does std.process 
do something special in that sense? No, possible problems are 
more related to lack of general configurable allocator.

Please do not fill voting thread with noise if you have no 
_blocking_ objections.
Apr 12 2013
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
12-Apr-2013 12:26, Manu пишет:

 On 12 April 2013 18:18, Vladimir Panteleev <vladimir thecybershadow.net
 <mailto:vladimir thecybershadow.net>> wrote:

     On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:

         Which OS are we talking about?
         What OS runs on an a Nintendo Wii? There's only 24mb of system
         memory in
         that machine, can we afford to allocate it frivolously?


     I wasn't aware that writing games for the Wii involves creating
     processes in the thousands.


 Again, I'm just suggesting possibilities, and trying to illustrate that
 it's a STANDARD library, you can never predict where users will want to
 use it.


Very true.

-- 
Dmitry Olshansky
Apr 12 2013
prev sibling next sibling parent reply "Manipulator" <volcz kth.se> writes:
Please correct me if I'm wrong.
It's not the allocation itself that is the problem it's that it 
will trigger the GC to stop the world at some point.
Apr 12 2013
parent Manu <turkeyman gmail.com> writes:
On 12 April 2013 18:21, Manipulator <volcz kth.se> wrote:


 Please correct me if I'm wrong.
 It's not the allocation itself that is the problem it's that it will
 trigger the GC to stop the world at some point.


It's just another source of garbage. Which means this function of off
limits beneath a routine that's disabled collection, for instance. There
are lots of reasons to avoid making unnecessary garbage.
Apr 12 2013
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:

 Honestly, any new module could have appeared for approval at 
 this
 particular moment and I would have made the same criticisms.


Observation: this move wasn't something I would encourage. Jesse, 
the review manager for std.process, has specifically asked:


 Please place any further comments in the official review thread 
 leaving only your vote and a short comment (there should be no 
 need to reply to anyone).


There was ample time to discuss std.process during the informal 
and formal review. However, the situation with your replies is 
even worse: they don't even concern std.process in particular. 
Please consider starting a new thread next time, and stating your 
criticism aside to whatever poor module comes up the review queue 
;)
Apr 12 2013
prev sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:

 On 12 April 2013 17:30, Vladimir Panteleev 
 <vladimir thecybershadow.net>wrote:


 On Friday, 12 April 2013 at 07:22:42 UTC, Manu wrote:


 I agree that spawning processes is a low-frequency operation, 
 but it's a
 principle I'm trying to illustrate here.


 My point was that it is not that it's low-frequency, it's that 
 the OS
 process creation operation is so expensive, that a few memory 
 allocations
 will not make much of a difference in comparison. It's the 
 same as
 optimizing memory allocations in a program which is 
 intrinsically disk- or
 network-bound.



 Which OS are we talking about?
 What OS runs on an a Nintendo Wii? There's only 24mb of system 
 memory in
 that machine, can we afford to allocate it frivolously?

 Will I avoid phobos as a policy? Yes.


Do you create process at all?

I always had the impression those environments used real time OS 
like libraries and multitasking is achieved by co-routines or 
threads, not real processes.

Just a dummy comment from an industry outsider.

--
Paulo
Apr 12 2013
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 03:04:08 -0400, Manu <turkeyman gmail.com> wrote:


 On 12 April 2013 14:46, Jesse Phillips <Jessekphillips+d gmail.com>  
 wrote:


 It is that time, If you would like to see the proposed std.process  
 include
 into Phobos please vote yes. If one condition must be met specify under
 what condition, otherwise vote no.


 I didn't think I had much of an opinion on std.process, but I just gave  
 it
 a quick once over and noticed, relating to our recent discussions about
 flagrant GC usage throughout phobos, that this is riddled with the exact
 sorts of issues I've been talking about:


It's spawning a new process.  What is the issue with allocating a bit of  
memory?  The spawning of the process performance is going to dwarf that of  
any memory allocations.


 string[string] is used in the main API to receive environment variables;
 perhaps kinda convenient, but it's impossible to supply environment
 variables with loads of allocations.


I think you meant "without"?

What would be your suggestion?  string[string] is the built-in map type.   
How do you pass an environment map without having some allocations?


 toStringz is used liberally; alternatively, alloca() could allocate the
 c-string's on the stack and zero terminate them there, passing a pointer  
 to
 the stack string to the OS functions.


This would be a lot of effort for pretty much zero gain for the majority  
of cases.


 String concatenation is rampant! Look at this code to parse the env
 variables (which are already an AA):

     foreach (var, val; childEnv)
         envz[pos++] = (var~'='~val~'\0').ptr;


This could be improved.  It could also be optimized into a single  
allocation automatically by the compiler (it might already be).  The API  
would not be affected by this improvement, though.


 And many many more of the same...

 This is a lib that spawns a process, I can't imagine why this library
 should need to allocate a single byte, but it allocates wildly just to
 perform a trivial pass-through to the OS.


It is not a trivial pass-through.  Different OSes require different  
parameter types.  Look at the difference between the windows and posix  
implementations.  We are writing a cross-platform library, so whatever we  
pick will have to be converted on at least one OS.


 This module is no exception either, basically all of phobos is like this,
 and I'd like to see careful consideration made to these details in the
 future.


I too like to avoid memory allocations.  But the level to which you  
require is too stringent.  Not even Tango, which made low heap-allocations  
a priority, avoided allocations in it's process library.


 I've said before, I sadly have to avoid phobos like the plague. Some
 modules (like this one) that provide fundamental functionality - not just
 helper functions - can't be avoided. Requirements for those should be  
 extra
 strict in my opinion.


We cannot cater to all developers.  To put all developers through a lot of  
pain in order to attract the fringe ones is not a practical goal.

I think the convenience of std.process is what it is there for.  For the  
highest performance, you are able to call the OS functions directly.

-Steve
Apr 12 2013
parent reply Manu <turkeyman gmail.com> writes:
On 12 April 2013 17:35, Steven Schveighoffer <schveiguy yahoo.com> wrote:


 On Fri, 12 Apr 2013 03:04:08 -0400, Manu <turkeyman gmail.com> wrote:


 string[string] is used in the main API to receive environment variables;

 perhaps kinda convenient, but it's impossible to supply environment
 variables with loads of allocations.


 I think you meant "without"?


Yes.


What would be your suggestion?  string[string] is the built-in map type.

  How do you pass an environment map without having some allocations?



I'd use string[].


 toStringz is used liberally; alternatively, alloca() could allocate the

 c-string's on the stack and zero terminate them there, passing a pointer
 to
 the stack string to the OS functions.


 This would be a lot of effort for pretty much zero gain for the majority
 of cases.



A trivial ... mixin template (?) could wrap it up, I don't see it
particularly less convenient than calling toStringz().
Perhaps there are other tools missing from phobos if this is somehow hard...


 String concatenation is rampant! Look at this code to parse the env

 variables (which are already an AA):

     foreach (var, val; childEnv)
         envz[pos++] = (var~'='~val~'\0').ptr;


 This could be improved.  It could also be optimized into a single
 allocation automatically by the compiler (it might already be).  The API
 would not be affected by this improvement, though.


I've never seem the compiler apply that optimisation, although I often wish
it would.
I saw an appender appear a few pages below, that would be an improvement
here too I guess.


 And many many more of the same...

 This is a lib that spawns a process, I can't imagine why this library
 should need to allocate a single byte, but it allocates wildly just to
 perform a trivial pass-through to the OS.


 It is not a trivial pass-through.  Different OSes require different
 parameter types.  Look at the difference between the windows and posix
 implementations.  We are writing a cross-platform library, so whatever we
 pick will have to be converted on at least one OS.



Writing cross-platform code is my life.
They are trivial conversions. It could be done on the stack easily.
Granted, ideally the compiler could theoretically perform some of those
improvements, but it doesn't, and probably depends on other changes to be
able to do so anyway.
For instance, the compiler couldn't confidently lower the allocations to
the stack unless 'scope' arguments worked, so it knew it wouldn't escape
the OS call it was being handed to.


 This module is no exception either, basically all of phobos is like this,

 and I'd like to see careful consideration made to these details in the
 future.


 I too like to avoid memory allocations.  But the level to which you
 require is too stringent.  Not even Tango, which made low heap-allocations
 a priority, avoided allocations in it's process library.



But it would be trivial to use the stack in these cases. It doesn't matter
if it adds a few lines under the hood.

I'm really just trying to make the point that it should ALWAYS be a key
consideration, and people should make it habit to think about/criticise
these things when accepting code into phobos.
This is the standard library, it will be used in more D code than any other
library ever. Why skimp here?


 I've said before, I sadly have to avoid phobos like the plague. Some

 modules (like this one) that provide fundamental functionality - not just
 helper functions - can't be avoided. Requirements for those should be
 extra
 strict in my opinion.


 We cannot cater to all developers.



In this case you certainly can, it would be fairly trivial to do.


 To put all developers through a lot of pain in order to attract the fringe

 ones is not a practical goal.


What pain? Are you saying a stack variable is somehow hard?
If that's the case, then there's clearly some other fundamental tools
missing from phobos.


I think the convenience of std.process is what it is there for.  For the

 highest performance, you are able to call the OS functions directly.


No, it's to save (ideally) _all_ programmers the effort of rewriting this
module themselves.
This is exactly the sort of thing I waste loads of my life doing, rewriting
wheels because the author was a PC programmer, and simply didn't give a
shit.
What a waste of life >_<

I re-iterate, maybe my points aren't of critical importance in relation to
std.process specifically, I'm trying to raise some general issues with
phobos submissions (that shouldn't exclude std.process either).
Apr 12 2013
next sibling parent reply "Dicebot" <m.strashun gmail.com> writes:
On Friday, 12 April 2013 at 08:14:25 UTC, Manu wrote:

 ...


While I generally agree with your point, std.process is hardly 
the place to start from. It is not performance-critical 
functionality and it is not basic functionality that may be used 
in other critical modules. It is blaming consequences instead of 
reasons, not really practical.
Apr 12 2013
parent Manu <turkeyman gmail.com> writes:
On 12 April 2013 18:21, Dicebot <m.strashun gmail.com> wrote:


 On Friday, 12 April 2013 at 08:14:25 UTC, Manu wrote:


 ...


 While I generally agree with your point, std.process is hardly the place
 to start from. It is not performance-critical functionality and it is not
 basic functionality that may be used in other critical modules. It is
 blaming consequences instead of reasons, not really practical.


And I generally agree with your point. Like I said, I would have made this
criticisms of whatever came alone ;)
Apr 12 2013
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 04:14:15 -0400, Manu <turkeyman gmail.com> wrote:


 On 12 April 2013 17:35, Steven Schveighoffer <schveiguy yahoo.com> wrote:
 What would be your suggestion?  string[string] is the built-in map type.

  How do you pass an environment map without having some allocations?



 I'd use string[].


You mean with format "a=b"?  I suppose that's possible, though horrible to  
work with before passing in.  Plus what happens if you have ["a=b", "a=c"]  
?  AA's prevent these kinds of mistakes/contradictions.

I think someone suggested using a templated map type, that is probably a  
good idea.  Then you can provide whatever type you wish, with opIndex  
capability.


 toStringz is used liberally; alternatively, alloca() could allocate the
  c-string's on the stack and zero terminate them there, passing a  
 pointer to
  the stack string to the OS functions.





  This would be a lot of effort for pretty much zero gain for the  
 majority of cases.


 A trivial ... mixin template (?) could wrap it up, I don't see it  
 particularly less convenient than calling toStringz().
 Perhaps there are other tools missing from phobos if this is somehow  
 hard...


alloca is hard to use, specifically because it can't be wrapped into a  
function.  I don't know how easy it would be to do with a mixin, I haven't  
tried it.

Note that toStringz likely is going to avoid reallocation or copying if  
possible -- it's very likely that a string already has a 0 after it, and  
also very likely that if it doesn't, it has room to append a 0.

I still feel that if it can't be done easily, it over-complicates code for  
very little gain.

As others have said too, this is an implementation detail.  It can be done  
as a later pull request if you feel up to making it work!

-Steve
Apr 12 2013
parent reply "Tove" <tove fransson.se> writes:
On Friday, 12 April 2013 at 15:43:27 UTC, Steven Schveighoffer 
wrote:

 On Fri, 12 Apr 2013 04:14:15 -0400, Manu <turkeyman gmail.com>

 I'd use string[].


 You mean with format "a=b"?  I suppose that's possible, though 
 horrible to work with before passing in.  Plus what happens if 
 you have ["a=b", "a=c"] ?  AA's prevent these kinds of 
 mistakes/contradictions.


I prefer Manu's idea with the API accepting string[], it's closer 
to the native format.

Then you could simply provide a convenience conversion from a 
map...
ex env!["foo" : "bar"] which would convert it to ["foo=bar"].

This also has the added benefit of being self 
documenting(considering the lack of named parameters).

But most importantly So the user has a free choice of 
constructing the env parameter manually in the most efficient way 
or using the lazy convenience function.
Apr 12 2013
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 15:26:12 -0400, Tove <tove fransson.se> wrote:


 On Friday, 12 April 2013 at 15:43:27 UTC, Steven Schveighoffer wrote:

 On Fri, 12 Apr 2013 04:14:15 -0400, Manu <turkeyman gmail.com>

 I'd use string[].


 You mean with format "a=b"?  I suppose that's possible, though horrible  
 to work with before passing in.  Plus what happens if you have ["a=b",  
 "a=c"] ?  AA's prevent these kinds of mistakes/contradictions.


 I prefer Manu's idea with the API accepting string[], it's closer to the  
 native format.


That's great for the library on POSIX systems, but for the user, it's not  
as straightforward.  One must construct "x=y" strings instead of setting x  
to y.  Unless you are using literals, this involves an allocation anyway.

Also, this is not the native format on Windows which passes the  
environment in one large string delimited by nulls.


 Then you could simply provide a convenience conversion from a map...
 ex env!["foo" : "bar"] which would convert it to ["foo=bar"].

 This also has the added benefit of being self documenting(considering  
 the lack of named parameters).


So for the most convenient/common case, you want to add an allocation?


 But most importantly So the user has a free choice of constructing the  
 env parameter manually in the most efficient way or using the lazy  
 convenience function.


I think the best suggestion so far is to allow a templated version for  
which you can provide an opIndex method, and a possible toEnv method  
(versioned for both Posix and Windows of course).  Then you have the power  
to avoid allocation as much as possible.  At the moment with AA's being  
the interface, we have no possibility to avoid allocation, and I  
understand that point.  But I don't want to eliminate that case or make it  
specifically more inefficient.

-Steve
Apr 12 2013
parent reply "Tove" <tove fransson.se> writes:
On Friday, 12 April 2013 at 20:24:05 UTC, Steven Schveighoffer 
wrote:

 On Fri, 12 Apr 2013 15:26:12 -0400, Tove <tove fransson.se> 
 wrote:


 So for the most convenient/common case, you want to add an 
 allocation?


with the original proposal there is one anyway...

But with my suggested approach you could create many processes 
reusing the same env... only paying the conversion/allocation 
cost once(outside of the loop).
Apr 12 2013
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 16:32:37 -0400, Tove <tove fransson.se> wrote:


 On Friday, 12 April 2013 at 20:24:05 UTC, Steven Schveighoffer wrote:

 On Fri, 12 Apr 2013 15:26:12 -0400, Tove <tove fransson.se> wrote:


 So for the most convenient/common case, you want to add an allocation?


 with the original proposal there is one anyway...


I meant add an additional allocation to what is there.  There needs to be  
one allocation to collect all the variables into one long string (on  
Windows), and with your suggestion, it has to go through an intermediate  
"key=value" format.


 But with my suggested approach you could create many processes reusing  
 the same env... only paying the conversion/allocation cost once(outside  
 of the loop).


This would be attractive.  But I still want a indexable object.  Having to  
generate a=b strings is awkward.

It could be something that does the right thing on POSIX (generate "a=b"  
under the hood) or Windows (Probably can cache the generated environment  
string).

-Steve
Apr 12 2013
parent "Tove" <tove fransson.se> writes:
On Friday, 12 April 2013 at 20:52:55 UTC, Steven Schveighoffer 
wrote:

 On Fri, 12 Apr 2013 16:32:37 -0400, Tove <tove fransson.se> 
 wrote:


 On Friday, 12 April 2013 at 20:24:05 UTC, Steven Schveighoffer 
 wrote:

 On Fri, 12 Apr 2013 15:26:12 -0400, Tove <tove fransson.se> 
 wrote:


 So for the most convenient/common case, you want to add an 
 allocation?


 with the original proposal there is one anyway...


 I meant add an additional allocation to what is there.  There 
 needs to be one allocation to collect all the variables into 
 one long string (on Windows), and with your suggestion, it has 
 to go through an intermediate "key=value" format.


 But with my suggested approach you could create many processes 
 reusing the same env... only paying the conversion/allocation 
 cost once(outside of the loop).


 This would be attractive.  But I still want a indexable object.
  Having to generate a=b strings is awkward.

 It could be something that does the right thing on POSIX 
 (generate "a=b" under the hood) or Windows (Probably can cache 
 the generated environment string).

 -Steve


Hmhm, I see your point. Could our custom indexable object have

'alias this' to an 'union env'? which is the in param to the 
process api?
Apr 12 2013
prev sibling parent Martin Nowak <code dawg.eu> writes:
On 04/12/2013 10:14 AM, Manu wrote:

         String concatenation is rampant! Look at this code to parse the env
         variables (which are already an AA):

              foreach (var, val; childEnv)
                  envz[pos++] = (var~'='~val~'\0').ptr;


     This could be improved.  It could also be optimized into a single
     allocation automatically by the compiler (it might already be).  The
     API would not be affected by this improvement, though.


 I've never seem the compiler apply that optimisation, although I often
 wish it would.
 I saw an appender appear a few pages below, that would be an improvement
 here too I guess.


It is only a single runtime call to _d_arraycatnT which precomputes the 
resulting length.
https://github.com/D-Programming-Language/druntime/blob/e5415ed4f0638fcd1d0f8e06aec4197ad108449f/src/rt/lifetime.d#L2031
Apr 13 2013
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 12 Apr 2013 17:04:08 +1000
schrieb Manu <turkeyman gmail.com>:


 
 I've said before, I sadly have to avoid phobos like the plague. Some
 modules (like this one) that provide fundamental functionality - not
 just helper functions - can't be avoided. Requirements for those
 should be extra strict in my opinion.
 


Most (GC) allocations could be fixed without breaking the API. I can
see 2 places where the API forces _GC_ allocations:
* string[string]
* thrown Exceptions are allocated with the GC

There is no simple solution for these. Maybe we'll have a hashtable in
the standard library at some point which will allow to use custom
allocators. Then std.process could use these. Exceptions require some
thinking. You can of course allocate them using any allocator, but
freeing them is difficult...

If you want to get rid of all (not only GC) allocations, there's another
issue: As D strings are not zero terminated we'll always have
allocations passing those to C code. Maybe we should have a cstring
type in phobos which would just be a string which is guaranteed to be
zero terminated.

http://dpaste.dzfl.pl/e76aa995 (needs some inout though)
Apr 12 2013
next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Friday, 12 April 2013 at 08:34:15 UTC, Johannes Pfau wrote:

 Am Fri, 12 Apr 2013 17:04:08 +1000
 schrieb Manu <turkeyman gmail.com>:


 
 I've said before, I sadly have to avoid phobos like the 
 plague. Some
 modules (like this one) that provide fundamental functionality 
 - not
 just helper functions - can't be avoided. Requirements for 
 those
 should be extra strict in my opinion.
 


 Most (GC) allocations could be fixed without breaking the API. 
 I can
 see 2 places where the API forces _GC_ allocations:
 * string[string]
 * thrown Exceptions are allocated with the GC

 There is no simple solution for these. Maybe we'll have a 
 hashtable in
 the standard library at some point which will allow to use 
 custom
 allocators. Then std.process could use these. Exceptions 
 require some
 thinking. You can of course allocate them using any allocator, 
 but
 freeing them is difficult...

 If you want to get rid of all (not only GC) allocations, 
 there's another
 issue: As D strings are not zero terminated we'll always have
 allocations passing those to C code. Maybe we should have a 
 cstring
 type in phobos which would just be a string which is guaranteed 
 to be
 zero terminated.

 http://dpaste.dzfl.pl/e76aa995 (needs some inout though)



This is the Object Pascal way, although you can reach to a point 
where you get language native strings and C strings mixed all 
over the place.

Not to mention that C strings should not be allowed in safe code.

--
Paulo
Apr 12 2013
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On 12 April 2013 18:34, Johannes Pfau <nospam example.com> wrote:


 Am Fri, 12 Apr 2013 17:04:08 +1000
 schrieb Manu <turkeyman gmail.com>:


 I've said before, I sadly have to avoid phobos like the plague. Some
 modules (like this one) that provide fundamental functionality - not
 just helper functions - can't be avoided. Requirements for those
 should be extra strict in my opinion.


 Most (GC) allocations could be fixed without breaking the API. I can
 see 2 places where the API forces _GC_ allocations:
 * string[string]
 * thrown Exceptions are allocated with the GC

 There is no simple solution for these. Maybe we'll have a hashtable in
 the standard library at some point which will allow to use custom
 allocators. Then std.process could use these. Exceptions require some
 thinking. You can of course allocate them using any allocator, but
 freeing them is difficult...


Exceptions are exceptional, I don't think that's an issue.
the string[string] in the api is guaranteed garbage, and will never be
changed. What's the advantage of using an AA in this case?

If you want to get rid of all (not only GC) allocations, there's another

 issue: As D strings are not zero terminated we'll always have
 allocations passing those to C code.



They are being passed through to a C function that doesn't retain the
pointer; they can be allocated on the stack.

Maybe we should have a cstring

 type in phobos which would just be a string which is guaranteed to be
 zero terminated.


Maybe, although the point here is to wrap up and hide the C API, so that's
not useful here. A helper in phobos for allocating a (c)string on the stack
may be useful...
Apr 12 2013
prev sibling parent Manu <turkeyman gmail.com> writes:
On 12 April 2013 18:34, Johannes Pfau <nospam example.com> wrote:


 Am Fri, 12 Apr 2013 17:04:08 +1000
 schrieb Manu <turkeyman gmail.com>:


 I've said before, I sadly have to avoid phobos like the plague. Some
 modules (like this one) that provide fundamental functionality - not
 just helper functions - can't be avoided. Requirements for those
 should be extra strict in my opinion.


 Most (GC) allocations could be fixed without breaking the API. I can
 see 2 places where the API forces _GC_ allocations:
 * string[string]
 * thrown Exceptions are allocated with the GC

 There is no simple solution for these. Maybe we'll have a hashtable in
 the standard library at some point which will allow to use custom
 allocators. Then std.process could use these. Exceptions require some
 thinking. You can of course allocate them using any allocator, but
 freeing them is difficult...


Exceptions are exceptional, I don't think that's an issue.
the string[string] in the api is guaranteed garbage, and will never be
changed. What's the advantage of using an AA in this case?

If you want to get rid of all (not only GC) allocations, there's another

 issue: As D strings are not zero terminated we'll always have
 allocations passing those to C code.



They are being passed through to a C function that doesn't retain the
pointer; they can be allocated on the stack.

Maybe we should have a cstring

 type in phobos which would just be a string which is guaranteed to be
 zero terminated.


Maybe, although the point here is to wrap up and hide the C API, so that's
not useful here. A helper in phobos for allocating a (c)string on the stack
may be useful...
Apr 12 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 12 Apr 2013 17:04:08 +1000
schrieb Manu <turkeyman gmail.com>:



 
 toStringz is used liberally; alternatively, alloca() could allocate
 the c-string's on the stack and zero terminate them there, passing a
 pointer to the stack string to the OS functions.
 


Are you sure this is possible in this specific case?
The execve manpage says:

execve() does not return on success, and the text, data, bss, and
_stack_ of the calling process are overwritten by that of the program
loaded.

There is no documentation if the environment variables are copied
internally in execve.
Apr 12 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-04-12 09:04, Manu wrote:


 I've said before, I sadly have to avoid phobos like the plague. Some
 modules (like this one) that provide fundamental functionality - not
 just helper functions - can't be avoided. Requirements for those should
 be extra strict in my opinion.


Have you tried Tango? For most functions that do some array or string 
processing you have the option of passing your own buffer. If the buffer 
is too small, the function will allocate, otherwise it won't.

Docs: http://www.dsource.org/projects/tango/docs/current/
Tango D2: https://github.com/SiegeLord/Tango-D2

-- 
/Jacob Carlborg
Apr 12 2013
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
12-Apr-2013 11:04, Manu пишет:

 On 12 April 2013 14:46, Jesse Phillips <Jessekphillips+d gmail.com
 <mailto:Jessekphillips+d gmail.com>> wrote:

     It is that time, If you would like to see the proposed std.process
     include into Phobos please vote yes. If one condition must be met
     specify under what condition, otherwise vote no.


 I didn't think I had much of an opinion on std.process, but I just gave
 it a quick once over and noticed, relating to our recent discussions
 about flagrant GC usage throughout phobos, that this is riddled with the
 exact sorts of issues I've been talking about:

 string[string] is used in the main API to receive environment variables;
 perhaps kinda convenient, but it's impossible to supply environment
 variables with loads of allocations.

 toStringz is used liberally; alternatively, alloca() could allocate the
 c-string's on the stack and zero terminate them there, passing a pointer
 to the stack string to the OS functions.


Most of this is unsolvable until there is a STANDARD 
interface/policy/hooks for allocators in Phobos. All work to get around 
the cases like that at the moment is just a waste of time or specific 
hacks for particular bottlenecks.

I did a fair amount of these in std.regex to make matching GC-free and 
almost heap-free - it preallocates on the first call. It's an example of 
a necessary hack and it's not pretty. More then that I'm sure I'd have 
to re-write the whole memory allocation code once we (finally!) have he 
standard way to do it.

TL;DR - until using allocators is easy and straight-forward (and 
standard) there would be no end of GC-only hegeomny 'cause it's 
easier/simpler. Not much to discuss ;)


 String concatenation is rampant! Look at this code to parse the env
 variables (which are already an AA):

 foreach (var, val; childEnv)

 envz[pos++] = (var~'='~val~'\0').ptr;


Horrible...
But let's focus on the API parts. The internals could be fixed with a 
pull request or two. Freaking out because of a GC happy code doesn't 
help those waiting for *any* sane process management module in std.


-- 
Dmitry Olshansky
Apr 12 2013
prev sibling parent "Jesse Phillips" <Jessekphillips+d gmail.com> writes:
This is good feedback and all, but it would be good to have had 
this placed in the official review thread.

But it is too late now and I would not want to break the 
discussion up.
Apr 12 2013
prev sibling next sibling parent reply "Dicebot" <m.strashun gmail.com> writes:
Yes.
Apr 12 2013
parent "timotheecour" <timothee.cour2 gmail.com> writes:
Yes.
Apr 12 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 12 Apr 2013 06:46:51 +0200
schrieb "Jesse Phillips" <Jessekphillips+d gmail.com>:


 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes. If one condition 
 must be met specify under what condition, otherwise vote no.
 
 std.process by Lars Kyllingstad and Steven Schveighoffer is a 
 suggested improvement to the existing std.process and is a major 
 change to the API. The original API remains but these will be 
 going through deprecation.
 
 In summary of the discussion there was concern of the use of 
 Error and Exception. Lars is very interested in getting the 
 standard hierarchy http://wiki.dlang.org/DIP33 thus leaving these 
 at the safest level to allow for change without breaking code.
 
 Please place any further comments in the official review thread 
 leaving only your vote and a short comment (there should be no 
 need to reply to anyone).
 
 Docs: 
 http://www.kyllingen.net/code/std-process2/phobos-prerelease/std_process.html
 
 Source: 
 https://github.com/kyllingstad/phobos/blob/std-process2/std/process.d
 
 Friday April 19 PST will be the last day of voting.


Yes!
Apr 12 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Apr 12, 2013 at 06:46:51AM +0200, Jesse Phillips wrote:

 It is that time, If you would like to see the proposed std.process
 include into Phobos please vote yes. If one condition must be met
 specify under what condition, otherwise vote no.


Yes!


--T
Apr 12 2013
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:

 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes.


Yes! :)
Apr 12 2013
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Fri, 12 Apr 2013 05:46:51 +0100, Jesse Phillips  
<Jessekphillips+d gmail.com> wrote:


 It is that time, If you would like to see the proposed std.process  
 include into Phobos please vote yes. If one condition must be met  
 specify under what condition, otherwise vote no.


Yes.

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/
Apr 12 2013
prev sibling next sibling parent "Andrea Fontana" <nospam example.com> writes:
On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:

 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes. If one 
 condition must be met specify under what condition, otherwise 
 vote no.

 std.process by Lars Kyllingstad and Steven Schveighoffer is a 
 suggested improvement to the existing std.process and is a 
 major change to the API. The original API remains but these 
 will be going through deprecation.


Yes.

A compiler warning like for deprecation should added for 
"experimental" IMO rather than use exp. vs std.
Apr 12 2013
prev sibling next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 4/12/13, Manu <turkeyman gmail.com> wrote:

 string[string] is used in the main API to receive environment variables;
 perhaps kinda convenient, but it's impossible to supply environment
 variables with loads of allocations.


I think the functions which take the environment as an associative
array can be changed without breaking any user code. E.g.:

Pid spawnProcess(Env)(in char[][] args,
                 File stdin = std.stdio.stdin,
                 File stdout = std.stdio.stdout,
                 File stderr = std.stdio.stderr,

                 /* was const string[string] env = null */
                 const Env env = Env.init,

                 Config config = Config.none)
    if (isSomeAssociativeArray!Env)  // introduce constraint
{
}

Then the user can provide their custom stack-based type which implements opIn_r.
Apr 12 2013
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 11:24:37 -0400, Andrej Mitrovic  
<andrej.mitrovich gmail.com> wrote:


 On 4/12/13, Manu <turkeyman gmail.com> wrote:

 string[string] is used in the main API to receive environment variables;
 perhaps kinda convenient, but it's impossible to supply environment
 variables with loads of allocations.


 I think the functions which take the environment as an associative
 array can be changed without breaking any user code. E.g.:

 Pid spawnProcess(Env)(in char[][] args,
                  File stdin = std.stdio.stdin,
                  File stdout = std.stdio.stdout,
                  File stderr = std.stdio.stderr,

                  /* was const string[string] env = null */
                  const Env env = Env.init,

                  Config config = Config.none)
     if (isSomeAssociativeArray!Env)  // introduce constraint
 {
 }

 Then the user can provide their custom stack-based type which implements  
 opIn_r.


It's a good idea.  I believe it can be done later without breaking any  
code.

-Steve
Apr 12 2013
prev sibling next sibling parent "Joshua Niehus" <jm.niehus gmail.com> writes:
yes
Apr 12 2013
prev sibling next sibling parent Mike Wey <mike-wey example.com> writes:
Yes.

-- 
Mike Wey
Apr 12 2013
prev sibling next sibling parent "Matej Nanut" <matejnanut gmail.com> writes:
On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:

 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes. If one 
 condition must be met specify under what condition, otherwise 
 vote no.


Yes.
Apr 13 2013
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
YES
Apr 13 2013
prev sibling next sibling parent "Kapps" <opantm2+spam gmail.com> writes:
On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:

 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes. If one 
 condition must be met specify under what condition, otherwise 
 vote no.


Yes
Apr 14 2013
prev sibling parent "Jesse Phillips" <Jessekphillips+d gmail.com> writes:
Yes
Apr 16 2013