www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Official stdx

reply "John Colvin" <john.loughran.colvin gmail.com> writes:
Adding new (or replacement) phobos modules without wider testing 
is not a scalable approach for D.
New modules go from unofficial to official in a single step and 
are therefore inadequately battle-tested before becoming part of 
the somewhat ossified environment of a standard library.

Assertions:

1) In the current situation, modules go from independent projects 
to becoming official parts of phobos at a single point. It's a 
binary switch.

2) New modules, in the form proposed for submission, are fully 
read through by a small number of people.

3) New modules, in the form proposed for submission, have often 
been seriously used by only an even smaller number of people.

4) Breaking changes to phobos are currently undesirable.


Argument:
Due to the combination of 1,2 and 3, we unwittingly introduce 
bugs and poor design decisions in to the standard library, with 
all the inflexibility that entails. Due to 4 we then either 
cannot or do not fix these problems in an optimal fashion, if at 
all. We can do better.


Necessary Solution:
People need to really use these new modules before they are 
pulled in to phobos. In particular, the API must be stress tested 
in a variety of situations.*


Implementation:
In the current situation all the code is public, so people are 
totally free to try out the proposed modules and test them. They 
don't and they won't.

I propose that the current review process be redirected to a new 
target package in the phobos repo, stdx, which would then have a 
separate review process for inclusion in std.

I would imagine a compulsory waiting period of the order of 
months, combined with a requirement of evidence that the module 
has been effective in the real world and that any outstanding 
problems have been resolved appropriately**. Stdx would make no 
explicit promises about API stability, but modules would be 
considered as semi-finished products; Revisions that effectively 
amount to a significant rewrite (in particular to the API) would 
require re-submission for initial review.

Stdx would occupy a space between phobos and unaffiliated 
packages, allowing for subtle (i.e. missed during initial review) 
but critical problems requiring breakage to be identified BEFORE 
inclusion in the standard library proper. I believe providing 
this official stepping stone for new modules will result in 
significantly wider usage and testing, benefiting phobos, D and 
it's community.


*API design is *hard*. Knowing what is a good decision ahead of 
time is a matter of experience.
** The appropriate action may be to not fix it. There are always 
trade-offs.

P.S.
I am aware that this is not a new idea at all. I thought it was 
worth presenting with a clean slate.
Oct 04 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
I liked the proposal to add special `dub` category for those 
(`dub` needs some categorization anyway). Only drawback is that 
it actually needs to be implemented instead of simply making a 
decision :)
Oct 04 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-10-04 12:17, Dicebot wrote:

 I liked the proposal to add special `dub` category for those (`dub`
 needs some categorization anyway). Only drawback is that it actually
 needs to be implemented instead of simply making a decision :)


For that, dub needs to be bundled with DMD.

-- 
/Jacob Carlborg
Oct 04 2013
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 4 October 2013 at 13:30:39 UTC, Jacob Carlborg wrote:

 On 2013-10-04 12:17, Dicebot wrote:

 I liked the proposal to add special `dub` category for those 
 (`dub`
 needs some categorization anyway). Only drawback is that it 
 actually
 needs to be implemented instead of simply making a decision :)


 For that, dub needs to be bundled with DMD.


I think it will at some moment. Adding such categorization 
capabilities is just another step towards official endorsement.
Oct 04 2013
prev sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Friday, 4 October 2013 at 13:30:39 UTC, Jacob Carlborg wrote:

 On 2013-10-04 12:17, Dicebot wrote:

 I liked the proposal to add special `dub` category for those 
 (`dub`
 needs some categorization anyway). Only drawback is that it 
 actually
 needs to be implemented instead of simply making a decision :)


 For that, dub needs to be bundled with DMD.


Why? As long as the installation process is simple enough, I 
don't see a big problem with pointing users to an extra tool for 
access to ${favorite_awesome_library}.

David
Oct 05 2013
next sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Saturday, 5 October 2013 at 14:46:49 UTC, David Nadlinger 
wrote:

 Why? As long as the installation process is simple enough, I 
 don't see a big problem with pointing users to an extra tool 
 for access to ${favorite_awesome_library}.


(That being said, I think we should indeed try to offer 
batteries-included packages for people to jump into D 
development.)
Oct 05 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-10-05 16:46, David Nadlinger wrote:


 Why? As long as the installation process is simple enough, I don't see a
 big problem with pointing users to an extra tool for access to
 ${favorite_awesome_library}.


I just don't think it would be really official.

-- 
/Jacob Carlborg
Oct 05 2013
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 04/10/13 12:02, John Colvin wrote:

 I propose that the current review process be redirected to a new target package
 in the phobos repo, stdx, which would then have a separate review process for
 inclusion in std.


As you say, this has been proposed before and I think the objection was along 
the lines that in practice, so many people would just use stdx (because they 
wanted those features NOW, NOW, NOW!) that you'd have the same backwards 
compatibility issues arising there -- you'd wind up with the same fear of 
breaking change and deprecation as we currently have in the standard library, 
and it would also cause breaking change if stdx.somemodule was approved and 
moved to std.somemodule.

I remember someone pointed to a similar "experimental" module namespace in
Java, 
which effectively became a standard namespace in this way.


 Stdx would occupy a space between phobos and unaffiliated packages, allowing
for
 subtle (i.e. missed during initial review) but critical problems requiring
 breakage to be identified BEFORE inclusion in the standard library proper. I
 believe providing this official stepping stone for new modules will result in
 significantly wider usage and testing, benefiting phobos, D and it's community.


There is an alternative risk -- that people will be more inclined to just throw 
stuff over the wall into stdx because, hey, it's an experimental/testing area, 
the whole idea is for people to try out imperfect code and work out what's
wrong 
with it ...

... but then people are kind of locked into that imperfect code, because even
if 
it's imperfect it still provides something that no other library/module does, 
and they and their code suffer from the breaking changes all the same.


 I am aware that this is not a new idea at all. I thought it was worth
presenting
 with a clean slate.


I've presented Devil's Advocate positions above, but of course it was worth 
re-raising the idea.  Anything that does in practice raise the bar of Phobos 
quality is very welcome.
Oct 04 2013
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 4 October 2013 at 10:30:56 UTC, Joseph Rushton 
Wakeling wrote:

 On 04/10/13 12:02, John Colvin wrote:

 I propose that the current review process be redirected to a 
 new target package
 in the phobos repo, stdx, which would then have a separate 
 review process for
 inclusion in std.


 As you say, this has been proposed before and I think the 
 objection was along the lines that in practice, so many people 
 would just use stdx (because they wanted those features NOW, 
 NOW, NOW!) that you'd have the same backwards compatibility 
 issues arising there -- you'd wind up with the same fear of 
 breaking change and deprecation as we currently have in the 
 standard library


The lack of promise of API stability in stdx would have to be 
very explicit and obvious. The maintainers of phobos would have 
to remain stalwart in preventing any alteration of this, whether 
de facto or deliberate.


 and it would also cause breaking change if stdx.somemodule was 
 approved and moved to std.somemodule.


That is the least bad type of breaking change.


 I remember someone pointed to a similar "experimental" module 
 namespace in Java, which effectively became a standard 
 namespace in this way.


 Stdx would occupy a space between phobos and unaffiliated 
 packages, allowing for
 subtle (i.e. missed during initial review) but critical 
 problems requiring
 breakage to be identified BEFORE inclusion in the standard 
 library proper. I
 believe providing this official stepping stone for new modules 
 will result in
 significantly wider usage and testing, benefiting phobos, D 
 and it's community.


 There is an alternative risk -- that people will be more 
 inclined to just throw stuff over the wall into stdx because, 
 hey, it's an experimental/testing area, the whole idea is for 
 people to try out imperfect code and work out what's wrong with 
 it ...


The bar for passing initial review should remain exactly where it 
is now. Nothing should be allowed in to stdx unless it appears 
for all intents and purposes to be a finished, high quality 
module that would - to the best of the evidence available - be 
fine to add straight to phobos. The purpose of stdx is to gather 
more evidence before making the commitment.


 I am aware that this is not a new idea at all. I thought it 
 was worth presenting
 with a clean slate.


 I've presented Devil's Advocate positions above, but of course 
 it was worth re-raising the idea.  Anything that does in 
 practice raise the bar of Phobos quality is very welcome.
Oct 04 2013
parent reply "Kagamin" <spam here.lot> writes:
On Friday, 4 October 2013 at 10:49:58 UTC, John Colvin wrote:

 The lack of promise of API stability in stdx would have to be 
 very explicit and obvious.


Then call it `experimental` instead of stdx. That's obvious (stdx 
sound like an extension of std library). I wonder if someone 
would complain that experimental module broke.
Oct 07 2013
parent "eles" <eles eles.com> writes:
On Monday, 7 October 2013 at 15:36:23 UTC, Kagamin wrote:

 On Friday, 4 October 2013 at 10:49:58 UTC, John Colvin wrote:
 Then call it `experimental` instead of stdx. That's obvious 
 (stdx sound like an extension of std library). I wonder if 
 someone would complain that experimental module broke.


if nobody complains, then experimental would be useless: code 
could go directly into std.
Oct 07 2013
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 4 October 2013 at 10:30:56 UTC, Joseph Rushton 
Wakeling wrote:

 There is an alternative risk -- that people will be more 
 inclined to just throw stuff over the wall into stdx because, 
 hey, it's an experimental/testing area, the whole idea is for 
 people to try out imperfect code and work out what's wrong with 
 it ...


And to be 100% clear - I meant this dub category to be moderated 
and only filled after formal review process. We simply need some 
place for modules which got conceptually approved but can't yet 
meet API stability and/or implementation quality required for 
inclusion into Phobos as-is.
Oct 04 2013
prev sibling next sibling parent "Chris" <wendlec tcd.ie> writes:
The thing is, code is never perfect and as a language keeps 
evolving your latest tricks will soon become obsolete or 
deprecated. It happens all the time (Java, Cocoa etc. "use this 
instead"). If I look back at the code I wrote years ago, it still 
works, but it's no longer up to current standards (my own as well 
as the language's). I think including things in the std. library 
really makes people (like me) use and thereby _test_ them, asking 
(not so) stupid questions on D.learn etc. If I have learned 
anything it's that my most ignorant (innocent) questions trigger 
responses from people who are more experienced, pointing out this 
or that flaw, telling you tricks and workarounds. D is a language 
based on practical experience, not on ideology or anything 
(that's why I like it). So stdx, while it seems to be a good 
idea, would be a limbo for code, eternally in a state of "not 
bad, but not quite there yet". And all code is like that. We'll 
never quite get there.
Oct 04 2013
prev sibling next sibling parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Friday, 4 October 2013 at 10:02:21 UTC, John Colvin wrote:

 I would imagine a compulsory waiting period of the order of 
 months, combined with a requirement of evidence that the module 
 has been effective in the real world and that any outstanding 
 problems have been resolved appropriately**.


I think this requirement is unobtainable. You're basically 
saying, "Hey go use this in your real world applications. We've 
got a mandated break of that application, just don't know when 
that is. But totally use this like you can rely on it."

I know it is really nice to have these test libraries released 
with the compiler, but people really just need to go out and use 
these libraries before the inclusion. I'm guilty too. I had use 
for std.uuid, but didn't test it against my real code. It went 
through review and even after inclusion I was still using the 
basic generator I created from an RFC doc. I've since made the 
switch, but I didn't do it because a review needed my help.

I think a stdx could be beneficial in our current state but I 
think it should be temporary. One maybe two years. The timing for 
move should be clear, accepted => stdx => next release => std.
Oct 04 2013
parent "barryharris" <slackovsky gmail.com> writes:
On Saturday, 5 October 2013 at 01:53:13 UTC, Jesse Phillips wrote:

 On Friday, 4 October 2013 at 10:02:21 UTC, John Colvin wrote:

 I would imagine a compulsory waiting period of the order of 
 months, combined with a requirement of evidence that the 
 module has been effective in the real world and that any 
 outstanding problems have been resolved appropriately**.


 I think this requirement is unobtainable. You're basically 
 saying, "Hey go use this in your real world applications. We've 
 got a mandated break of that application, just don't know when 
 that is. But totally use this like you can rely on it."

 I know it is really nice to have these test libraries released 
 with the compiler, but people really just need to go out and 
 use these libraries before the inclusion. I'm guilty too. I had 
 use for std.uuid, but didn't test it against my real code. It 
 went through review and even after inclusion I was still using 
 the basic generator I created from an RFC doc. I've since made 
 the switch, but I didn't do it because a review needed my help.

 I think a stdx could be beneficial in our current state but I 
 think it should be temporary. One maybe two years. The timing 
 for move should be clear, accepted => stdx => next release => 
 std.


I quite like the idea of a stdx and imagine it to be similar to 
what we have in boost for C++.

A set of high quality, peer-reviewed libraries that are worthy of 
Phobos but haven't yet had enough real-world testing to be 
accepted into Phobos proper. Hopefully the peer-review process 
would include core Phobos developers to ensure the D-essence and 
quality is maintained throughout.


 I think a stdx could be beneficial in our current state but I 
 think it should be temporary. One maybe two years. The timing 
 for move should be clear, accepted => stdx => next release => 
 std.


I agree partially with this. Once a library is flagged for Phobos 
from stdx it has to have a consistent and set migration path. But 
I don't think all stdx libraries need to be Phobos targets. For 
example, I don't think Phobos needs a CSV reader module, nor 
std.net.curl, even though both are very useful. Opt-in stdx 
modules via the official dub package manager would be a good 
option, allowing Phobos proper to remain small and light as 
possible.

G.
Oct 05 2013
prev sibling next sibling parent Robert Schadek <realburner gmx.de> writes:
On 10/04/2013 12:02 PM, John Colvin wrote:

 Adding new (or replacement) phobos modules without wider testing is
 not a scalable approach for D.
 New modules go from unofficial to official in a single step and are
 therefore inadequately battle-tested before becoming part of the
 somewhat ossified environment of a standard library.


I don't think that this is a good idea. Just consider linux pre and post
2.6. Pre, the % 2 == 1 branches took forever and now they have more
changes than ever and put out multiple release per year (I know we're
not linux, yet). I also would consider stdx as some sort of dead end for
code, like "it's ok, but not phobos worthy".
Oct 05 2013
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 04/10/13 12:02, John Colvin wrote:

 I propose that the current review process be redirected to a new target package
 in the phobos repo, stdx, which would then have a separate review process for
 inclusion in std.


Just a thought after reading through the discussion on std.d.lexer, but --
isn't 
this what etc library modules are for?

See: http://dlang.org/phobos/index.html

-----------------------------------------------------------------------------------
etc
     This is the root of a hierarchy of modules mirroring the std hierarchy.
     Modules in etc are not standard D modules. They are here because they are
     experimental, or for some other reason are not quite suitable for std,
     although they are still useful.
-----------------------------------------------------------------------------------
Oct 06 2013
parent "growler" <growlercab gmail.com> writes:
On Sunday, 6 October 2013 at 16:33:24 UTC, Joseph Rushton 
Wakeling wrote:

 On 04/10/13 12:02, John Colvin wrote:

 I propose that the current review process be redirected to a 
 new target package
 in the phobos repo, stdx, which would then have a separate 
 review process for
 inclusion in std.


 Just a thought after reading through the discussion on 
 std.d.lexer, but -- isn't this what etc library modules are for?

 See: http://dlang.org/phobos/index.html

 -----------------------------------------------------------------------------------
 etc
     This is the root of a hierarchy of modules mirroring the 
 std hierarchy.
     Modules in etc are not standard D modules. They are here 
 because they are
     experimental, or for some other reason are not quite 
 suitable for std,
     although they are still useful.
 -----------------------------------------------------------------------------------


I like the idea of dub.stdx: peer-reviewed, officially endorsed 
and rubber stamped modules

  * It provides a BOOST like repository of high quality 
peer-reviewed D libraries.

  * It decouples the modules from the Phobos+compiler release 
schedule.

  * It allows users to download official libraries individually, 
without upgrading compiler/Phobos or using git head.

  * It provides a place official D libraries too big for Phobos. 
I'm thinking of a Matlab toolbox type of repository with high 
quality code. 3D/2D Gfx, GUI, image processing libraries, 
linear-algebra, machine learning libraries etc. etc.

  * Dub might be able to collect stats on downloads per module and 
provide information about the popularity and amount of real-world 
use a module receives before proposing for Phobos.std.

Cheers
Oct 06 2013
prev sibling parent reply "eles" <eles eles.com> writes:
On Friday, 4 October 2013 at 10:02:21 UTC, John Colvin wrote:

 I propose that the current review process be redirected to a 
 new target package in the phobos repo, stdx, which would then 
 have a separate review process for inclusion in std.


not good. stdx is the same trap as javax: throw-away that never 
gets thown away and sticks around until forever.

better is to include it in std with a big pragma that warns at 
compilation time about its instability (a kind of staging... 
stage)

warnings about interface/behavior changes should be issued with 
every new release of the module *in code* (and in doc, of course)

old interfaces could be or could not be not kept with 
deperecation pragmas

in the medium/long term, pragmas will get removed progressively
Oct 07 2013
parent reply "Kagamin" <spam here.lot> writes:
On Monday, 7 October 2013 at 08:20:20 UTC, eles wrote:

 not good. stdx is the same trap as javax: throw-away that never 
 gets thown away and sticks around until forever.


That's just a matter of enforcing the policy. The only problem I 
see is that stdx can be good enough to be an excuse for not 
having the feature in phobos.
Oct 07 2013
parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 8 October 2013 at 06:23:41 UTC, Kagamin wrote:

 On Monday, 7 October 2013 at 08:20:20 UTC, eles wrote:

 not good. stdx is the same trap as javax: throw-away that 
 never gets thown away and sticks around until forever.


 That's just a matter of enforcing the policy. The only problem 
 I see is that stdx can be good enough to be an excuse for not 
 having the feature in phobos.


That is exactly why it is not suitable for D. Enforcing policies 
is not something we are good at ;) Generally if something is not 
going to work in an anarchic mode with some tooling automation 
support, it is not going to work at all ;)
Oct 08 2013