www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Phobos 2

reply Brad Anderson <eco gnuk.net> writes:
A (surely controversial) idea popped into my head while talking 
in #d on Freenode. The C++ guys are making an STL2 (the highlight 
of it being that it is range based). What about taking all the 
lessons learned from Phobos and creating a Phobos 2? It wouldn't 
replace the current version. You could import either in one 
program. It also wouldn't be a radical redesign. Most of Phobos 
could be used as is. What it would do is allow fixing some hard 
or impossible problems without losing backward compatibility.

We could do away with auto-decoding. Design it around using 
Andrei's allocators throughout. Make the GC optional from the 
start. Fix a few important naming conflicts and discrepancies.

There are problems, of course. Rampant code duplication is 
probably the biggest (though maybe public imports of identical 
code would help a lot).

I don't really expect this to go anywhere but I am curious to 
hear what changes you'd all like to see made to Phobos that can't 
happen because of backward compatibility. Also, how would you 
approach doing this? An on disk copy of Phobos with changes would 
not be an acceptable approach, I think.
Jun 01 2017
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2017-06-01 20:40, Brad Anderson wrote:


 I am curious to hear what changes you'd all like to see made to Phobos that
can't happen because
 of backward compatibility.


Perhaps completely asynchronous IO throughout.

-- 
/Jacob Carlborg
Jun 01 2017
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Jun 01, 2017 at 06:40:05PM +0000, Brad Anderson via Digitalmars-d wrote:

 A (surely controversial) idea popped into my head while talking in #d
 on Freenode. The C++ guys are making an STL2 (the highlight of it
 being that it is range based). What about taking all the lessons
 learned from Phobos and creating a Phobos 2? It wouldn't replace the
 current version. You could import either in one program. It also
 wouldn't be a radical redesign. Most of Phobos could be used as is.
 What it would do is allow fixing some hard or impossible problems
 without losing backward compatibility.


+1. I'm for this.  There's just too much code in the existing Phobos
that certain (wrong, in hindsight) design decisions are essentially
impossible to fix.



 We could do away with auto-decoding.


+1000. ;-)



 Design it around using Andrei's allocators throughout. Make the GC
 optional from the start.


+1.



 Fix a few important naming conflicts and discrepancies.


As well as certain less-than-ideal names that we were forced to use
because the good names have been taken up by not-so-nice implementations
dating from before we had the hindsight to know what the best design
was.



 There are problems, of course. Rampant code duplication is probably
 the biggest (though maybe public imports of identical code would help
 a lot).
 
 I don't really expect this to go anywhere but I am curious to hear
 what changes you'd all like to see made to Phobos that can't happen
 because of backward compatibility.


1) A big one, not necessarily caused by backward compatibility but more
by the sheer amount of code that would need to be fixed / rewritten, is
the nasty hairball of a dependency graph among the current Phobos
modules.

One particularly egregious example is std.format.  Due to the way it was
originally implemented, it depended on a lot of other code (IIRC
including std.bigint), so even if all you ever did was writefln("%d"),
for example, your program would end up depending on std.bigint,
std.complex, and all sorts of other stuff that never actually gets used.
This in itself is not necessarily *that* bad, but what really makes it
hurt is that many other modules depend on std.format. So you could be
importing a seemingly completely-unrelated module, and it would pull in
std.format, which in turn pulls in other stuff, and by the end you've
basically imported about half of Phobos, with the associated executable
bloat, even though you really only needed to call 1 function.

And the icing on the cake is that some of these complicated dependencies
are circular, which every now and then cause mysterious linker errors
and/or surprising dependence on declaration order, that mysteriously
vanish when you move seemingly-unrelated code around.

If we were to go the route of Phobos2, I'd like the principle of
pay-as-you-go to be baked into its design from the very beginning. If
all you need is to format an int, it shouldn't also pull in std.complex
and std.bigint. This can be done by compile-time format strings (which
we now have, though it's currently just a frontend to the existing
hairball code).  If function X and function Y are essentially
independent of each other, they really ought to go into their own
(sub)modules, now that we have package.d support in the compiler.

Of course, certain things like std.range.primitives and std.allocator
will be depended on by 90% of the other modules, which is unavoidable,
but which also means that we need to think VERY carefully about what we
put into these foundational modules. If at all possible they should not
depend on anything else.


2) A lesser item, perhaps would be to change the definitions of input /
forward ranges, so that forward ranges are just by-value ranges, and
input ranges are by-reference ranges, instead of forward ranges needing
explicit calling of .save to save the iteration state. See:

http://lists.cubik.org/pipermail/digitalmars-d/2015-November/241712.html


3) An even lesser item, but now that we have a chance: clean up the mess
that is the current isSomeString, isNarrowString, etc.. Well, some of
them will be useless since we wouldn't have autodecoding, but still, we
need to sit down and properly design these traits so that they are (1)
useful, (2) orthogonal, and (3) have sensible names.



 Also, how would you approach doing this? An on disk copy of Phobos
 with changes would not be an acceptable approach, I think.


I'd say, start in a different namespace than std, so that the two can
coexist until Phobos2 is ready to take over prime time. (And even then,
Phobos1 should probably stick around for a while until people have
migrated over.)  This new namespace can either be completely outside
std, like std2 (bad name, I know), or it can be a dedicated
sub-namespace, like std.new.

Preferably it should be something unique and easily greppable, so that
when Phobos1 is finally gone, we can replace the std namespace with a
simple search-and-replace.  But perhaps it would be best to just choose
a new namespace with a nice name instead, like phobos.*, so that std.*
will eventually just go the way of the dinosaur.

Once the namespace is sorted out, how we distribute the code should be
pretty easy to decide.  It could start off as an independent github
repo, perhaps, or a dub package or whatever, then when it becomes mature
enough we could graft it into the current Phobos repo, and thereby make
it available for all D users, and thus begin the gradual transition from
std.*.


T

-- 
Not all rumours are as misleading as this one.
Jun 01 2017
parent Mike B Johnson <Mikey Ikes.com> writes:
On Thursday, 1 June 2017 at 20:58:52 UTC, H. S. Teoh wrote:

 On Thu, Jun 01, 2017 at 06:40:05PM +0000, Brad Anderson via 
 Digitalmars-d wrote:

 A (surely controversial) idea popped into my head while 
 talking in #d on Freenode. The C++ guys are making an STL2 
 (the highlight of it being that it is range based). What about 
 taking all the lessons learned from Phobos and creating a 
 Phobos 2? It wouldn't replace the current version. You could 
 import either in one program. It also wouldn't be a radical 
 redesign. Most of Phobos could be used as is. What it would do 
 is allow fixing some hard or impossible problems without 
 losing backward compatibility.


 +1. I'm for this.  There's just too much code in the existing 
 Phobos that certain (wrong, in hindsight) design decisions are 
 essentially impossible to fix.



 We could do away with auto-decoding.


 +1000. ;-)



 Design it around using Andrei's allocators throughout. Make 
 the GC optional from the start.


 +1.



 Fix a few important naming conflicts and discrepancies.


 As well as certain less-than-ideal names that we were forced to 
 use because the good names have been taken up by not-so-nice 
 implementations dating from before we had the hindsight to know 
 what the best design was.



 There are problems, of course. Rampant code duplication is 
 probably the biggest (though maybe public imports of identical 
 code would help a lot).
 
 I don't really expect this to go anywhere but I am curious to 
 hear what changes you'd all like to see made to Phobos that 
 can't happen because of backward compatibility.


 1) A big one, not necessarily caused by backward compatibility 
 but more by the sheer amount of code that would need to be 
 fixed / rewritten, is the nasty hairball of a dependency graph 
 among the current Phobos modules.

 One particularly egregious example is std.format.  Due to the 
 way it was originally implemented, it depended on a lot of 
 other code (IIRC including std.bigint), so even if all you ever 
 did was writefln("%d"), for example, your program would end up 
 depending on std.bigint, std.complex, and all sorts of other 
 stuff that never actually gets used. This in itself is not 
 necessarily *that* bad, but what really makes it hurt is that 
 many other modules depend on std.format. So you could be 
 importing a seemingly completely-unrelated module, and it would 
 pull in std.format, which in turn pulls in other stuff, and by 
 the end you've basically imported about half of Phobos, with 
 the associated executable bloat, even though you really only 
 needed to call 1 function.

 And the icing on the cake is that some of these complicated 
 dependencies are circular, which every now and then cause 
 mysterious linker errors and/or surprising dependence on 
 declaration order, that mysteriously vanish when you move 
 seemingly-unrelated code around.

 If we were to go the route of Phobos2, I'd like the principle 
 of pay-as-you-go to be baked into its design from the very 
 beginning. If all you need is to format an int, it shouldn't 
 also pull in std.complex and std.bigint. This can be done by 
 compile-time format strings (which we now have, though it's 
 currently just a frontend to the existing hairball code).  If 
 function X and function Y are essentially independent of each 
 other, they really ought to go into their own (sub)modules, now 
 that we have package.d support in the compiler.

 Of course, certain things like std.range.primitives and 
 std.allocator will be depended on by 90% of the other modules, 
 which is unavoidable, but which also means that we need to 
 think VERY carefully about what we put into these foundational 
 modules. If at all possible they should not depend on anything 
 else.


 2) A lesser item, perhaps would be to change the definitions of 
 input / forward ranges, so that forward ranges are just 
 by-value ranges, and input ranges are by-reference ranges, 
 instead of forward ranges needing explicit calling of .save to 
 save the iteration state. See:

 http://lists.cubik.org/pipermail/digitalmars-d/2015-November/241712.html


 3) An even lesser item, but now that we have a chance: clean up 
 the mess
 that is the current isSomeString, isNarrowString, etc.. Well, 
 some of
 them will be useless since we wouldn't have autodecoding, but 
 still, we
 need to sit down and properly design these traits so that they 
 are (1)
 useful, (2) orthogonal, and (3) have sensible names.



 Also, how would you approach doing this? An on disk copy of 
 Phobos with changes would not be an acceptable approach, I 
 think.


 I'd say, start in a different namespace than std, so that the 
 two can coexist until Phobos2 is ready to take over prime time. 
 (And even then, Phobos1 should probably stick around for a 
 while until people have migrated over.)  This new namespace can 
 either be completely outside std, like std2 (bad name, I know), 
 or it can be a dedicated sub-namespace, like std.new.

 Preferably it should be something unique and easily greppable, 
 so that when Phobos1 is finally gone, we can replace the std 
 namespace with a simple search-and-replace.  But perhaps it 
 would be best to just choose a new namespace with a nice name 
 instead, like phobos.*, so that std.* will eventually just go 
 the way of the dinosaur.

 Once the namespace is sorted out, how we distribute the code 
 should be pretty easy to decide.  It could start off as an 
 independent github repo, perhaps, or a dub package or whatever, 
 then when it becomes mature enough we could graft it into the 
 current Phobos repo, and thereby make it available for all D 
 users, and thus begin the gradual transition from std.*.


 T


This could be done if a clear plan and layout was created before 
any real work was to start. Phobos should have the look and feel 
of .net IMO(or better, but not worse).


.Net is extremely well organized, logical, and useful.
Jun 02 2017
prev sibling next sibling parent Dsby <dushibaiyu yahoo.com> writes:
On Thursday, 1 June 2017 at 18:40:05 UTC, Brad Anderson wrote:

 A (surely controversial) idea popped into my head while talking 
 in #d on Freenode. The C++ guys are making an STL2 (the 
 highlight of it being that it is range based). What about 
 taking all the lessons learned from Phobos and creating a 
 Phobos 2? It wouldn't replace the current version. You could 
 import either in one program. It also wouldn't be a radical 
 redesign. Most of Phobos could be used as is. What it would do 
 is allow fixing some hard or impossible problems without losing 
 backward compatibility.

 [...]


Maybe we do not have to replace, just expand on the line. Now the 
standard library is a lot of things are common, do not rely on GC。
I write a lib to expand the Phobos : 
https://github.com/dushibaiyu/yu
Jun 01 2017
prev sibling next sibling parent 9il <ilyayaroshenko gmail.com> writes:
On Thursday, 1 June 2017 at 18:40:05 UTC, Brad Anderson wrote:

 A (surely controversial) idea popped into my head while talking 
 in #d on Freenode. The C++ guys are making an STL2 (the 
 highlight of it being that it is range based). What about 
 taking all the lessons learned from Phobos and creating a 
 Phobos 2? It wouldn't replace the current version. You could 
 import either in one program. It also wouldn't be a radical 
 redesign. Most of Phobos could be used as is. What it would do 
 is allow fixing some hard or impossible problems without losing 
 backward compatibility.


Mir libraries are alternative to Phobos
https://github.com/libmir


 We could do away with auto-decoding. Design it around using


Primitives that do not perform auto-decoding.
http://docs.algorithm.dlang.io/latest/mir_array_primitives.html


 Andrei's allocators throughout. Make the GC optional from the 
 start. Fix a few important naming conflicts and discrepancies.


It is a huge work.

Allocators are part of the Phobos, GC is part of the runtime.
So, allocators should be moved to runtime.


 There are problems, of course. Rampant code duplication is 
 probably the biggest (though maybe public imports of identical 
 code would help a lot).

 I don't really expect this to go anywhere but I am curious to 
 hear what changes you'd all like to see made to Phobos that 
 can't happen because of backward compatibility. Also, how would 
 you approach doing this? An on disk copy of Phobos with changes 
 would not be an acceptable approach, I think.


Mir uses separate dub packages for algorithmic and system parts. 
We are looking for new contributors.

Best,
Ilya
Jun 01 2017
prev sibling parent reply qznc <qznc web.de> writes:
On Thursday, 1 June 2017 at 18:40:05 UTC, Brad Anderson wrote:

 A (surely controversial) idea popped into my head while talking 
 in #d on Freenode. The C++ guys are making an STL2 (the 
 highlight of it being that it is range based). What about 
 taking all the lessons learned from Phobos and creating a 
 Phobos 2? It wouldn't replace the current version. You could 
 import either in one program. It also wouldn't be a radical 
 redesign. Most of Phobos could be used as is. What it would do 
 is allow fixing some hard or impossible problems without losing 
 backward compatibility.

 We could do away with auto-decoding. Design it around using 
 Andrei's allocators throughout. Make the GC optional from the 
 start. Fix a few important naming conflicts and discrepancies.

 There are problems, of course. Rampant code duplication is 
 probably the biggest (though maybe public imports of identical 
 code would help a lot).

 I don't really expect this to go anywhere but I am curious to 
 hear what changes you'd all like to see made to Phobos that 
 can't happen because of backward compatibility. Also, how would 
 you approach doing this? An on disk copy of Phobos with changes 
 would not be an acceptable approach, I think.


Frankly, I do not see the need for Phobos2. If you want to build 
alternative packages, just go ahead and publish them via dub like 
Mir, for example. You can even make a meta package, if you find 
yourself using the same group of packages all the time. Still, 
why would you call that meta package "Phobos2"? It only confuses 
people.

If you want to rewrite parts of the standard library, build the 
alternatives first and then we can adopt them piecewise.

Nevertheless, I would love to read a detailed analysis of Phobos 
and what should be improved. Please, write a blog post somewhere. 
However, do not mention "Phobos2".

D has a painful history with two competing standard libraries. If 
you seriously propose this path, I hope Andrei and Walter will 
publicly and vehemently oppose it. Otherwise that ghost from the 
past becomes a PR disaster for D.
Jun 02 2017
parent reply Brad Anderson <eco gnuk.net> writes:
On Friday, 2 June 2017 at 09:14:14 UTC, qznc wrote:

 Frankly, I do not see the need for Phobos2. If you want to 
 build alternative packages, just go ahead and publish them via 
 dub like Mir, for example. You can even make a meta package, if 
 you find yourself using the same group of packages all the 
 time. Still, why would you call that meta package "Phobos2"? It 
 only confuses people.

 If you want to rewrite parts of the standard library, build the 
 alternatives first and then we can adopt them piecewise.


Makes sense. I've said in the past that the standard library 
should be smaller now that we have Dub. The standard library 
should serve as a common framework that lets libraries 
interoperate easier.

If an effort was undertaken it should probably not be called 
Phobos2 until such time (if ever) that it becomes Phobos2 by 
community vote and Walter and Andrei approval.

C++'s Boost has a problem where libraries people are working on 
get called Boost Whatever in anticipation of being proposed for 
Boost and many of them never even make it to the review stage so 
it becomes confusing about what is and isn't in Boost. It's also 
confusing because competing libraries will be proposed and a 
non-Boost library is already taking Boost ObviousName so they 
have to come up with some weird name instead.


 Nevertheless, I would love to read a detailed analysis of 
 Phobos and what should be improved. Please, write a blog post 
 somewhere. However, do not mention "Phobos2".


This is the heart of why I made the post. I want to read what 
people want myself.


 D has a painful history with two competing standard libraries. 
 If you seriously propose this path, I hope Andrei and Walter 
 will publicly and vehemently oppose it. Otherwise that ghost 
 from the past becomes a PR disaster for D.


Not very seriously. I'm more interested in the discussion than 
anything. Like I said, I don't think this will go anywhere.

I do want to say that I don't think the Phobos/Tango situation is 
relevant. Those were two competing, mutually exclusive standard 
libraries. That problem has long since been resolved with people 
using Tango and Phobos together in their D2 programs just fine. I 
understand why anyone would be weary at the idea though.
Jun 02 2017
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Friday, 2 June 2017 at 09:14:14 UTC, qznc wrote:
[...]

 D has a painful history with two competing standard libraries. If you
 seriously propose this path, I hope Andrei and Walter will publicly
 and vehemently oppose it. Otherwise that ghost from the past becomes a
 PR disaster for D.


[...]

I think this is overreacting.  "Phobos 2" is not supposed to be a
*competing* library, but, as I see it, more like an alpha version of the
next "major iteration" of Phobos. The "next major version" with a brand
new paradigm, if you will, as opposed to incremental changes to the
"current major version".


T

-- 
Give me some fresh salted fish, please.
Jun 02 2017
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2017 01:43 PM, H. S. Teoh via Digitalmars-d wrote:

 On Friday, 2 June 2017 at 09:14:14 UTC, qznc wrote:
 [...]

 D has a painful history with two competing standard libraries. If you
 seriously propose this path, I hope Andrei and Walter will publicly
 and vehemently oppose it. Otherwise that ghost from the past becomes a
 PR disaster for D.


 [...]
 
 I think this is overreacting.  "Phobos 2" is not supposed to be a
 *competing* library, but, as I see it, more like an alpha version of the
 next "major iteration" of Phobos. The "next major version" with a brand
 new paradigm, if you will, as opposed to incremental changes to the
 "current major version".


A clean slate is alluring, and there are several things that can be done 
differently in Phobos, as there are in any project that's been around 
for a while. It may, however, be difficult to find enough people able 
and willing to take such a large project off the ground. There are 
plenty of great things that can be done with the standard library, 
ranging from the trivial - documentation, fixes of bugs triaged as 
"trivial" or "bootcamp" etc - to the most creative.

As an example, eliminating phobos' and druntime's reliance on "static 
this" would improve conviviality of D in mixed-language projects, but we 
have yet to find folks to do the work. To the best of my knowledge only 
David, Stanislav, and myself have been involved so far.


Andrei
Jun 02 2017
next sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Friday, 2 June 2017 at 18:43:32 UTC, Andrei Alexandrescu wrote:


 A clean slate is alluring, and there are several things that 
 can be done differently in Phobos, as there are in any project 
 that's been around for a while. It may, however, be difficult 
 to find enough people able and willing to take such a large 
 project off the ground. There are plenty of great things that 
 can be done with the standard library, ranging from the trivial 
 - documentation, fixes of bugs triaged as "trivial" or 
 "bootcamp" etc - to the most creative.


Indeed, the idea is an inviting one. Fixing mistakes of the past 
may be tedious, but it is a good way of moving forward. I don't 
think such a project actually needs a large amount of full-time 
participants: if not any other reason, it is impractical. We 
can't reasonably hope to have experts on everything in an OSS 
project like this, at least not on a day-to-day basis. A small 
core group inviting PRs and perhaps using a voting system for 
accepting/rejecting features should suffice. The biggest 
challenge is to dilute incoming features into most basic forms, 
simple, efficient and reusable components, which may occasionally 
fall out of the collective expertise.
However, it might be prudent to adopt a waiting stance on this, 
and let several key language features to mature (i.e. the current 
DIP queue). The big problem of Phobos is that it grew together 
with the language, often lagging behind in terms of use of 
language features, while still growing the code base, depending 
on obsolete functionality, and making future fixes harder and 
harder. Repeating this will be a huge mistake, and inevitably 
will lead to talks about Phobos 3 in a few years.
Jun 02 2017
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Jun 02, 2017 at 02:43:32PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]

 A clean slate is alluring, and there are several things that can be done
 differently in Phobos, as there are in any project that's been around for a
 while. It may, however, be difficult to find enough people able and willing
 to take such a large project off the ground. There are plenty of great
 things that can be done with the standard library, ranging from the trivial
 - documentation, fixes of bugs triaged as "trivial" or "bootcamp" etc - to
 the most creative.


Haha, I happened to be reading Spolsky's blog, and came across this:

	https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/

Especially the bit about throwing away years of accumulated programming
work.


T

-- 
Ignorance is bliss... until you suffer the consequences!
Jun 02 2017
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 2 June 2017 at 20:37:05 UTC, H. S. Teoh wrote:

 Especially the bit about throwing away years of accumulated 
 programming work.


Our IRC discussion was more about simplifying the Phobos 
interface through breaking changes than actually just dropping 
and rewriting. Most the "phobos 2" would just be today's phobos 
with mistakes changed, autodecoding deprecated then removed, 
eager functions same deal (the discussion that started it was 
split vs splitter), static array interfaces removed (you'd have 
to explicitly slice at the call site instead of Phobos trying to 
two-layer ref them), etc.

So step one of this would be to copy existing code. Then cut the 
mistakes out.

The modified modules are just given a new package name (or 
something) to ease the transition.
Jun 02 2017
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, Jun 03, 2017 at 02:16:16AM +0000, Adam D. Ruppe via Digitalmars-d wrote:

 On Friday, 2 June 2017 at 20:37:05 UTC, H. S. Teoh wrote:

 Especially the bit about throwing away years of accumulated
 programming work.


 
 Our IRC discussion was more about simplifying the Phobos interface
 through breaking changes than actually just dropping and rewriting.
 Most the "phobos 2" would just be today's phobos with mistakes
 changed, autodecoding deprecated then removed, eager functions same
 deal (the discussion that started it was split vs splitter), static
 array interfaces removed (you'd have to explicitly slice at the call
 site instead of Phobos trying to two-layer ref them), etc.
 
 So step one of this would be to copy existing code. Then cut the
 mistakes out.
 
 The modified modules are just given a new package name (or something)
 to ease the transition.


Now that you put it this way, this suddenly seems a lot less disruptive
and intimidating than it first sounded.

And it sounds more possible to actually use the deprecation cycle to
gradually pull this off, as opposed to throwing out the baby with the
bath water and restarting from ground zero.

And by putting the modified modules in a new temporary namespace, we can
ease the transition enough that perhaps we might actually see this
happen, one day. (Haha, so I hope.)


T

-- 
Some days you win; most days you lose.
Jun 02 2017
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 3 June 2017 at 06:37:27 UTC, H. S. Teoh wrote:

 And by putting the modified modules in a new temporary 
 namespace, we can ease the transition enough that perhaps we 
 might actually see this happen, one day. (Haha, so I hope.)


Javax ?
Jun 03 2017
prev sibling parent aberba <karabutaworld gmail.com> writes:
On Saturday, 3 June 2017 at 06:37:27 UTC, H. S. Teoh wrote:

 On Sat, Jun 03, 2017 at 02:16:16AM +0000, Adam D. Ruppe via 
 Digitalmars-d wrote:

 On Friday, 2 June 2017 at 20:37:05 UTC, H. S. Teoh wrote:

 Especially the bit about throwing away years of accumulated 
 programming work.


 
 Our IRC discussion was more about simplifying the Phobos 
 interface through breaking changes than actually just dropping 
 and rewriting. Most the "phobos 2" would just be today's 
 phobos with mistakes changed, autodecoding deprecated then 
 removed, eager functions same deal (the discussion that 
 started it was split vs splitter), static array interfaces 
 removed (you'd have to explicitly slice at the call site 
 instead of Phobos trying to two-layer ref them), etc.
 
 So step one of this would be to copy existing code. Then cut 
 the mistakes out.
 
 The modified modules are just given a new package name (or 
 something)
 to ease the transition.


 Now that you put it this way, this suddenly seems a lot less 
 disruptive and intimidating than it first sounded.

 And it sounds more possible to actually use the deprecation 
 cycle to gradually pull this off, as opposed to throwing out 
 the baby with the bath water and restarting from ground zero.

 And by putting the modified modules in a new temporary 
 namespace, we can ease the transition enough that perhaps we 
 might actually see this happen, one day. (Haha, so I hope.)


 T


Same will there come to be a need to replace Phobos 2 (when it 
happens) with 3 all because we prefer things to be renamed and 
relocated... when we've not fully discovered what phobos 
currently offers or improvements to be made.

stdx.data.json was not well documented last time I checked. 
Couldn't see any major benefit over std.json besides the 
performance claim. Besides, phobos is an MVP, there are more 
faster dub packages and nothing stops u from using them.

Does it really matter if it belongs in the std namespace?
Jun 03 2017
prev sibling next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Friday, June 02, 2017 13:37:05 H. S. Teoh via Digitalmars-d wrote:

 On Fri, Jun 02, 2017 at 02:43:32PM -0400, Andrei Alexandrescu via
 Digitalmars-d wrote: [...]


 A clean slate is alluring, and there are several things that can be done
 differently in Phobos, as there are in any project that's been around
 for a while. It may, however, be difficult to find enough people able
 and willing to take such a large project off the ground. There are
 plenty of great things that can be done with the standard library,
 ranging from the trivial - documentation, fixes of bugs triaged as
 "trivial" or "bootcamp" etc - to the most creative.


 Haha, I happened to be reading Spolsky's blog, and came across this:

   https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part
 -i/

 Especially the bit about throwing away years of accumulated programming
 work.


I don't entirely agree with his sentiments on that one, but he does have a
point. Throwing the baby out with the bath water is obviously something to
avoid, but sometimes redoing code from scratch does really fix problems -
especially when the original has serious design problems that can't be fixed
without redoing it. However, thinking about what he's saying there, it
starts looking like the most valuable part of your code base is your tests
(assuming that they're thorough), because then if you do completely rewrite
something, you can verify that it's correct.

As for "Phobos 2" specifically though, I have very mixed feelings about such
an idea. Being able to rearrange things how we'd like and fix some of the
mistakes that we've been stuck with would be great, but you're basically
planning on breaking every single D program, even if it's not immediately
and folks could switch over to std2 (or whatever the package was) at a point
that made sense for them. And especially if you're talking about
range-related stuff, you'd be essentially splitting the D community and its
code in two, because you'd almost certainly be dealing with incompatible
libraries - especially if you're actually fixing some of the range-related
problems. And that sort of thing just seems like it would be too destructive
to be worth the risk. It's not quite a language split like you got with
something like python 2 and python 3 or D1 and D2, but if the old and new
standard libraries were not compatible, it would be close to that - it also
seems too much like the whole Phobos vs Tango mess, even if the idea would
be to be transitioning from one to the other rather than really having
competing libraries (but as we've seen with python 2 vs 3 and D1 vs D2,
major transitions like this tend to result in the old and the new competing
even if that's not the intent).

And honestly, for the most part, I don't think that redesigning the standard
library would actually buy us much. Aside from range-related issues,
everything I can think of is essentially a minor annoyance or wort. Sure, it
would be nice if we could fix it, but it's not worth a massive upheaval of
the D ecosystem. The only stuff that I can think of right now that would be
a huge gain if we could change it is range-related stuff. I can think of
several items there that would be nice to fix or at least look at changing:

1. Get rid of auto-decoding ranges and the concept of narrow strings. string
and wstring would be ranges of char and wchar, everything operating on
characters would then be operating at the code unit, code point, or grapheme
level (depending on what was appropriate for the algorithm), and generic
stuff would just operate on the elements regardless of whether they were
code units, code points, or graphemes, leaving it to the programmer to
handle it correctly.

2. Get rid of save and disallow classes as ranges. Postblit constructors
then do what save does now. If you want a class to be a range, then wrap it
in a struct, kind of like a smart pointer but for ranges.

3. Examine how to better deal with input ranges vs forward ranges and look
at requiring that input ranges have reference semantics whereas forward
ranges must have value semantics (as it stands, an input range _without_
reference semantics really should be a forward range, but forward ranges can
still have reference semantics - hence the need for save).

4. Investigate adjusting the range hierarchy so that we have random-access
ranges which aren't input ranges. I don't know for sure that that makes
sense, but as I understand it, Mir does something like that, because it has
a definite performance benefit under some circumstances.

5. Look at whether a car/cdr approach would be better for input ranges (and
maybe any range that isn't random-access) than front/popFront. Andrei wants
to do something with that for immutable ranges, and it could help resolve
the issue of const ranges if that's what we did in general (but without
something like Phobos 2, I personally think that such a change is too
drastic even if Andrei is currently thinking about it for immutable
containers).

6. Look at improving output ranges so that you actually know whether
something is going to fit in them - possibly by having a type of output
range that has a length or some other member indicating the space remaining.

7. Look at formalizing what std.digest did with output ranges and have a way
to indicate that you're done putting stuff in them so that they can do
something and "close" or "complete" the range.

And there are probably more things that we could look at with ranges, which
would be nice to fix. So, being able to go back to the drawing board on some
level with ranges would be _fantastic_, but to do so would break most
everything, and aside from ranges, I really don't see much gain in a "Phobos
2". Aside from nitpicks with names and whatnot, it should be possible to
make most changes without doing a full, new iteration of the standard
library.

And as Daniel Murphy pointed out several times at dconf, if you really step
back and just use D as it is, it's pretty darn fantastic, and there's too
much going with folks trying to make it perfect. Yes, D and Phobos have
warts. Yes, it would be nice to make some improvements which would not be
backwards compatible. But what we have is already pretty amazing, and when
fixing something requires pulling the rug out from _everyone_ using D, you
have to really consider that it's better having something amazing but
imperfect than to try and fix it. Yes, sometimes, drastic changes are worth
the breakage - especially when most of the required changes actually results
in bugs being fixed - but redoing the standard library? That really seems
like too big a change.

- Jonathan M Davis
Jun 02 2017
parent 9il <ilyayaroshenko gmail.com> writes:
On Friday, 2 June 2017 at 21:59:42 UTC, Jonathan M Davis wrote:

 On Friday, June 02, 2017 13:37:05 H. S. Teoh via Digitalmars-d 
 wrote:

 On Fri, Jun 02, 2017 at 02:43:32PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]


 A clean slate is alluring, and there are several things that 
 can be done differently in Phobos, as there are in any 
 project that's been around for a while. It may, however, be 
 difficult to find enough people able and willing to take 
 such a large project off the ground. There are plenty of 
 great things that can be done with the standard library, 
 ranging from the trivial - documentation, fixes of bugs 
 triaged as "trivial" or "bootcamp" etc - to the most 
 creative.


 Haha, I happened to be reading Spolsky's blog, and came across 
 this:

   
 https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part
 -i/

 Especially the bit about throwing away years of accumulated 
 programming work.


 I don't entirely agree with his sentiments on that one, but he 
 does have a point. Throwing the baby out with the bath water is 
 obviously something to avoid, but sometimes redoing code from 
 scratch does really fix problems - especially when the original 
 has serious design problems that can't be fixed without redoing 
 it. However, thinking about what he's saying there, it starts 
 looking like the most valuable part of your code base is your 
 tests (assuming that they're thorough), because then if you do 
 completely rewrite something, you can verify that it's correct.

 As for "Phobos 2" specifically though, I have very mixed 
 feelings about such an idea. Being able to rearrange things how 
 we'd like and fix some of the mistakes that we've been stuck 
 with would be great, but you're basically planning on breaking 
 every single D program, even if it's not immediately and folks 
 could switch over to std2 (or whatever the package was) at a 
 point that made sense for them. And especially if you're 
 talking about range-related stuff, you'd be essentially 
 splitting the D community and its code in two, because you'd 
 almost certainly be dealing with incompatible libraries - 
 especially if you're actually fixing some of the range-related 
 problems. And that sort of thing just seems like it would be 
 too destructive to be worth the risk. It's not quite a language 
 split like you got with something like python 2 and python 3 or 
 D1 and D2, but if the old and new standard libraries were not 
 compatible, it would be close to that - it also seems too much 
 like the whole Phobos vs Tango mess, even if the idea would be 
 to be transitioning from one to the other rather than really 
 having competing libraries (but as we've seen with python 2 vs 
 3 and D1 vs D2, major transitions like this tend to result in 
 the old and the new competing even if that's not the intent).

 And honestly, for the most part, I don't think that redesigning 
 the standard library would actually buy us much. Aside from 
 range-related issues, everything I can think of is essentially 
 a minor annoyance or wort. Sure, it would be nice if we could 
 fix it, but it's not worth a massive upheaval of the D 
 ecosystem. The only stuff that I can think of right now that 
 would be a huge gain if we could change it is range-related 
 stuff. I can think of several items there that would be nice to 
 fix or at least look at changing:

 1. Get rid of auto-decoding ranges and the concept of narrow 
 strings. string and wstring would be ranges of char and wchar, 
 everything operating on characters would then be operating at 
 the code unit, code point, or grapheme level (depending on what 
 was appropriate for the algorithm), and generic stuff would 
 just operate on the elements regardless of whether they were 
 code units, code points, or graphemes, leaving it to the 
 programmer to handle it correctly.

 2. Get rid of save and disallow classes as ranges. Postblit 
 constructors then do what save does now. If you want a class to 
 be a range, then wrap it in a struct, kind of like a smart 
 pointer but for ranges.

 3. Examine how to better deal with input ranges vs forward 
 ranges and look at requiring that input ranges have reference 
 semantics whereas forward ranges must have value semantics (as 
 it stands, an input range _without_ reference semantics really 
 should be a forward range, but forward ranges can still have 
 reference semantics - hence the need for save).

 4. Investigate adjusting the range hierarchy so that we have 
 random-access ranges which aren't input ranges. I don't know 
 for sure that that makes sense, but as I understand it, Mir 
 does something like that, because it has a definite performance 
 benefit under some circumstances.


Mir random access ranges (ndslices) are input ranges. The 
performance benefit comes from the way how they are implemented, 
constructed, and iterated.

Also, ndslice can be created in top of a range that have only one 
primitive opIndex. A user can define his own ndslices with all 
possible range primitives very fast - define a struct with 
opIndex method and use sliceField to construct ndslice on top of 
it.

Ilya
Jun 02 2017
prev sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Friday, June 02, 2017 14:43:32 Andrei Alexandrescu via Digitalmars-d 
wrote:

 On 06/02/2017 01:43 PM, H. S. Teoh via Digitalmars-d wrote:

 On Friday, 2 June 2017 at 09:14:14 UTC, qznc wrote:
 [...]


 D has a painful history with two competing standard libraries. If you
 seriously propose this path, I hope Andrei and Walter will publicly
 and vehemently oppose it. Otherwise that ghost from the past becomes a
 PR disaster for D.


 [...]

 I think this is overreacting.  "Phobos 2" is not supposed to be a
 *competing* library, but, as I see it, more like an alpha version of the
 next "major iteration" of Phobos. The "next major version" with a brand
 new paradigm, if you will, as opposed to incremental changes to the
 "current major version".


 A clean slate is alluring, and there are several things that can be done
 differently in Phobos, as there are in any project that's been around
 for a while. It may, however, be difficult to find enough people able
 and willing to take such a large project off the ground. There are
 plenty of great things that can be done with the standard library,
 ranging from the trivial - documentation, fixes of bugs triaged as
 "trivial" or "bootcamp" etc - to the most creative.

 As an example, eliminating phobos' and druntime's reliance on "static
 this" would improve conviviality of D in mixed-language projects, but we
 have yet to find folks to do the work. To the best of my knowledge only
 David, Stanislav, and myself have been involved so far.


A related point is what's happened with modules like std.json and std.xml.
We've said for years that they need to be replaced. But have they been? No.
Some work has been done with the idea that it would replace them, but either
it's never been done, or there was too much arguing over the replacement and
what it should do for it to get through the review process. We can't replace
these two modules, and we're now talking about the possibility of redoing
the entire standard library? When you think of it that way, it seems insane,
even if it might be desirable.

Sure, if we were going to redo Phobos, most of it would probably just be
transferred to "Phobos 2" but rearranged as appropriate rather than
completely redone, but stuff like revamping ranges would affect almost
everything in Phobos, making that a _major_ task even without considering
how it would affect all of the D code that isn't in Phobos but uses it. I
think that we've pretty much proven that we can't do that with the current
level of manpower.

And of course, doing stuff like adjusting Phobos to mitigate the effect of
auto-decoding or to revamp function signatures such that most overloads are
done internally with static ifs where possible - or the static constructor
issue - all are smaller things that we have trouble getting done even when
we mostly agree that they should be done. We do get some work done, to be
sure, but we need more eyes on pull requests so that they get merged in a
timely manner and more pull requests to fix these smaller issues. If we
can't even do those right, how could we possibly revamp all of the standard
library?

I can certainly empathize with the sentiment that it would be nice to go
back to the drawing board on some things, but I think that doing that with
the whole standard library is just too much - both from the standpoint of
manpower and with regards to how much code it would ultimately break.

- Jonathan M Davis
Jun 02 2017