www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - no-autodecode -- game plan

reply Steven Schveighoffer <schveiguy gmail.com> writes:
Hi all,

TL;DR: I'm working on removing autodecoding from phobos. I have the 
first real PR: https://github.com/dlang/phobos/pull/7595


So what is the plan here? I have added hooks for dlang's CI so that any 
work we do on making phobos work without autodecoding does not go back 
to being broken.

As expected, I'm finding bugs. But I have probably the most relevant 
module passing Phobos unittests without autodecode: std.utf.

My plan is to pick a module or group of modules each week and spend some 
time getting it to pass.

Help is appreciated! So what it the process, you ask? I hope this is 
simple enough for everyone to follow:

1. set up your phobos dev environment (for now, I'm doing everything in 
the posix makefile, so it cannot be done via Windows -- if you want to 
fix this, please do!)
2. build phobos with no autodecoding:
NO_AUTODECODE=1 make -f posix.mak
3a. If you want to test a specific module:

NO_AUTODECODE=1 make -f posix.mak std/somemodule.test

3b. If you want to test all modules (and find one that breaks you can 
work on):

NO_AUTODECODE=1 make -f posix.mak unittest

4. Fix it so it passes. It should pass both with and without 
autodecoding. Try not to disable specific tests to get it to pass, we 
want no-autodecode phobos to be usable! Instead, change them to either 
not depend on autodecoding [1]  (preferred) or add a static if that 
tests both modes [2].

5. Add the modules that now pass to the posix.mak variable named 
NO_AUTODECODE_MODULES. Once this is merged, any changes to that module 
should have to pass without autodecoding.

The autodecode test is run in buildkite. PLEASE don't merge any PRs for 
Phobos if this test is failing! instead, ping me.

If you want to work on this with me, please let's coordinate either on 
slack or you can post here. We shouldn't be duplicating efforts (as much 
as possible), and Phobos is pretty big.

-Steve

[1] examples of places where I changed the test to be autodecode agnostic:
https://github.com/dlang/phobos/pull/7595/files#diff-27410a0be96392a06647e61c73131b64L2401-R2402
https://github.com/dlang/phobos/pull/7595/files#diff-27410a0be96392a06647e61c73131b64L3375-R3379

[2] examples of places where I configured tests to test both autodecode 
and non-autodecode:
https://github.com/dlang/phobos/pull/7595/files#diff-27410a0be96392a06647e61c73131b64L3619-R3639
Aug 16 2020
next sibling parent reply aberba <karabutaworld gmail.com> writes:
On Sunday, 16 August 2020 at 17:00:52 UTC, Steven Schveighoffer 
wrote:

 Hi all,

 TL;DR: I'm working on removing autodecoding from phobos. I have 
 the first real PR: https://github.com/dlang/phobos/pull/7595

 [...]


Awesome!
Aug 16 2020
parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Sunday, 16 August 2020 at 20:41:09 UTC, aberba wrote:

 Awesome!


Agreed!
Aug 16 2020
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/16/2020 10:00 AM, Steven Schveighoffer wrote:

 TL;DR: I'm working on removing autodecoding from phobos.


Yay!
Aug 16 2020
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 8/16/20 1:00 PM, Steven Schveighoffer wrote:

 Hi all,
 
 TL;DR: I'm working on removing autodecoding from phobos. I have the 
 first real PR: https://github.com/dlang/phobos/pull/7595


Why not do it in std.v2021.*?
Aug 18 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/18/20 6:21 PM, Andrei Alexandrescu wrote:

 On 8/16/20 1:00 PM, Steven Schveighoffer wrote:

 Hi all,

 TL;DR: I'm working on removing autodecoding from phobos. I have the 
 first real PR: https://github.com/dlang/phobos/pull/7595


 
 Why not do it in std.v2021.*?
 


What do you mean? Have a complete alternative phobos library?

autodecoding infects so many other modules, so it would be really hard 
to single out only some modules, we'd likely have to make a copy of 
nearly everything.
Aug 19 2020
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 19 August 2020 at 12:23:54 UTC, Steven 
Schveighoffer wrote:

 On 8/18/20 6:21 PM, Andrei Alexandrescu wrote:

 On 8/16/20 1:00 PM, Steven Schveighoffer wrote:

 Hi all,

 TL;DR: I'm working on removing autodecoding from phobos. I 
 have the first real PR: 
 https://github.com/dlang/phobos/pull/7595


 
 Why not do it in std.v2021.*?
 


 What do you mean? Have a complete alternative phobos library?

 autodecoding infects so many other modules, so it would be 
 really hard to single out only some modules, we'd likely have 
 to make a copy of nearly everything.


I do believe he is joking .... but maybe he's not?
Aug 19 2020
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 19 August 2020 at 12:23:54 UTC, Steven 
Schveighoffer wrote:

 On 8/18/20 6:21 PM, Andrei Alexandrescu wrote:

 Why not do it in std.v2021.*?
 


 What do you mean? Have a complete alternative phobos library?

 autodecoding infects so many other modules, so it would be 
 really hard to single out only some modules, we'd likely have 
 to make a copy of nearly everything.


Andrei's suggestion is a good idea to allow forward massive 
changes without breaking any existing code.

The only regret is we didn't think to version it like this from 
the start.

A copy of everything is an acceptable cost.
Aug 19 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 19 August 2020 at 12:40:59 UTC, Adam D. Ruppe wrote:


 A copy of everything is an acceptable cost.


That's a rather unique position.
The only way that would work properly is to deprecate the non 
v2021 namespace.
And not develop it anymore.

Otherwise it'll diverge more and more.
Aug 19 2020
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 19 August 2020 at 12:56:33 UTC, Stefan Koch wrote:

 That's a rather unique position.


It is a major version bump to indicate a split in backward 
compatibility. You'd treat it just like that in any other 
context; the module name just includes the x in semver's x.y.z.

This beats using a git branch because both can exist 
simultaneously. Suppose you import library old that uses the 
autodecoding version but you want to use library new that 
doesn't. No problem, both co-exist now.

Managing it can be tricky though, perhaps you would use a git 
branch internally to distribute and cherry-pick patches but then 
the release includes all the supported versions for simultaneous 
import. Though I suspect if we don't make a new one until the old 
one is frozen it won't be that much work anyway (at least if the 
compiler stops breaking code).

That's why it is v2021, indicating it is the year 2021 release. 
It gets frozen at some point in the year and then other breaking 
changes are put in v2022 which carries on the tradition.
Aug 19 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/19/20 8:40 AM, Adam D. Ruppe wrote:

 On Wednesday, 19 August 2020 at 12:23:54 UTC, Steven Schveighoffer wrote:

 On 8/18/20 6:21 PM, Andrei Alexandrescu wrote:

 Why not do it in std.v2021.*?


 What do you mean? Have a complete alternative phobos library?

 autodecoding infects so many other modules, so it would be really hard 
 to single out only some modules, we'd likely have to make a copy of 
 nearly everything.


 
 Andrei's suggestion is a good idea to allow forward massive changes 
 without breaking any existing code.


I think it's an OK idea. But I don't want to wait forever for this to 
gel. And these PRs are not going to break code, it shouldn't affect 
anything, since I'm not disabling autodecoding at all except in a 
separate test.

When we get to the point where all of Phobos can be built without 
autodecoding, then we have to make the decision how to proceed with the 
breakage.


 The only regret is we didn't think to version it like this from the start.
 
 A copy of everything is an acceptable cost.


The copy of everything is not acceptable to me. I had to change 
something like 53 lines of std.utf. That's about 1.5% of the 4k+ line file.

If we have to maintain 2 copies of std.utf, most of which are EXACTLY 
the same, it would be a nightmare. And that's one module.

The only way this works is if we split the files at the point of release 
(when no-autodecoding is completely ready), and only backport bug fixes 
to the prior version.

In this sense, it's better IMO to do this based on semantic versioning 
and git branches rather than copies of files.

-Steve
Aug 19 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 8/19/20 9:06 AM, Steven Schveighoffer wrote:

 When we get to the point where all of Phobos can be built without 
 autodecoding, then we have to make the decision how to proceed with the 
 breakage.


That doesn't seem like a good gameplan to me because it's only THE 


If you cut phobos open, do surgery after surgery on it, and only then 
think of how you're going to close the patient, you'll get some 
Frankenstein monster as a result.

STOP CHANGING.

START ADDING.
Aug 19 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 8/19/20 8:40 AM, Adam D. Ruppe wrote:

 On Wednesday, 19 August 2020 at 12:23:54 UTC, Steven Schveighoffer wrote:

 On 8/18/20 6:21 PM, Andrei Alexandrescu wrote:

 Why not do it in std.v2021.*?


 What do you mean? Have a complete alternative phobos library?

 autodecoding infects so many other modules, so it would be really hard 
 to single out only some modules, we'd likely have to make a copy of 
 nearly everything.


 
 Andrei's suggestion is a good idea to allow forward massive changes 
 without breaking any existing code.
 
 The only regret is we didn't think to version it like this from the start.
 
 A copy of everything is an acceptable cost.


Don't copy everything, alias everything that doesn't need change and 
only add the declarations that should be changed.

The only problem I see with this is documentation. We need to have an 
easy way to define and navigate documentation for different versions of 
the stdlib.

That article by Yegge once more convinced me - we need to stop (thinking 
of) changing and start adding. We've been paralyzed by the fear of 
change while the opportunity (made even better by a nice module system) 
was always there.
Aug 19 2020
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 19 August 2020 at 13:10:29 UTC, Andrei Alexandrescu 
wrote:

 Don't copy everything, alias everything that doesn't need 
 change and only add the declarations that should be changed.


Yes, that too.


 The only problem I see with this is documentation. We need to 
 have an easy way to define and navigate documentation for 
 different versions of the stdlib.


My dpldocs.info can already pull any specific version you want 
from git:

http://phobos.dpldocs.info/v2.076.0/std.algorithm.html

just a fun fact. I've kinda wanted it to show automatic change 
logs in the past but haven't gotten around to that yet...

But anyway, with aliases, they'd be hyperlinked automatically (at 
least by my doc gen) and with different major version package 
namespaces, all would be available naturally.

I'm convinced we could make it work, if it doesn't already, it 
shouldn't be insurmountable tweaks to the doc generators.


 That article by Yegge once more convinced me - we need to stop 
 (thinking of) changing and start adding. We've been paralyzed 
 by the fear of change while the opportunity (made even better 
 by a nice module system) was always there.


Glad to see others finally seeing this! Back before dub came out 
I argued we should use the module system for this and found no 
traction. I even gave up on doing it myself (but instead I simply 
don't break my code - the arsd library has had a total of 8 
breaking changes going back a decade across all 70+ of its 
modules and most of them are trivial).

But it still remains on my mind as a viable thing to explore in 
more detail someday.
Aug 19 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 8/19/20 9:21 AM, Adam D. Ruppe wrote:

 On Wednesday, 19 August 2020 at 13:10:29 UTC, Andrei Alexandrescu wrote:

 Don't copy everything, alias everything that doesn't need change and 
 only add the declarations that should be changed.


 
 Yes, that too.
 

 The only problem I see with this is documentation. We need to have an 
 easy way to define and navigate documentation for different versions 
 of the stdlib.


 
 My dpldocs.info can already pull any specific version you want from git:
 
 http://phobos.dpldocs.info/v2.076.0/std.algorithm.html
 
 just a fun fact. I've kinda wanted it to show automatic change logs in 
 the past but haven't gotten around to that yet...
 
 But anyway, with aliases, they'd be hyperlinked automatically (at least 
 by my doc gen) and with different major version package namespaces, all 
 would be available naturally.
 
 I'm convinced we could make it work, if it doesn't already, it shouldn't 
 be insurmountable tweaks to the doc generators.
 

 That article by Yegge once more convinced me - we need to stop 
 (thinking of) changing and start adding. We've been paralyzed by the 
 fear of change while the opportunity (made even better by a nice 
 module system) was always there.


 
 Glad to see others finally seeing this! Back before dub came out I 
 argued we should use the module system for this and found no traction. I 
 even gave up on doing it myself (but instead I simply don't break my 
 code - the arsd library has had a total of 8 breaking changes going back 
 a decade across all 70+ of its modules and most of them are trivial).
 
 But it still remains on my mind as a viable thing to explore in more 
 detail someday.


I thought of adding: "Cue plug of adrdox" to my message, but I knew it 
was unnecessary :o).

Versioning stdlib might require significant changes to ddoc or an 
opportunity for adrdox to differentiate itself.
Aug 19 2020
prev sibling next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/19/20 9:10 AM, Andrei Alexandrescu wrote:

 On 8/19/20 8:40 AM, Adam D. Ruppe wrote:

 On Wednesday, 19 August 2020 at 12:23:54 UTC, Steven Schveighoffer wrote:

 On 8/18/20 6:21 PM, Andrei Alexandrescu wrote:

 Why not do it in std.v2021.*?


 What do you mean? Have a complete alternative phobos library?

 autodecoding infects so many other modules, so it would be really 
 hard to single out only some modules, we'd likely have to make a copy 
 of nearly everything.


 Andrei's suggestion is a good idea to allow forward massive changes 
 without breaking any existing code.

 The only regret is we didn't think to version it like this from the 
 start.

 A copy of everything is an acceptable cost.


 
 Don't copy everything, alias everything that doesn't need change and 
 only add the declarations that should be changed.


I've done exactly this for another library: converting mysql-native to 
 safe. See the result: https://github.com/mysql-d/mysql-native/pull/214. 
It's not as simple as you say. I had to split out the common parts into 
an "impl" package, and then create "safe" and "unsafe" packages. Not 
only that, but the unsafe ones are going to suffer from performance 
because everything underneath has to be safe. And this is a pretty 
unchanging library, not like Phobos.

Consider std.algorithm. Nearly all the functions are going to have to 
import a separate std.range, to get the no-autodecoding functionality 
(i.e. anything that depends on hasLength, hasIndexing, etc.). But the 
functions themselves will be identical. So it will be mostly a copy, 
with a few aliases.

I don't think this idea works without copying a large majority of code, 
whereas versioning it means changing very little.


 The only problem I see with this is documentation. We need to have an 
 easy way to define and navigate documentation for different versions of 
 the stdlib.


That is also a problem that I faced in mysql-native. I don't have a good 
solution.


 
 That article by Yegge once more convinced me - we need to stop (thinking 
 of) changing and start adding. We've been paralyzed by the fear of 
 change while the opportunity (made even better by a nice module system) 
 was always there.


The context of that article is somewhat similar, but not completely. 
With google services there is no using a previous version of their cloud 
service if they take it away. With the D compiler, you can alaways 
compile with previous versions until you can handle the new version. We 
aren't taking away the downloads of previous versions of the compiler. 
In fact, we can maintain older versions if we want to!

I think the better answer is to move to semantic versioning of the 
language, and have a scheduled "breaking" upgrade once a year (or 
longer), and commit to maintaining bug fixes for the previous major 
revision for a period of time. Give people the option of using older 
code if they want to keep some old package compiling and running. If 
they are actively maintaining it, then they likely need to upgrade.

-Steve
Aug 19 2020
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 19.08.20 15:10, Andrei Alexandrescu wrote:


 
 Don't copy everything, alias everything that doesn't need change and 
 only add the declarations that should be changed.


But which declarations are those? Aliasing declarations that would 
otherwise be copied verbatim does not work. Consider:

module a;

auto front(R)(R r)if(...){ /* autodecode here */ }
auto algorithm(R)(R r)if(...){ ... r.front; ... }

module b;

auto front(R)(R r)if(...){ /* no autodecode here */
static import a;
alias algorithm=a.algorithm;

This will not do what you want, because aliasing is not equivalent to 
copying.



 Aug 19 2020



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.com>  writes:


On 8/19/20 10:27 AM, Timon Gehr wrote:

 On 19.08.20 15:10, Andrei Alexandrescu wrote:


 Don't copy everything, alias everything that doesn't need change and 
 only add the declarations that should be changed.


 
 But which declarations are those? Aliasing declarations that would 
 otherwise be copied verbatim does not work. Consider:
 
 module a;
 
 auto front(R)(R r)if(...){ /* autodecode here */ }
 auto algorithm(R)(R r)if(...){ ... r.front; ... }
 
 module b;
 
 auto front(R)(R r)if(...){ /* no autodecode here */
 static import a;
 alias algorithm=a.algorithm;
 
 This will not do what you want, because aliasing is not equivalent to 
 copying.


Of course, that was implied. The difficulty is to figure whether said 
algorithm (and others) would be affected by autodecoding, even if it 
doesn't in and of itself do anything about it specifically. Those that 
are would need to be forked across the two versions. Ideally instead of 
copying them, they'd be separated to minimize duplication.



 Aug 19 2020



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com>  writes:


On 8/19/20 8:23 AM, Steven Schveighoffer wrote:

 On 8/18/20 6:21 PM, Andrei Alexandrescu wrote:

 On 8/16/20 1:00 PM, Steven Schveighoffer wrote:

 Hi all,

 TL;DR: I'm working on removing autodecoding from phobos. I have the 
 first real PR: https://github.com/dlang/phobos/pull/7595


 Why not do it in std.v2021.*?


 
 What do you mean? Have a complete alternative phobos library?


We discussed this here a couple of times before - just add a new package 
v2021 in std. Inside it, alias all declarations that you don't plan to 
change to the corresponding symbols in std, then add declarations for 
the things you do want to change.

It's like patching - you only add the deltas. It's simple and easy to 
set up.


 autodecoding infects so many other modules, so it would be really hard 
 to single out only some modules, we'd likely have to make a copy of 
 nearly everything.


I wouldn't be so sure. I think there are large swaths of the stdlib that 
don't do anything with autodecoding. Like, threading stuff, parallel 
stuff, file stuff, numeric stuff, conversion stuff (sans strings), most 
of datetime stuff, and probably a ton more. I know there's likely to 
find some autodecoding parts in the above but the point is they are not 
about strings or autodecoding.

Even those that do can be arranged so you only write deltas. Let me pick 
one at random:

// Current code
auto cmp(R1, R2)(R1 r1, R2 r2)
if (isInputRange!R1 && isInputRange!R2)
{
     ... somewhere inside ...
     static if (isDynamicArray!R1 && isDynamicArray!R2
         && __traits(isUnsigned, E1) && __traits(isUnsigned, E2)
         && E1.sizeof == 1 && E2.sizeof == 1
         // Both or neither must auto-decode.
         && (is(immutable E1 == immutable char) == is(immutable E2 == 
immutable char)))
     {
         ... special case arrays of ubyte/char ...
     }
     else static if (!(isSomeString!R1 && isSomeString!R2))
     {
         ... not influenced by autodecoding ...
     }
     else
     {
         ... autodecode stuff ...
     }
}


// New code in std.v2021
auto cmp(R1, R2)(R1 r1, R2 r2)
if (isInputRange!R1 && isInputRange!R2)
{
     enum bool influencedByAutodecode = ...;
     static if (!influencedByAutodecode)
         return std.cmp(r1, r2);
     else
     {
         ... patch ...
     }
}

Even the patch can reuse std.cmp if you pass std.cmp a range of code 
units so you undo its penchant to autodecode.

* * *

Lastly, I should add the following. You wrote, and I quote it again 
because it is so important:


 autodecoding infects so many other modules, so it would be really hard 
 to single out only some modules, we'd likely have to make a copy of 
 nearly everything.


This is a very powerful argument AGAINST your current approach to CHANGE 
things. You are basically saying the semantics of the standard library 
will be deeply affected by your work. Put another way, the rework of the 
standard library will be very incompatible with its current version. So 
you are killing all compatibility.

We don't want another Python 3 vs. Python 2. It almost killed Python, 
and they could afford to lose a lot more than we do. And let's face it - 
nobody is singing an ode to Python 3 now that they have Unicode built 
in. They solved no real problem and in the best of cases they made it 
marginally easy to do work that was very well done by a couple of 
specialized libraries.

Changing the stdlib to remove autodecoding would kill the D programming 
language.

You don't want to kill the D programming language, do you?

Stop changing.

Start adding.



 Aug 19 2020



 next sibling parent reply Steven Schveighoffer <schveiguy gmail.com>  writes:


On 8/19/20 9:35 AM, Andrei Alexandrescu wrote:

 Even the patch can reuse std.cmp if you pass std.cmp a range of code 
 units so you undo its penchant to autodecode.


Here's what I actually did do:

https://github.com/dlang/phobos/blob/bf258e56f2dbc28cf810e9329a0696afbe114907/std/algorithm/comparison.d#L890-L895

Essentially, if you compared 2 ranges of code units which differed, 
equal would integer promote one to the other, and compare that.

I fixed that bug, and in the process, handled both autodecoding and 
nonautodecoding strings.

This brings "equal" outside of the domain of having to think about 
autodecoding. If you have autodecoding enabled, it works. If you have it 
disabled, it works. There is no need to have a separate equal function 
in std.v2021. There is no need to call a separate equal function if you 
want no-autodecoding, equal(someString, someOtherString) works no matter 
what you care about autodecoding.

The more of this that we can do, the smaller the "drastic" change of 
moving to non-autodecoding becomes.


 
 * * *
 
 Lastly, I should add the following. You wrote, and I quote it again 
 because it is so important:
 

 autodecoding infects so many other modules, so it would be really hard 
 to single out only some modules, we'd likely have to make a copy of 
 nearly everything.


 
 This is a very powerful argument AGAINST your current approach to CHANGE 
 things. You are basically saying the semantics of the standard library 
 will be deeply affected by your work. Put another way, the rework of the 
 standard library will be very incompatible with its current version. So 
 you are killing all compatibility.


Nearly ALL the changes are going to be on template constraints or 
imported traits, which means the code in the alternative package will be 
identical (but maintained separately). The end result would be a 
maintenance nightmare. Maintaining almost identical code bases, 
especially where one is used and the other isn't is just going to result 
in diverging code bases where the new one doesn't work and has to be 
fixed. See for instance how Daniel Murphy migrated the compiler from C++ 
to D without having to worry about other people breaking his changes. It 
would not have happened with a copy of everything hand-maintained.

My plea is: let's see how far I can get with an alternative view of 
Phobos (tested simultaneously), without actually pulling the trigger to 
deprecate autodecoding. At that point, we still can make a decision to 
do it a different way. Nothing is set in stone.


 Changing the stdlib to remove autodecoding would kill the D programming 
 language.
 
 You don't want to kill the D programming language, do you?


I can't agree with either of these statements. At all.

-Steve



 Aug 19 2020



 next sibling parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.com>  writes:


On 8/19/20 10:12 AM, Steven Schveighoffer wrote:

 My plea is: let's see how far I can get with an alternative view of 
 Phobos (tested simultaneously), without actually pulling the trigger to 
 deprecate autodecoding. At that point, we still can make a decision to 
 do it a different way. Nothing is set in stone.


Good. Virtually all I worry about applies to after that point. Making 
mnst of the code decoding-neutral is a good endeavor.



 Aug 19 2020



prev sibling  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.com>  writes:


On 8/19/20 10:12 AM, Steven Schveighoffer wrote:

 On 8/19/20 9:35 AM, Andrei Alexandrescu wrote:

 Even the patch can reuse std.cmp if you pass std.cmp a range of code 
 units so you undo its penchant to autodecode.


 
 Here's what I actually did do:
 
 https://github.com/dlang/phobos/blob/bf258e56f2dbc28cf810e9329a0696afbe114907/std/algorithm/co
parison.d#L890-L895 


I have a number of strong objections to that, starting with the obvious 
fact that you add an undocumented symbol that the user interface then 
depends on. But that's not the main problem:

https://github.com/dlang/phobos/pull/7595#pullrequestreview-471042128

The main problem is we're butchering the entire standard library based 
on a `version` thing that essentially forks the entire library. THE 
ENTIRE LIBRARY. To change its semantics in ever so daintly ways. I have 
yet to see this anywhere else, ever. And THEN that stupid version flag 
gives carte blanche to anyone with a scalpel to keep on butchering, like 
this PR copiusly does:

https://github.com/dlang/phobos/pull/7595

This is a hack job. The version idea is awful. I told so Walter when he 
introduced it, he wouldn't listen.

Literally this is why we can't have good things.

IN THE NAME OF EVERYTHING GOOD, PLEASE STOP CHANGING.

PLEASE START ADDING.


 I fixed that bug, and in the process, handled both autodecoding and 
 nonautodecoding strings.
 
 This brings "equal" outside of the domain of having to think about 
 autodecoding. If you have autodecoding enabled, it works. If you have it 
 disabled, it works. There is no need to have a separate equal function 
 in std.v2021. There is no need to call a separate equal function if you 
 want no-autodecoding, equal(someString, someOtherString) works no matter 
 what you care about autodecoding.
 
 The more of this that we can do, the smaller the "drastic" change of 
 moving to non-autodecoding becomes.


This sounded nice this morning when I read it. Then during the day I had 
this growing feeling that the reality is much worse than that. I went to

https://github.com/dlang/phobos/pull/7595

which confirms my worst fear. This is a hack job that will butcher the 
standard library beyond recognition.

IN THE NAME OF EVERYTHING GOOD, PLEASE STOP CHANGING.

PLEASE START ADDING.



 Aug 19 2020



prev sibling  parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx>  writes:


On Wed, Aug 19, 2020 at 09:35:34AM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 8/19/20 8:23 AM, Steven Schveighoffer wrote:


[...]

 autodecoding infects so many other modules, so it would be really
 hard to single out only some modules, we'd likely have to make a
 copy of nearly everything.


 
 This is a very powerful argument AGAINST your current approach to
 CHANGE things. You are basically saying the semantics of the standard
 library will be deeply affected by your work. Put another way, the
 rework of the standard library will be very incompatible with its
 current version. So you are killing all compatibility.


[...]

Whoa, hold your guns there!  We're not actually changing semantics --
yet.  Steven's changes are controlled by version directives; even after
all changes have been made *current semantics remain unchanged*. Unless
you deliberately compile with the no-autodecoding version set. Which
users won't have to.

When we get to the point where the entire Phobos can be compiled with
no-autodecoding, *then* we can gauge the true extent of breakage (by
compiling public D repos with autodecoding on -- *experimentally*) to
obtain some REAL DATA on which to base decisions, instead of gut
feelings and other totally subjective criteria, and thereby be in a much
better position to make a sane decision about where to go from there.
If we absolutely do not want to break backward compatibility, then *at
that time* we can freeze the current std.*, make a copy of everything in
std.v2021, turn on no-autodecoding for the latter, and go from there.

Please let's not let the best be the enemy of the good yet again.
Autodecoding has been our scourge for how many years now, and now that
somebody's FINALLY making progress on it, please let's not gun it down
prematurely.


T

-- 
What's an anagram of "BANACH-TARSKI"?  BANACH-TARSKI BANACH-TARSKI.



 Aug 19 2020



 next sibling parent  aberba <karabutaworld gmail.com>  writes:


On Wednesday, 19 August 2020 at 15:39:58 UTC, H. S. Teoh wrote:

 On Wed, Aug 19, 2020 at 09:35:34AM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote:

 On 8/19/20 8:23 AM, Steven Schveighoffer wrote:


 [...]



 When we get to the point where the entire Phobos can be 
 compiled with no-autodecoding, *then* we can gauge the true 
 extent of breakage (by compiling public D repos with 
 autodecoding on -- *experimentally*) to obtain some REAL DATA 
 on which to base decisions, instead of gut feelings and other 
 totally subjective criteria, and thereby be in a much better 
 position to make a sane decision about where to go from there.


+ 1


 If we absolutely do not want to break backward compatibility, 
 then *at that time* we can freeze the current std.*, make a 
 copy of everything in std.v2021, turn on no-autodecoding for 
 the latter, and go from there.


At that time, it might not be a bad idea either. Eventually D 
will need something like this to ensure it's not held behind by 
future mistakes (which definitely will happen)...as long as it 
can be corrected in the next version of std. Guess we'll see when 
that time to comes.



 Aug 19 2020



prev sibling  parent  sarn <sarn theartofmachinery.com>  writes:


On Wednesday, 19 August 2020 at 15:39:58 UTC, H. S. Teoh wrote:

 On Wed, Aug 19, 2020 at 09:35:34AM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote:

 This is a very powerful argument AGAINST your current approach 
 to CHANGE things. You are basically saying the semantics of 
 the standard library will be deeply affected by your work. Put 
 another way, the rework of the standard library will be very 
 incompatible with its current version. So you are killing all 
 compatibility.


 [...]
 When we get to the point where the entire Phobos can be 
 compiled with no-autodecoding, *then* we can gauge the true 
 extent of breakage (by compiling public D repos with 
 autodecoding on -- *experimentally*) to obtain some REAL DATA 
 on which to base decisions, instead of gut feelings and other 
 totally subjective criteria, and thereby be in a much better 
 position to make a sane decision about where to go from there.


No, we won't ever know.  Unicode bugs are subtle, so we won't 
know how much is broken unless someone goes and writes regression 
tests for a good chunk of the code in the Dub registry.



 Aug 19 2020



prev sibling next sibling parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com>  writes:


On Sunday, 16 August 2020 at 17:00:52 UTC, Steven Schveighoffer 
wrote:

 Hi all,

 TL;DR: I'm working on removing autodecoding from phobos. I have 
 the first real PR: https://github.com/dlang/phobos/pull/7595


Will this in the long run affect user-code anyway?

And how will it affect compiled code?



 Aug 19 2020



  parent reply Steven Schveighoffer <schveiguy gmail.com>  writes:


On 8/19/20 4:45 AM, Per Nordlöw wrote:

 On Sunday, 16 August 2020 at 17:00:52 UTC, Steven Schveighoffer wrote:

 Hi all,

 TL;DR: I'm working on removing autodecoding from phobos. I have the 
 first real PR: https://github.com/dlang/phobos/pull/7595


 
 Will this in the long run affect user-code anyway?


Yes, user code that depends on "xyz".front being dchar will be affected, 
etc. I expect that at some point soon (as I fix modules), we will get to 
a point where no-autodecode phobos is cohesive enough that we can start 
testing other projects. Then we will see the extent of the breakage. I 
don't really want to commit to saying how much there will be. My gut 
tells me there will be little breakages everywhere, but that are easily 
fixed.

Having a straightforward strategy as to how to migrate to 
non-autodecoding phobos would be key to getting this finalized.


 
 And how will it affect compiled code?


Considering that nearly all "optimizations" that concern autodecode go 
back to treating narrow strings as arrays, I'd say it's going to be a 
net positive in terms of performance.

-Steve



 Aug 19 2020



  parent  Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com>  writes:


On Wednesday, 19 August 2020 at 12:47:46 UTC, Steven 
Schveighoffer wrote:

 Having a straightforward strategy as to how to migrate to 
 non-autodecoding phobos would be key to getting this finalized.


Could warnings and/or deprecation messages for `.front` on arrays 
come to the rescue here? In order to adjust projects in advance 
for at least a year?



 Aug 19 2020



prev sibling  parent reply Paul Backus <snarwin gmail.com>  writes:


On Sunday, 16 August 2020 at 17:00:52 UTC, Steven Schveighoffer 
wrote:

 The autodecode test is run in buildkite. PLEASE don't merge any 
 PRs for Phobos if this test is failing! instead, ping me.


This test breaks CI for pull requests to dmd/druntime/phobos 
stable branches. For example:

https://github.com/dlang/dmd/pull/11633
https://github.com/dlang/dmd/pull/11634

I think it would be best to disable this until autodecode-test is 
available in phobos/stable.



 Aug 28 2020



  parent  Seb <seb wilzba.ch>  writes:


On Friday, 28 August 2020 at 13:27:33 UTC, Paul Backus wrote:

 On Sunday, 16 August 2020 at 17:00:52 UTC, Steven Schveighoffer 
 wrote:

 The autodecode test is run in buildkite. PLEASE don't merge 
 any PRs for Phobos if this test is failing! instead, ping me.


 This test breaks CI for pull requests to dmd/druntime/phobos 
 stable branches. For example:

 https://github.com/dlang/dmd/pull/11633
 https://github.com/dlang/dmd/pull/11634

 I think it would be best to disable this until autodecode-test 
 is available in phobos/stable.


Cherry-pick of the Makefile changes to stable: 
https://github.com/dlang/phobos/pull/7610



 Aug 28 2020