www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Release process and backwards compatibility

reply "Dicebot" <m.strashun gmail.com> writes:
Now that http://wiki.dlang.org/Release_Process is slowly adopted 
I want to rise this discussion again.

Two somewhat contradictory aims meet here, both frequently raised 
in IRC and newsgroup:

1) Breaking user code with release is incredibly painful and 
brings lot of dissatisfaction.
2) Language specification is not mature enough yet and it will 
need to be changed at some point unless we want to stay with same 
design issues forever (D3 is forever enough).

As far as I see it, both points have a quite wide support in the 
community.

It is important to note that when I speak of breaking code and 
language changes, I mean ALL changes. Currently weird hypocrisy 
has place when bug fixes that involve changing language semantics 
are not considered to be backwards-incompatible and this actually 
every release breaks code in practice. As dmd front-end IS the 
specification currently, there is no way for user to know if 
certain behavior is bug or feature thus any semantics change must 
be considered breaking change.

That is it, now we are in position when we both need to change 
stuff and can't. Every time this issue reappears, new hacks and 
workarounds are introduced without addressing it in general. I 
think this can be solved via some strict additions to the release 
process.

I'd gladly hear any proposals on topic, but here is mine:

Currently bug fixes go only to last major version branches. 
Considering how fast major version changes (and that we don't 
have minor versions in practice) that is hardly different from 
fixing only one branch. This is what makes 
backwards-compatibility problem that unpleasant - there are no 
versions of compiler user can stick to for a longer time period 
with hard 100% guarantees no semantics change will happen, 
receiving non-breaking fixes at the same time. There are no LTS 
releases. I think adding ones will allow to remove breaking 
change ban from master and allow continuous improvement of 
language design with no hard consequences for real-world users.

By LTS I mean certain announced version that gets back-ported all 
bug fixes that does not introduce any semantics change. To be 
usable in practice I thing those need to be released once a year 
with two LTS releases supported at time. Announcing, both via 
dlang.org and D.announce, is important here. That will require 
considerable efforts and may slow down development on master but 
at least it is step aside from current stalemate.

Opinions/proposals?
Mar 08 2013
next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
I don't know much about the development process, but as you
said some bugs may be features or vice versa. Sometimes real
bugs are fixed and peoples' code breaks. But keeping the bug
around isn't an option.
The next code breakage comes from making array slices
consistently rvalues (the slice structure itself, not the
data). It's not a new idea, like introducing immutable, just
wasn't correctly implemented from the start. I don't know if
this warrants a LTS release already. The problem will be
obvious and easy to fix by introducing a temp variable to pass
as lvalue into functions taking slices by ref. Or changing
those functions to auto-ref.
Mar 08 2013
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Marco Leise:


 I don't know much about the development process, but as you
 said some bugs may be features or vice versa. Sometimes real
 bugs are fixed and peoples' code breaks. But keeping the bug
 around isn't an option.


Fixing issues like this is right, despite they will cause some 
breakage in user code.

In DMD 2.063alpha this code gives a warning 
(http://d.puremagic.com/issues/show_bug.cgi?id=7444 ):

void main() {
     int[10] a;
     a = 1;
}


temp.d(3): Warning: explicit element-wise assignment (a)[] = 1 is 
better than a = 1


My suggestion for D users is to update their DMD compiler often, 
to avoid cumulating many breakages at the same time, that is 
quite bad. It's better to fix one thing at a time.

All bug fixes are potential code breakages, but not all breakages 
have the same magnitude an importance. My suggestion for D 
developers is to focus their efforts first on the D/DMD changes 
that will cause most breaking because the more time passes, the 
more D2 code is around, and the more breakage will happen.

This also means implementing things like in/scope sooner than 
later, that probably are sometimes used wrongly, and that will 
cause some breakage once implemented. Another similar case is 
"final switch" on ints that is currently broken 
(http://d.puremagic.com/issues/show_bug.cgi?id=5713) and likely 
misused in some D program. So once it's fixed it will break some 
user code.


Time ago I have suggested to statically disallow code like this 
(starting first with a deprecation, a warning, and then an 
error), that currently compiles without errors or warnings:


string[4] colors = ["red",
                     "blue"
                     "green",
                     "yellow"];
void main() {}


If similar code will be statically disallowed, it will break some 
user code (Walter agreed to forbid half of that combo-pitfall).


I have a bit of statistical proof that code like this is a common 
source of bugs, and I'd like this to be statically disallowed 
(http://d.puremagic.com/issues/show_bug.cgi?id=5409 ):

(!x & y)

But disallowing this will break some user code (not much, because 
it's likely a bug in the first place).

Bye,
bearophile
Mar 08 2013
parent "Daniel Murphy" <yebblies nospamgmail.com> writes:
"bearophile" <bearophileHUGS lycos.com> wrote in message 
news:hohjkspdzavtiqxqimms forum.dlang.org...

 string[4] colors = ["red",
                     "blue"
                     "green",
                     "yellow"];
 void main() {}


I hate this, a lot.


 I have a bit of statistical proof that code like this is a common source 
 of bugs, and I'd like this to be statically disallowed 
 (http://d.puremagic.com/issues/show_bug.cgi?id=5409 ):

 (!x & y)


This would be a good dmd patch for you to tackle bearophile.

Should be as simple as adding a check to the semantic routine of AndExp, 
looking at what type of expression e1 is.
Mar 08 2013
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 8 March 2013 at 20:33:20 UTC, Marco Leise wrote:

 I don't know much about the development process, but as you
 said some bugs may be features or vice versa. Sometimes real
 bugs are fixed and peoples' code breaks. But keeping the bug
 around isn't an option.


The problem isn't really breaking the code. It is known to be 
necessary on some subjects. It is more about how it is done.


 The next code breakage comes from making array slices
 consistently rvalues (the slice structure itself, not the
 data). It's not a new idea, like introducing immutable, just
 wasn't correctly implemented from the start. I don't know if
 this warrants a LTS release already. The problem will be
 obvious and easy to fix by introducing a temp variable to pass
 as lvalue into functions taking slices by ref. Or changing
 those functions to auto-ref.


Very good example (many code of mine broke on that one).

Yes this is the way things should have been from day 1. And 
solving that is clearly the way to go.

However, this change is right now into master and will go out 
with the next version of D, breaking a truly large amount of code 
(most code that uses slice as range will break).

This isn't the type of change you can release without any damage 
mitigation plan.
Mar 08 2013
parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Saturday, 9 March 2013 at 04:55:06 UTC, deadalnix wrote:

 This isn't the type of change you can release without any 
 damage mitigation plan.


Don't leave us hanging. What is your damage mitigation plan?

My desire is made elsewhere, but I don't think there is anything 
to do, code will break, instead dmd 2.062.1 should get released 
and contain regression and non-breaking bug fixes.
Mar 08 2013
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 9 March 2013 at 05:11:40 UTC, Jesse Phillips wrote:

 On Saturday, 9 March 2013 at 04:55:06 UTC, deadalnix wrote:

 This isn't the type of change you can release without any 
 damage mitigation plan.


 Don't leave us hanging. What is your damage mitigation plan?


I feel like a parot having to repeat the same shit again and 
again and again. So I wont, I expanded myself on such topic so 
many time already.

See how PHP handled the transition from 4 to 5 for a good example 
of how it can be done.
Mar 08 2013
parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Saturday, 9 March 2013 at 05:15:12 UTC, deadalnix wrote:

 I feel like a parot having to repeat the same shit again and 
 again and again. So I wont, I expanded myself on such topic so 
 many time already.

 See how PHP handled the transition from 4 to 5 for a good 
 example of how it can be done.


No, I'm not asking about improving the release process, I'm 
asking how do you mitigate damage done by this specific change. 
You can give people a compiler with some non-breaking updates to 
delay the inevitable, but if they take this change what needs to 
be done to make the transition easier?
Mar 08 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 9 March 2013 at 06:07:12 UTC, Jesse Phillips wrote:

 On Saturday, 9 March 2013 at 05:15:12 UTC, deadalnix wrote:

 I feel like a parot having to repeat the same shit again and 
 again and again. So I wont, I expanded myself on such topic so 
 many time already.

 See how PHP handled the transition from 4 to 5 for a good 
 example of how it can be done.


 No, I'm not asking about improving the release process, I'm 
 asking how do you mitigate damage done by this specific change. 
 You can give people a compiler with some non-breaking updates 
 to delay the inevitable, but if they take this change what 
 needs to be done to make the transition easier?


Many things can be done to mitigate that stuff.

The first, obvious one, is to provide a version of the compiler 
without the fix (bug with other fixes that don't break most code).

A second thing to do is to adapt phobos to take rvalues at 
several places. (This have the problem that ranges have 
unspecified behavior when passed by value).

Finally, an useful error message, with a workaround proposal 
would be welcome,
Error, range used as lvalue. You may want to do BLAH to solve 
that.
Mar 08 2013
parent "Jesse Phillips" <Jessekphillips+D gmail.com> writes:
On Saturday, 9 March 2013 at 06:22:15 UTC, deadalnix wrote:

 Many things can be done to mitigate that stuff.


Thank you,

Now we need to come up with ways to help facilitate making these 
changes.


 The first, obvious one, is to provide a version of the compiler 
 without the fix (bug with other fixes that don't break most 
 code).


This is supposed to be addressed by the new release process. I 
don't think helps in mitigation, it is avoidance which has some 
positives in its own right.


 A second thing to do is to adapt phobos to take rvalues at 
 several places. (This have the problem that ranges have 
 unspecified behavior when passed by value).


Phobos makes use of itself and so a disruption in the language 
will cause Phobos to show issue. If we adjust Phobos to work with 
itself/unittests, things we miss are likely because we didn't 
have a complete set of unittests on behavior?

Since unittest won't cover every case, maybe we could try a tag 
on the bugzilla entry to indicate Phobos was changed as a result. 
A comment including the need changes is added. When reviewing a 
pull request for such one would theorize on possible other 
implications and can request further changes if needed.

Something similar can be done for Phobos changes, as accepting a 
range by rvalue could have other implications and thus would need 
a good statement of breaking change.


 Finally, an useful error message, with a workaround proposal 
 would be welcome,
 Error, range used as lvalue. You may want to do BLAH to solve 
 that.


Yeah review should look for good error messages. Though we may 
want to be careful about suggestions (too many options, too long 
to describe, only relevant to broken code and not code originally 
incorrect).
Mar 11 2013
prev sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Saturday, 9 March 2013 at 05:11:40 UTC, Jesse Phillips wrote:

 On Saturday, 9 March 2013 at 04:55:06 UTC, deadalnix wrote:

 This isn't the type of change you can release without any 
 damage mitigation plan.


 Don't leave us hanging. What is your damage mitigation plan?

 My desire is made elsewhere, but I don't think there is 
 anything to do, code will break, instead dmd 2.062.1 should get 
 released and contain regression and non-breaking bug fixes.


As deadalnix have said (and I fully agree here), "The problem 
isn't really breaking the code. It is more about how it is 
done.". Thus, proper announcements and release model change 
actually is damage mitigation plan. Not really an excellent one, 
but better than having none currently.
Mar 08 2013
prev sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Friday, 8 March 2013 at 20:33:20 UTC, Marco Leise wrote:

 I don't know much about the development process, but as you
 said some bugs may be features or vice versa. Sometimes real
 bugs are fixed and peoples' code breaks. But keeping the bug
 around isn't an option.
 The next code breakage comes from making array slices
 consistently rvalues (the slice structure itself, not the
 data). It's not a new idea, like introducing immutable, just
 wasn't correctly implemented from the start. I don't know if
 this warrants a LTS release already. The problem will be
 obvious and easy to fix by introducing a temp variable to pass
 as lvalue into functions taking slices by ref. Or changing
 those functions to auto-ref.


LTS release should be something time-based, not feature based. It 
is not that important what exactly goes there if it is guaranteed 
to have support for some long time.

Your post reminds me of another idea though - it could have been 
a good attitude to have a separate "breaking changes/fixes" block 
in change log and, probably, announce those upon merged pull 
request before release is even made. One of least pleasant 
qualities of breaking bug fixes is that they are discussed in 
github mostly and come in a form of surprise upon release.
Mar 08 2013
prev sibling next sibling parent "Dicebot" <m.strashun gmail.com> writes:
P.S. Short discussion in IRC have shown that intent of the post 
is not clear enough: I am not here to rant about stability or how 
bad breaking changes are. Only thing I do want is to define some 
means to differentiate needs of those who want changes to improve 
language and those that require absolute stability, so that then 
can happily co-exist.
Mar 08 2013
prev sibling next sibling parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Friday, 8 March 2013 at 20:07:50 UTC, Dicebot wrote:

 Opinions/proposals?


I completely agree that the needs of users who want stability 
over everything are not being met. There's no way to choose to 
get just the updates that don't break code (such as non-breaking 
bug fixes), and I think you're right in that there should be a 
way to do that.

However, I just want to make clear that there's another (probably 
rather big) camp out there: the camp that thinks the current D 
platform needs a lot of improvement and are more than willing to 
accept breaking improvements.

I want more language features to be implemented and cleaned up. I 
want Phobos to be changed, cleaned up, trimmed as well as 
expanded. All this regardless of the cost of breaking code. I 
don't mind fixing broken code if the replacement code is simply 
better.

Thankfully, both the language and standard library are currently 
being improved in such ways, but not without a fair amount of 
inertia from parts of the community (particularly from Walter) 
who want stability at all costs.

For example, I don't think the current std.process is adequate at 
all and it would feel like a defeat if we have to name the new 
one std.process2 and have it live alongside the old trash (unless 
it's temporary). What do you think new users will think when they 
see or hear about the story around this? Many other standard 
modules are in the same position or are going to be in the same 
position at some point in the foreseeable future.

I just hope we can find a way to satisfy both camps, both 
ensuring the viability of the language at present by providing a 
stable interface, as well as securing the future of the language 
by iteratively improving upon it without compromising at every 
turn. Having LTS releases might be a good way to do that.
Mar 08 2013
next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Friday, 8 March 2013 at 21:07:23 UTC, Jakob Ovrum wrote:

 On Friday, 8 March 2013 at 20:07:50 UTC, Dicebot wrote:

 Opinions/proposals?


 I completely agree that the needs of users who want stability 
 over everything are not being met. There's no way to choose to 
 get just the updates that don't break code (such as 
 non-breaking bug fixes), and I think you're right in that there 
 should be a way to do that.

 However, I just want to make clear that there's another 
 (probably rather big) camp out there: the camp that thinks the 
 current D platform needs a lot of improvement and are more than 
 willing to accept breaking improvements.

 I want more language features to be implemented and cleaned up. 
 I want Phobos to be changed, cleaned up, trimmed as well as 
 expanded. All this regardless of the cost of breaking code. I 
 don't mind fixing broken code if the replacement code is simply 
 better.

 Thankfully, both the language and standard library are 
 currently being improved in such ways, but not without a fair 
 amount of inertia from parts of the community (particularly 
 from Walter) who want stability at all costs.

 For example, I don't think the current std.process is adequate 
 at all and it would feel like a defeat if we have to name the 
 new one std.process2 and have it live alongside the old trash 
 (unless it's temporary). What do you think new users will think 
 when they see or hear about the story around this? Many other 
 standard modules are in the same position or are going to be in 
 the same position at some point in the foreseeable future.

 I just hope we can find a way to satisfy both camps, both 
 ensuring the viability of the language at present by providing 
 a stable interface, as well as securing the future of the 
 language by iteratively improving upon it without compromising 
 at every turn. Having LTS releases might be a good way to do 
 that.


Personally I think we need to consider D3 as breaking. We can 
leave out any major changes till then or at least that would be 
nice. That way we can use D2 for LTS once we begin on D3.
How ever this will bring back e.g. the old python 2.x vs 3.x 
divide which would be a shame. But at the same time it'll mean we 
can do some big stuff that would just not be acceptable in 
breaking old projects.
Mar 08 2013
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, March 09, 2013 04:38:08 Rikki Cattermole wrote:

 Personally I think we need to consider D3 as breaking. We can
 leave out any major changes till then or at least that would be
 nice. That way we can use D2 for LTS once we begin on D3.
 How ever this will bring back e.g. the old python 2.x vs 3.x
 divide which would be a shame. But at the same time it'll mean we
 can do some big stuff that would just not be acceptable in
 breaking old projects.


D3 will not be happening any time soon. D2 will have to have been stable and 
around for a while before we plan to even consider it.

- Jonathan M Davis
Mar 08 2013
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Saturday, 9 March 2013 at 03:38:13 UTC, Rikki Cattermole wrote:

 How ever this will bring back e.g. the old python 2.x vs 3.x 
 divide which would be a shame. But at the same time it'll mean 
 we can do some big stuff that would just not be acceptable in 
 breaking old projects.


We don't need to go all the way over to a different language to 
learn from the past. We beat Python to it with D1. I don't think 
that was a mistake, but I think it would be now.
Mar 08 2013
prev sibling parent "Dicebot" <m.strashun gmail.com> writes:
D3 would imply language changes of D1->D2 scale, adding new 
paradigm at the very least. Otherwise,
1) it will ne a surprise
2) given amount of breaking changes/fixes, we will be in D42 
pretty soon
I don't think D3 can be even considered for that, it is all about 
making D2 mature, after all.
Mar 08 2013
prev sibling parent "Dicebot" <m.strashun gmail.com> writes:
I am in "changes are good" camp, too. The very reason that made 
me start this thread is that many good proposals in DIP's where 
rejected because they break the code while it breaks in practice 
anyway. It is simply not possible to have one product that suits 
perfectly two conflicting needs.
Mar 08 2013
prev sibling parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Friday, 8 March 2013 at 20:07:50 UTC, Dicebot wrote:

 Now that http://wiki.dlang.org/Release_Process is slowly 
 adopted I want to rise this discussion again.

 Two somewhat contradictory aims meet here, both frequently 
 raised in IRC and newsgroup:

 1) Breaking user code with release is incredibly painful and 
 brings lot of dissatisfaction.
 2) Language specification is not mature enough yet and it will 
 need to be changed at some point unless we want to stay with 
 same design issues forever (D3 is forever enough).


When a bug in the language design or compiler/libraries is fixed; 
the changes needed to upgrade aren't nearly as draining because 
it is the way forward. It is where the language should be going. 
But hitting a regression, waiting for the next release in hopes 
that a new regression won't crop up (I'm just assuming compiler 
is released with all known regressions completed) is detrimental 
to expectations. Updating code to match the "new" way just to 
have it reverted the next week (in git, thus no release); those 
things are what really kill the idea of stability.

So I would hope that a release is maintained until the next 
release. Regression and non-breaking bug fixes get applied to the 
release branch and are either released before the next release or 
at the same time. The point being that whatever issue was found 
was hopefully fixed and likely nothing new will be in the way to 
upgrade. A more elaborate system is fine, but I believe this 
truly the issues that need addressed first.

As for those who want no breaking changes, don't update your 
compiler or pay DigitalMars to maintain it. D will get to a point 
breaking changes aren't so frequent. But right now we have issue 
that must go.
Mar 08 2013
parent reply "Dicebot" <m.strashun gmail.com> writes:
On Saturday, 9 March 2013 at 05:05:42 UTC, Jesse Phillips wrote:

 When a bug in the language design or compiler/libraries is 
 fixed; the changes needed to upgrade aren't nearly as draining 
 because it is the way forward. It is where the language should 
 be going. But hitting a regression, waiting for the next 
 release in hopes that a new regression won't crop up (I'm just 
 assuming compiler is released with all known regressions 
 completed) is detrimental to expectations. Updating code to 
 match the "new" way just to have it reverted the next week (in 
 git, thus no release); those things are what really kill the 
 idea of stability.


Any well-thought change is the way forwards as it improves the 
language. And is actually a regression. You tend to separate 
"good regressions" and "bad regressions" but there is no possible 
way to separate those unless you are deep into D development. We 
have no spec (reference compiler is the spec) and for an end user 
all those break code equally surprisingly, having good intentions 
does not help.

And clearly because changes are _needed_ and bug-fixing can't be 
stopped, there needs to be clear accepted approach of "how do we 
break stuff" instead of "breaking stuff is bad". I have attempted 
to suggest one such approach.


 So I would hope that a release is maintained until the next 
 release. Regression and non-breaking bug fixes get applied to 
 the release branch and are either released before the next 
 release or at the same time. The point being that whatever 
 issue was found was hopefully fixed and likely nothing new will 
 be in the way to upgrade. A more elaborate system is fine, but 
 I believe this truly the issues that need addressed first.


This is already accepted as part of new release process, but, 
unfortunately, does not seem to be executed in practice. Problem 
with maintaining only last 2 releases is that they come out 
pretty fast, sometimes as fast as once in two months. That does 
not leave enough time to adapt without sacrificing mundane 
development.


 As for those who want no breaking changes, don't update your 
 compiler or pay DigitalMars to maintain it. D will get to a 
 point breaking changes aren't so frequent. But right now we 
 have issue that must go.


It is actually the other way around - I want _more_ breaking 
changes :) And that won't happen while two conflicting goals are 
pursued at the same time.
Mar 08 2013
parent "Jesse Phillips" <Jessekphillips+D gmail.com> writes:
On Saturday, 9 March 2013 at 07:19:34 UTC, Dicebot wrote:

 Any well-thought change is the way forwards as it improves the 
 language. And is actually a regression.


I disagree with your definition of regression and prefer:

"A return to a former or less developed state."


 You tend to separate "good regressions" and "bad regressions" 
 but there is no possible way to separate those unless you are 
 deep into D development. We have no spec (reference compiler is 
 the spec) and for an end user all those break code equally 
 surprisingly, having good intentions does not help.


I'm not talking hypothetical here, my experience. Update opEquals 
and toHash so they are const/pure/nothrow/fairy dust I'll grumble 
as I now need to update code I didn't write so don't understand 
the ramifications of my changes, but so be it. But to have to 
change it all back the next release because it was too disruptive 
and there might be a different approach that needs taken, that 
brings grumbling to a whole new level. (I'm not as annoyed as I 
make it sound, but I certainly am not sounding annoyed enough for 
many)


 And clearly because changes are _needed_ and bug-fixing can't 
 be stopped, there needs to be clear accepted approach of "how 
 do we break stuff" instead of "breaking stuff is bad". I have 
 attempted to suggest one such approach.


And that is what it comes down to, how do we do it. The Utopia, 
all non-breaking changes are applied to all past releases. We 
need a middle ground, somewhere probably not near the middle. And 
gave my approach.


 This is already accepted as part of new release process, but, 
 unfortunately, does not seem to be executed in practice.


Shouldn't we be trying to get this practiced first? Your 
suggestion still relies on maintaining something other than the 
current dev branch. If we can't get this, what are we left with?


 Problem with maintaining only last 2 releases is that they come 
 out pretty fast, sometimes as fast as once in two months. That 
 does not leave enough time to adapt without sacrificing mundane 
 development.


You can remain on the same compiler version you initially 
programmed for. I'm all for a Long term support like approach, 
but I see that as a secondary thing to properly releasing a dev.


 It is actually the other way around - I want _more_ breaking 
 changes :) And that won't happen while two conflicting goals 
 are pursued at the same time.


A Long Term Support does not allow for more breaking changes, it 
only delays how long before you have to deal with it. There are 
only two options to not have breaking changes, the C++/Windows 
approach (which fails) and never updating your program. But we 
can do a better job managing expectations.
Mar 11 2013