www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - github release procedure

reply Walter Bright <newshound2 digitalmars.com> writes:
As always, when I try to do a release, problems crop up. For example,
the github procedure agreed upon and outlined here:

http://wiki.dlang.org/Proposed_new_D_development_process#Release_a_new_version_of_D

Issues:

1. you cannot have a tag and a branch with the same name. At least, you cannot 
push them with:

     git push origin 2.N+1

because it's ambiguous. So I prepended a v to the tag name.

2. The:

    git checkout staging
    git merge master

It merges master into staging, wiping out my changes in staging, and does not 
delete staging. Now that the release is done, we're done with staging. What is 
needed is the ability to merge from staging to master all commits in staging 
that occurred after it branched off from master.

I did this by going through the git commit history and cherry-picking one by 
one. There's got to be better way.

3. There is no mention of where and when the:

    git push

and:

    git pull

get done. I also had to add staging to .git/config, can that be done from the 
push & pull command?

4. I think that staging should be deleted after the branch is done?

5. Since essentially the staging branch gets replaced by the 2.061 branch, why 
have a staging branch at all? Just make a 2.061 branch, then tag it when the 
release happens.
Jan 02 2013
next sibling parent reply "Jason House" <jason.james.house gmail.com> writes:
On Thursday, 3 January 2013 at 00:59:04 UTC, Walter Bright wrote:

 As always, when I try to do a release, problems crop up. For 
 example,
 the github procedure agreed upon and outlined here:

 http://wiki.dlang.org/Proposed_new_D_development_process#Release_a_new_version_of_D

 Issues:

 1. you cannot have a tag and a branch with the same name. At 
 least, you cannot push them with:

     git push origin 2.N+1

 because it's ambiguous. So I prepended a v to the tag name.


It looks like branch should be 2.N while tags are 2.N.M. It looks 
like the base of branch 2.N will be tagged as 2.N.0. If minor 
changes are needed, they go into the 2.N branch, and when ready 
the next tag on that branch will be 2.N.1, then 2.N.2, etc...



 2. The:

    git checkout staging
    git merge master

 It merges master into staging, wiping out my changes in 
 staging, and does not delete staging. Now that the release is 
 done, we're done with staging. What is needed is the ability to 
 merge from staging to master all commits in staging that 
 occurred after it branched off from master.


I believe the intent of staging is to get stable snapshots of 
master. At a minimum, I believe you'd merge master into staging 
when you're ready to do a dmdbeta release. I would recommend 
merging master into staging more frequently, but definitely not 
as often as master.
Jan 02 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 03 Jan 2013 03:51:48 +0100
schrieb "Jason House" <jason.james.house gmail.com>:


 On Thursday, 3 January 2013 at 00:59:04 UTC, Walter Bright wrote:

 As always, when I try to do a release, problems crop up. For 
 example,
 the github procedure agreed upon and outlined here:

 http://wiki.dlang.org/Proposed_new_D_development_process#Release_a_new_version_of_D

 Issues:

 1. you cannot have a tag and a branch with the same name. At 
 least, you cannot push them with:

     git push origin 2.N+1

 because it's ambiguous. So I prepended a v to the tag name.


 
 It looks like branch should be 2.N while tags are 2.N.M. It looks 
 like the base of branch 2.N will be tagged as 2.N.0. If minor 
 changes are needed, they go into the 2.N branch, and when ready 
 the next tag on that branch will be 2.N.1, then 2.N.2, etc...


This would mean changing the dmd versioning. We currently don't
increase the second number.

To match our versioning, branch should be 2.0.N and tag should be
2.0.N.M. But I like Walter's proposal more, as 2.0.61 should be the
same as 2.0.60.0 anyway so the .0 is redundant. 



 I believe the intent of staging is to get stable snapshots of 
 master. At a minimum, I believe you'd merge master into staging 
 when you're ready to do a dmdbeta release. I would recommend 
 merging master into staging more frequently, but definitely not 
 as often as master.


This would be an explanation, but we'd have to define what "stable
snapshot" means. If it's just "staging must always compile" that's
ok. But everything more advanced will need someone to review every
single commit. And when we actually want to prepare a release we need
all commits / new features anyway?

Imho staging isn't really necessary. We need a branch for every "big"
release, 2.0.60, 2.0.61 and so on. Then we add tags, v2.0.60 v2.0.61
and if we wan't a stable dmd, we use one of the release branches (e.g.
2.0.61) cherry pick bugfixes from master and the add a v2.0.61.1 tag
and release that.


What I think is weird in the current workflow is that we branch the
release branch if we actually ship the release (after beta testing,
etc). Why not just start the release branch if we start to prepare the
release (before beta testing).
Jan 03 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/3/2013 10:11 AM, Johannes Pfau wrote:

 What I think is weird in the current workflow is that we branch the
 release branch if we actually ship the release (after beta testing,
 etc). Why not just start the release branch if we start to prepare the
 release (before beta testing).


Yeah, that's what I was thinking.
Jan 03 2013
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Thursday, 3 January 2013 at 00:59:04 UTC, Walter Bright wrote:

 1. you cannot have a tag and a branch with the same name. At 
 least, you cannot push them with:

     git push origin 2.N+1

 because it's ambiguous. So I prepended a v to the tag name.


This is okay – the leading 'v' for version tags is common 
practice with Git, and if the Git workflow described on the wiki 
is intended to be official, it should be updated to reflect that 
(in a hurry right now, so I'll pass).

David
Jan 03 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-03 01:58, Walter Bright wrote:

 As always, when I try to do a release, problems crop up. For example,
 the github procedure agreed upon and outlined here:

 http://wiki.dlang.org/Proposed_new_D_development_process#Release_a_new_version_of_D


 Issues:

 1. you cannot have a tag and a branch with the same name. At least, you
 cannot push them with:

      git push origin 2.N+1

 because it's ambiguous. So I prepended a v to the tag name.


That is the convention used by git.


 2. The:

     git checkout staging
     git merge master

 It merges master into staging, wiping out my changes in staging, and
 does not delete staging. Now that the release is done, we're done with
 staging. What is needed is the ability to merge from staging to master
 all commits in staging that occurred after it branched off from master.

 I did this by going through the git commit history and cherry-picking
 one by one. There's got to be better way.


Can't you just merge staging into master?


 3. There is no mention of where and when the:

     git push

 and:

     git pull


Reading "Release a new version of D", I would say that after you run 
"git checkout staging" you need to make sure that you're local changes 
the the upstream changes are in sync. That can mean that you need to 
both run "git push" and "git pull".

After that the tag needs to be pushed. If you then also merges master 
into staging that should be pushed as well.


 get done. I also had to add staging to .git/config, can that be done
 from the push & pull command?

 4. I think that staging should be deleted after the branch is done?

 5. Since essentially the staging branch gets replaced by the 2.061
 branch, why have a staging branch at all? Just make a 2.061 branch, then
 tag it when the release happens.



-- 
/Jacob Carlborg
Jan 03 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/3/2013 3:59 AM, Jacob Carlborg wrote:

 On 2013-01-03 01:58, Walter Bright wrote:

 2. The:

     git checkout staging
     git merge master

 It merges master into staging, wiping out my changes in staging, and
 does not delete staging. Now that the release is done, we're done with
 staging. What is needed is the ability to merge from staging to master
 all commits in staging that occurred after it branched off from master.

 I did this by going through the git commit history and cherry-picking
 one by one. There's got to be better way.


 Can't you just merge staging into master?


And won't that remove the commits that are in master but not in staging?



 3. There is no mention of where and when the:

     git push

 and:

     git pull


 Reading "Release a new version of D", I would say that after you run "git
 checkout staging" you need to make sure that you're local changes the the
 upstream changes are in sync. That can mean that you need to both run "git
push"
 and "git pull".

 After that the tag needs to be pushed. If you then also merges master into
 staging that should be pushed as well.


I know there need to be push's and pull's. The question is the sequence, and
the 
actual commands, like push origin or just push.

The trouble is the wiki lists a sequence of commands, but assumes the user 
realizes that there are a bunch of other commands that need to be interleaved.

Imagine it's a script. Steps cannot be skipped :-)
Jan 03 2013
parent Johannes Pfau <nospam example.com> writes:
Am Thu, 03 Jan 2013 10:54:25 -0800
schrieb Walter Bright <newshound2 digitalmars.com>:


 On 1/3/2013 3:59 AM, Jacob Carlborg wrote:

 On 2013-01-03 01:58, Walter Bright wrote:

 2. The:

     git checkout staging
     git merge master

 It merges master into staging, wiping out my changes in staging,
 and does not delete staging. Now that the release is done, we're
 done with staging. What is needed is the ability to merge from
 staging to master all commits in staging that occurred after it
 branched off from master.

 I did this by going through the git commit history and
 cherry-picking one by one. There's got to be better way.


 Can't you just merge staging into master?


 
 And won't that remove the commits that are in master but not in
 staging?


No, merging will never remove commits. You aren't really supposed to
delete commits in git as it can lead to all kinds of trouble and
therefore deleting has to be done explicitly.

But I still advise not to merge staging into master. As all commits
should go to master first there should be no need to merge back.

If we have commits in staging now which are not in master you can
probably merge them into master. It won't look nice in the git history
but I shouldn't be a real problem.


 
 I know there need to be push's and pull's. The question is the
 sequence, and the actual commands, like push origin or just push.


The 'remote' to which you push the changes actually depends on your
setup. 'origin' is called a 'remote' and it's the repository where
you've git cloned from. So in a simple setup, it's always push origin.

Git can also 'track' branches. In that case your local branch tracks a
'remote' + branch. In that case you can use git push and it'll push to
the tracked remote branch. If you didn't create the staging branch
manually but just used git checkout staging it should be tracking
origin automatically. Anyway, you can just try git push, as long as you
don't have other remotes than origin, nothing can go wrong.

(you can use git remote to show all remotes)


 
 The trouble is the wiki lists a sequence of commands, but assumes the
 user realizes that there are a bunch of other commands that need to
 be interleaved.
 
 Imagine it's a script. Steps cannot be skipped :-)


I'll update the wiki if nobody outpaces me. But I'd like to have some
feedback/conclusion on the staging issue first.
Jan 03 2013
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 02 Jan 2013 16:58:22 -0800
schrieb Walter Bright <newshound2 digitalmars.com>:


 
 2. The:
 
     git checkout staging
     git merge master
 
 It merges master into staging, wiping out my changes in staging


What changes? All changes should be made in master, then applied to
staging via cherry picking. So there should never be changes in staging
that are not in master so there's nothing that could be overwritten?


, and does not delete staging. Now that the release is done, we're done
 with staging.


Why would you want to delete staging? This could be done, but I don't
see the benefit. Actually staging is quite redundant afaik. Why not
just create the release branch directly when we start to prepare a
release, then cherry pick from master into that release branch (but
never the other way round). Once the release is stable, give it a tag.
Then leave the branch alone (or continue the branch by cherry picking
further commits for a bugfix 2.061.1 release)

What's the benefit of first cherry-picking into staging and the
creating the release branch? If we use release branches directly we can
avoid the merge from master to staging after every release as we'll
just create a new branch for the new release. (Mixing cherry picking
and merging isn't exactly a best practice)


 What is needed is the ability to merge from staging to
 master all commits in staging that occurred after it branched off
 from master.


Wait- The wiki page isn't too clear here ("Development takes place on
the master branch"), but as far as I understand all changes should go
to master first. Even if those are bugfixes which will be used in
staging. The those will be cherry picked into staging if necessary. We
should never merge staging into master.


 
 I did this by going through the git commit history and cherry-picking
 one by one. There's got to be better way.


We should cherry pick one by one from master into staging. But we
should never have to merge anything from staging back into master.

(Technically you can probably pick many commits at once, so it's not
really one by one)

 
 3. There is no mention of where and when the:
 
     git push
 
 and:
 
     git pull
 
 get done. I also had to add staging to .git/config, can that be done
 from the push & pull command?


git fetch origin #update remote branch list
git checkout staging #should automatically track origin/staging

The wiki page could need some enhancements regarding pushing/pulling.


 
 4. I think that staging should be deleted after the branch is done?


I don't know why staging was proposed. As described above I don't know
why it's necessary.
 

 5. Since essentially the staging branch gets replaced by the 2.061
 branch, why have a staging branch at all? Just make a 2.061 branch,
 then tag it when the release happens.


Ah right, exactly my question. I should have read the whole message
first ;-)
Jan 03 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/3/2013 9:56 AM, Johannes Pfau wrote:

 Am Wed, 02 Jan 2013 16:58:22 -0800
 schrieb Walter Bright <newshound2 digitalmars.com>:


 2. The:

      git checkout staging
      git merge master

 It merges master into staging, wiping out my changes in staging


 What changes? All changes should be made in master, then applied to
 staging via cherry picking.


I think that's a vast assumption that there would never be changes specific to 
the release.



 , and does not delete staging. Now that the release is done, we're done
 with staging.


 Why would you want to delete staging? This could be done, but I don't
 see the benefit.


Turn that around - what's the benefit of keeping it? It's just clutter.



 What's the benefit of first cherry-picking into staging and the
 creating the release branch?


Beats me. That's why I asked.


 3. There is no mention of where and when the:

      git push

 and:

      git pull

 get done. I also had to add staging to .git/config, can that be done
 from the push & pull command?


 git fetch origin #update remote branch list
 git checkout staging #should automatically track origin/staging

 The wiki page could need some enhancements regarding pushing/pulling.


As I suggested to Jacob, if the wiki lists git command sequences, it should be 
complete (like a script), and not full of assumptions about other commands that 
need to be inserted.
Jan 03 2013
next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 03 Jan 2013 10:58:46 -0800
schrieb Walter Bright <newshound2 digitalmars.com>:



 What changes? All changes should be made in master, then applied to
 staging via cherry picking.


 
 I think that's a vast assumption that there would never be changes
 specific to the release.


If there's a change which is 100% specific to a release it can be
pushed to staging directly, but then there's no need to merge it back
into master. If the change makes sense in master, push it to master then
cherry pick into staging.
Jan 03 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/3/13 1:58 PM, Walter Bright wrote:

 As I suggested to Jacob, if the wiki lists git command sequences, it
 should be complete (like a script), and not full of assumptions about
 other commands that need to be inserted.


I think this is a pertinent point - the process proposed at github is 
incomplete and scantily motivated. Can the experts make one more pass 
through it?

Thanks,

Andrei
Jan 03 2013
next sibling parent reply "Rob T" <rob ucora.com> writes:
On Thursday, 3 January 2013 at 19:19:49 UTC, Andrei Alexandrescu 
wrote:

 On 1/3/13 1:58 PM, Walter Bright wrote:

 As I suggested to Jacob, if the wiki lists git command 
 sequences, it
 should be complete (like a script), and not full of 
 assumptions about
 other commands that need to be inserted.


 I think this is a pertinent point - the process proposed at 
 github is incomplete and scantily motivated. Can the experts 
 make one more pass through it?

 Thanks,

 Andrei


I'm rather concerned when I see comments that suggest that the 
purpose of the staging branch was not understood in the 
slightest. There's a lot of discussion on the issue here 
http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com

The people who are attempting to use the new process have to at 
least take some time to read through the process discussion 
threads, certainly that is not ideal, and the wiki should explain 
everything much better.

The reason for a less than adequate process description, is that 
the process was not yet fully completed and was difficult to 
complete without a dry run to work out the bugs and oversights 
etc. Instead of a dry run, we're experiencing the real thing, 
started in the middle of the holidays to boot.

We have to expect some problems of course!

It would be nice to contain everything under one discussion 
thread so that the same people who started this whole thing are 
more likely to notice the complaints and misunderstandings and so 
forth.

--rt
Jan 03 2013
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 03 Jan 2013 21:17:08 +0100
schrieb "Rob T" <rob ucora.com>:


 On Thursday, 3 January 2013 at 19:19:49 UTC, Andrei Alexandrescu 
 wrote:

 On 1/3/13 1:58 PM, Walter Bright wrote:

 As I suggested to Jacob, if the wiki lists git command 
 sequences, it
 should be complete (like a script), and not full of 
 assumptions about
 other commands that need to be inserted.


 I think this is a pertinent point - the process proposed at 
 github is incomplete and scantily motivated. Can the experts 
 make one more pass through it?

 Thanks,

 Andrei


 
 I'm rather concerned when I see comments that suggest that the 
 purpose of the staging branch was not understood in the 
 slightest. There's a lot of discussion on the issue here 
 http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com


Sorry, but I'm not sure if we have a consensus on some things discussed
there:
This is probably the best summary:
http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com?page=16#post-woyboqigqbkqjxmshppn:40forum.dlang.org


"4. Feature can than be further refined and _integration bugs_ can 
be fixed by the general dev team."

What's the overhead for many small feature branches? DMD work is
probably 90% small bug fixes and 10% bigger features. So most of the
time there are no "integration bugs" and we just waste man power
reviewing commits again and copying those into staging?


"When the "dev" branch is considered stable enough by the team 
(exact criteria to be defined later), the changes are merged to 
the _2nd level of integration_ - the "staging" branch. This 
allows for a wider audience to test and provide real-world 
feedback."

The exact criteria would be kinda important. But as we have many small
bugfixes which are merged every day I don't see how the master branch
would suddenly get more "stable".


What I am most concerned about are the timespans discussed:
"I propose to go for a yearly release of the 
stable branches with one year support (In the beginning)."

The wiki discussion page even mentions "I don't think 4 months are a
ridiculously long time for staging if the release is made every 3
years."
Jan 03 2013
next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, January 03, 2013 21:42:37 Johannes Pfau wrote:

 What I am most concerned about are the timespans discussed:
 "I propose to go for a yearly release of the
 stable branches with one year support (In the beginning)."
 
 The wiki discussion page even mentions "I don't think 4 months are a
 ridiculously long time for staging if the release is made every 3
 years."


That makes it sound like they want the current stuff to be marked as staging 
and then have some other release that sits around for a very long time being 
treated as somehow more stable than the staging branch. In general, there's 
nothing about bug fixing which is stable, and separating out bug fixes between
a 
more stable branch and a less stable one doesn't make all that much sense. 
Separating out new features and not putting them in the "stable" branch makes 
sense, but that really doesn't make sense for bugs.

Also, the name "staging" implies that it's purely for preparing a release, in 
which case keeping it around makes _no_ sense. Not to mention, as already 
mentioned, it would make more sense to simply create a new branch for each 
release to begin with than have a staging branch if that's what it's for. And 
if that's not what it's for, then it's a horrible name.

- Jonathan M Davis
Jan 03 2013
parent reply "foobar" <foo bar.com> writes:
On Thursday, 3 January 2013 at 22:04:28 UTC, Jonathan M Davis 
wrote:

 On Thursday, January 03, 2013 21:42:37 Johannes Pfau wrote:

 What I am most concerned about are the timespans discussed:
 "I propose to go for a yearly release of the
 stable branches with one year support (In the beginning)."
 
 The wiki discussion page even mentions "I don't think 4 months 
 are a
 ridiculously long time for staging if the release is made 
 every 3
 years."


 That makes it sound like they want the current stuff to be 
 marked as staging
 and then have some other release that sits around for a very 
 long time being
 treated as somehow more stable than the staging branch. In 
 general, there's
 nothing about bug fixing which is stable, and separating out 
 bug fixes between a
 more stable branch and a less stable one doesn't make all that 
 much sense.
 Separating out new features and not putting them in the 
 "stable" branch makes
 sense, but that really doesn't make sense for bugs.

 Also, the name "staging" implies that it's purely for preparing 
 a release, in
 which case keeping it around makes _no_ sense. Not to mention, 
 as already
 mentioned, it would make more sense to simply create a new 
 branch for each
 release to begin with than have a staging branch if that's what 
 it's for. And
 if that's not what it's for, then it's a horrible name.

 - Jonathan M Davis


As I explained several times before, this is a HORRIBLE idea. If 
each release has its own branch than a bug fix needs to be 
applied to _all_ such branches _manually_ and it is easy to get a 
situation where a fix on say version 3.1 was accidentally not 
merged to version 4.0 (forgotten) and suddenly the bug was 
"unfixed".
Whatever the name of this branch is, the semantics/purpose of it 
is to provide integration.
Jan 03 2013
parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 00:47:08 foobar wrote:

 On Thursday, 3 January 2013 at 22:04:28 UTC, Jonathan M Davis
 
 wrote:

 On Thursday, January 03, 2013 21:42:37 Johannes Pfau wrote:

 What I am most concerned about are the timespans discussed:
 "I propose to go for a yearly release of the
 stable branches with one year support (In the beginning)."
 
 The wiki discussion page even mentions "I don't think 4 months
 are a
 ridiculously long time for staging if the release is made
 every 3
 years."


 
 That makes it sound like they want the current stuff to be
 marked as staging
 and then have some other release that sits around for a very
 long time being
 treated as somehow more stable than the staging branch. In
 general, there's
 nothing about bug fixing which is stable, and separating out
 bug fixes between a
 more stable branch and a less stable one doesn't make all that
 much sense.
 Separating out new features and not putting them in the
 "stable" branch makes
 sense, but that really doesn't make sense for bugs.
 
 Also, the name "staging" implies that it's purely for preparing
 a release, in
 which case keeping it around makes _no_ sense. Not to mention,
 as already
 mentioned, it would make more sense to simply create a new
 branch for each
 release to begin with than have a staging branch if that's what
 it's for. And
 if that's not what it's for, then it's a horrible name.
 
 - Jonathan M Davis


 
 As I explained several times before, this is a HORRIBLE idea. If
 each release has its own branch than a bug fix needs to be
 applied to _all_ such branches _manually_ and it is easy to get a
 situation where a fix on say version 3.1 was accidentally not
 merged to version 4.0 (forgotten) and suddenly the bug was
 "unfixed".
 Whatever the name of this branch is, the semantics/purpose of it
 is to provide integration.


??? Bug fixes go to master. New release branches are branched from master. So, 
you're not going to have future branches not getting previous bug fixes. The 
only way that you're going to get problems with bug fixes not making it into 
future releases is if you commit them to the branch first rather than directly 
to master (which is a bad idea IMHO) and then you forget to merge that back 
into master.

- Jonathan M Davis
Jan 03 2013
prev sibling parent reply "foobar" <foo bar.com> writes:
On Thursday, 3 January 2013 at 20:42:39 UTC, Johannes Pfau wrote:

 Am Thu, 03 Jan 2013 21:17:08 +0100
 schrieb "Rob T" <rob ucora.com>:


 On Thursday, 3 January 2013 at 19:19:49 UTC, Andrei 
 Alexandrescu wrote:

 On 1/3/13 1:58 PM, Walter Bright wrote:

 As I suggested to Jacob, if the wiki lists git command 
 sequences, it
 should be complete (like a script), and not full of 
 assumptions about
 other commands that need to be inserted.


 I think this is a pertinent point - the process proposed at 
 github is incomplete and scantily motivated. Can the experts 
 make one more pass through it?

 Thanks,

 Andrei


 
 I'm rather concerned when I see comments that suggest that the 
 purpose of the staging branch was not understood in the 
 slightest. There's a lot of discussion on the issue here 
 http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com


 Sorry, but I'm not sure if we have a consensus on some things 
 discussed
 there:
 This is probably the best summary:
 http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com?page=16#post-woyboqigqbkqjxmshppn:40forum.dlang.org


 "4. Feature can than be further refined and _integration bugs_ 
 can
 be fixed by the general dev team."

 What's the overhead for many small feature branches? DMD work is
 probably 90% small bug fixes and 10% bigger features. So most 
 of the
 time there are no "integration bugs" and we just waste man power
 reviewing commits again and copying those into staging?


 "When the "dev" branch is considered stable enough by the team
 (exact criteria to be defined later), the changes are merged to
 the _2nd level of integration_ - the "staging" branch. This
 allows for a wider audience to test and provide real-world
 feedback."

 The exact criteria would be kinda important. But as we have 
 many small
 bugfixes which are merged every day I don't see how the master 
 branch
 would suddenly get more "stable".


 What I am most concerned about are the timespans discussed:
 "I propose to go for a yearly release of the
 stable branches with one year support (In the beginning)."

 The wiki discussion page even mentions "I don't think 4 months 
 are a
 ridiculously long time for staging if the release is made every 
 3
 years."


Let's clarify a bunch of things here:
1. There is no overhead with maintaining many feature brunches.
2. Feature branches can be safely removed after their contents 
have been merged. This is useful for avoiding clutter.
3. There is no need to re-review the _same_ code on each branch 
it is merged to. That's just plain stupid.
4. Chery picking is a HORRIBLE idea for managing releases and 
should only be used on development. Chery-picking COPIES commits 
and thus loses the connections in the history graph.
5. There is no problem developing bug_fixes on separate topic 
branches.
6. There is no problem merging back to master.
7. Let me say this again. Merges are cheap and easy in git. So 
are re-merges. Avoiding merges is an artifact of client-server 
designs such as SVN and is NOT compatible with Git's conceptual 
model.

I suggest to read about how git works and how best to utilize its 
features, and NOT apply common wisdom from CVS/SVN/RCS which are 
based on a _completely different_ conceptual model.
Jan 03 2013
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/3/13 6:40 PM, foobar wrote:

 I suggest to read about how git works and how best to utilize its
 features, and NOT apply common wisdom from CVS/SVN/RCS which are based
 on a _completely different_ conceptual model.


Better yet, since you are already so conversant with git, you could help 
us improve the process directly. Care to make a pass through the wiki page?

Andrei
Jan 03 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 00:40:03 +0100
schrieb "foobar" <foo bar.com>:

 
 Let's clarify a bunch of things here:
 1. There is no overhead with maintaining many feature brunches.


I didn't mean the overhead of the branches itself. I meant the overhead
of merging those first into master. Then later determining if master is
stable (how?) and then merging into staging. The wiki says this is
useful to allow fixing integration bugs. I think with mostly bugfixes
there won't be many integration bugs. And merging into staging always
requires someone to do it, which is imho overhead (regarding "human
resources", not technical).


 2. Feature branches can be safely removed after their contents 
 have been merged. This is useful for avoiding clutter.


sure

 3. There is no need to re-review the _same_ code on each branch 
 it is merged to. That's just plain stupid.


Well, the wiki says things are merged into staging if they are
considered sufficiently stable. Who determines what is stable and how?


 4. Chery picking is a HORRIBLE idea for managing releases and 
 should only be used on development. Chery-picking COPIES commits 
 and thus loses the connections in the history graph.


Sure, this is why I said mixing cherry picking and merging is not a
good idea. As long as you only cherry pick into a branch and never
merge into that branch or merge it back there is no issue.

I missed the detail that you suppose basing bugfix branches on the
oldest applicable release branch. That should be made very clear in the
wiki.

I just think that this process is counter-productive for dmd as most
changes are bugfixes and regressions are caused by bug fixes as well.
Applying _every_ bugfix in older releases has a high risk of
reintroducing regressions (and / or breaking user code. If the compiler
didn't enforce const in some situations, fixing it is a bug fix. But
fixing it will break user code - so you have a 2.061 that works for
users and a 2.061.1 which breaks code).

It makes more sense to me to only fix regressions that were introduced
with the base release. So 2.061.1 would only fix regressions introduced
in 2.061. (This can still be done by basing that fix on the latest
applicable version and merging back into master / staging).

Also the lowest applicable version for most dmd bugs is probably 2.01.
So if we start this process now, we'll still be fixing bugs in 2.061 in
10 years, then merging into 2.062, 2.063.... Here "lowest _supported_
applicable version" makes more sense imho, where supported needs to be
defined (e.g. 2 releases). This is not a big problem if we only talk
about regressions, but I'd still prefer "lowest _supported_ applicable
version"


 5. There is no problem developing bug_fixes on separate topic 
 branches.


As we already do.

 6. There is no problem merging back to master.


As long as there are no cherry picked commits into that branch. AFAIK
we have already cherry picked commits from master into staging. Then
merging staging back into master is not good - It will probably work
but as you said cherry picking duplicates commits and we'll therefore
merge identical changes with different ids.

We should probably delete staging and start again as soon as the
workflow is clear to everyone.


 7. Let me say this again. Merges are cheap and easy in git. So 
 are re-merges. Avoiding merges is an artifact of client-server 
 designs such as SVN and is NOT compatible with Git's conceptual 
 model.
 
 I suggest to read about how git works and how best to utilize its 
 features, and NOT apply common wisdom from CVS/SVN/RCS which are 
 based on a _completely different_ conceptual model.


I don't have common wisdom from cvs/svn ;-)
Merges are not that easy if you want to merge only a single commit or
skip certain commits. Although you can tell merge to skip commits it's
in no way easy. http://stackoverflow.com/a/729723/471401

It's  probably best to not get into this situation. The workflow
described on the wiki can indeed work well without cherry picking.

But what was done till 2.061 was basing bug fixes on the master branch.
If you want to get these into a release branch without features, etc
cherry-picking is way easier than merging.
Jan 04 2013
prev sibling parent reply Russel Winder <russel winder.org.uk> writes:
On Fri, 2013-01-04 at 00:40 +0100, foobar wrote:
[=E2=80=A6]

 Let's clarify a bunch of things here:
 1. There is no overhead with maintaining many feature brunches.


Which is why Git is superior to Mercurial for the sort of process going
on here.


 2. Feature branches can be safely removed after their contents=20
 have been merged. This is useful for avoiding clutter.


See above :-)


 3. There is no need to re-review the _same_ code on each branch=20
 it is merged to. That's just plain stupid.


I am not so sure on this. A changeset in a different context is a
different changeset, which is why Mercurial is the way it is and why it
has some benefits overf Git.  The effect of a code change in one branch
is not necessarily the same as the effect of the same change in a
different branch. However review is not the right tool here, CI is. Each
and every branch in the mainline must have a CI listener that runs all
the code on every commit.

Branches in the mainline are CVCS thinking, not wrong, but must be
treated as an integral part of the project, hence the CI requirement.
Branches held in forks of the mainline have no such requirement, but
people should run all the tests after each commit as a matter of
professionalism. Others can of course pull these feature branches into
their own fork repository for working on. This is the DVCS way.
Eventually of course commit to the mainline is required otherwise a
project fork has been made.


 4. Chery picking is a HORRIBLE idea for managing releases and=20
 should only be used on development. Chery-picking COPIES commits=20
 and thus loses the connections in the history graph.


Agreed.


 5. There is no problem developing bug_fixes on separate topic=20
 branches.


See 1 above. Git makes this trivially easy.


 6. There is no problem merging back to master.


See 1 above. Git makes this trivially easy.


 7. Let me say this again. Merges are cheap and easy in git. So=20
 are re-merges. Avoiding merges is an artifact of client-server=20
 designs such as SVN and is NOT compatible with Git's conceptual=20
 model.


It is important to note that Mercurial has different properties again,
so leads to different workflow. Given the choice has been Git then Git
workflows apply.

Every developer should be keeping master as a mirror of the mainline
master and working on feature branches. When a changeset is accepted
after review, the committer commits to the mainline and then everyone
can pull and garbage collect, and in the process delete the feature
branch they have.

I tend to fork the mainline repository locally and also on GitHub and
then manually link my two forks. This means "git pull" locally pulls all
branches from the mainline repository. I work on feature branches pushed
to my GitHub fork, people can then pull them as they wish. You have to
remember to fetch from all the remote branches you are tracking. I can
then make a pull request on my GitHub fork and still continue to track
the mainline locally.


 I suggest to read about how git works and how best to utilize its=20
 features, and NOT apply common wisdom from CVS/SVN/RCS which are=20
 based on a _completely different_ conceptual model.


Indeed.

Star model has a role in DVCS, the mainline repository is the mainline
from which all else stems. But not all traffic has to be routed through
it with Git. People can share feature branches between themselves to
collaborate to create a pull request without ever having to use the
mainline. That is the whole point.

It means that the mainline committer role is not to do all the
development work, but is to coordinate and mentor activity which they
then review and when happy merge into mainline.

No-one should ever be doing development on master.

No-one should ever push to mainline without someone at least knowing
they are about to do it and have previously checked the work.

No mainline repository should ever be without a full CI.

http://d.puremagic.com/test-results/

Looks like part of a solution.  It should also provides artefacts for
download and install as nightly releases.  It assumes there is only
master in the repository. If there are any other branches in the
mainline repository then where are the CIs for those?

In my clone of DMD I see master, staging, dmd-1.x branches. Given DMD is
a single branch development, I don't see a need for staging. I shall
have to go through all the email to find the rationale, unless someone
has a short paragraph rationale.

What I find really bizarre is that the tags v2.061 and v1.076 are on
branches other than master. To me the whole Git history looks a mess,
essentially broken. I suggest some serious rebasing and merging to make
the history consistent with the version numbering.

The questions to ask:

If I want to create a feature branch, which branch do I use as a base?

Is staging just the 2.x maintenance branch?

When do changes on master make it to staging?

Should staging and dmd-1.x branch HEADS be moved forwards on the branch?

Sorry if this is all answered somewhere, just point me at the URL where
the answers are explained. =20

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Jan 04 2013
parent reply "mist" <none none.none> writes:
Weird. While I was reading "Next focus: PROCESS" topic and 
http://wiki.dlang.org/Proposed_new_D_development_process entries, 
I was completely convinced that switch to faster release model is 
proposed, so that master replaces staging the very moment one 
version is released (and it is the same moment as testing of next 
release begins). Plus once a year or so release is marked LTS and 
cherry-pick bug-fixes until next LTS is out. Was my understanding 
wrong?

Anyway, now I am reading all those comments and it feels absurdly 
like no one has discussed this stuff few weeks ago and different 
topic and argument has started from beginning. This is madness.
Jan 04 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 12:02:44 +0100
schrieb "mist" <none none.none>:


 Weird. While I was reading "Next focus: PROCESS" topic and 
 http://wiki.dlang.org/Proposed_new_D_development_process entries, 
 I was completely convinced that switch to faster release model is 
 proposed, so that master replaces staging the very moment one 
 version is released (and it is the same moment as testing of next 
 release begins). Plus once a year or so release is marked LTS and 
 cherry-pick bug-fixes until next LTS is out. Was my understanding 
 wrong?
 
 Anyway, now I am reading all those comments and it feels absurdly 
 like no one has discussed this stuff few weeks ago and different 
 topic and argument has started from beginning. This is madness.


So someone who was part of that discussion: Please give us a short
summary. You can't expect everyone to read through all those posts
again, 95% is probably obsolete.

And most important: Were the main developers part of that discussion /
have they agreed? While I agree LTS would be nice it needs someone to
do it. You also have to consider the available man power.

But this is not very relevant for this discussion: It's always possible
to increase support time later.
Jan 04 2013
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 15:20:02 Johannes Pfau wrote:

 And most important: Were the main developers part of that discussion /
 have they agreed? While I agree LTS would be nice it needs someone to
 do it. You also have to consider the available man power.


For the most part, I don't think that they were. I think that Andrei was 
really the only one of us involved at all, though I'd have to look at the 
thread again to be sure.

- Jonathan M Davis
Jan 04 2013
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 04, 2013 at 08:15:34AM -0800, Jonathan M Davis wrote:

 On Friday, January 04, 2013 15:20:02 Johannes Pfau wrote:

 And most important: Were the main developers part of that discussion
 / have they agreed? While I agree LTS would be nice it needs someone
 to do it. You also have to consider the available man power.


 
 For the most part, I don't think that they were. I think that Andrei
 was really the only one of us involved at all, though I'd have to look
 at the thread again to be sure.


[...]

We solicited for core dev involvement in that thread, but AFAIK, there
was only limited participation. It felt like we were just talking about
castles in the air, because the people who would actually be
implementing the process didn't say very much at all. So I wasn't at all
surprised at the misunderstandings in this thread.

Still, I think we're finally converging on the same page here, so that's
a good thing.


T

-- 
Amateurs built the Ark; professionals built the Titanic.
Jan 04 2013
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Jan 03, 2013 at 09:17:08PM +0100, Rob T wrote:

 On Thursday, 3 January 2013 at 19:19:49 UTC, Andrei Alexandrescu
 wrote:

On 1/3/13 1:58 PM, Walter Bright wrote:

As I suggested to Jacob, if the wiki lists git command sequences, it
should be complete (like a script), and not full of assumptions
about other commands that need to be inserted.


I think this is a pertinent point - the process proposed at github is
incomplete and scantily motivated. Can the experts make one more pass
through it?




[...]

 I'm rather concerned when I see comments that suggest that the
 purpose of the staging branch was not understood in the slightest.
 There's a lot of discussion on the issue here
 http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com


In my mind, 'staging' is just a placeholder name for the place where we
can freeze new features and stabilize them before a release. The actual
mechanics of doing this is independent of the concept itself.

We could either:

(1) Have an actual branch called 'staging', where changes made to master
since the last merge are merged in immediately after release, and when
things have sufficiently stabilized, we tag/branch that particular
commit as release 2.X.Y;

(2) Or, we can follow Walter's suggestion of branching master into 2.X.Y
after a release and just tag it as a release when it's sufficiently
stable. In this case, the latest 2.X.Y branch acts as 'staging'.

Part of the draw of (1) is that betatesters and adventurous people have
a fixed place to go to, to get the latest and greatest features, but
without the instability of master.

I think (2) is more commonly what is actually done by software companies
and projects. In this case, 'staging' isn't so much as an actual
physical branch, as the general concept of having a branch (2.X.Y, which
changes per release) to stabilize things for a release, while
development on master continues unhindered.

The Linux kernel also uses a similar kind of process: X.Y.Z is a
development release (can be thought of as 'master') when Y is odd, and a
stable release (can be thought of as 'staging' or Walter's release
branches) when Y is even.

Basically, the idea is that master should be able to receive the latest
and greatest features, completely unhindered, while stabilizing the code
for a release is done in a separate branch which doesn't receive new
features, but only bugfixes, prior to an actual release.



 The people who are attempting to use the new process have to at
 least take some time to read through the process discussion threads,
 certainly that is not ideal, and the wiki should explain everything
 much better.


[...]

I confess to having contributed some poorly-written material on the wiki
page. Please feel free to revise it.


T

-- 
The computer is only a tool. Unfortunately, so is the user. -- Armaphine, K5
Jan 03 2013
parent reply "Rob T" <rob ucora.com> writes:
On Thursday, 3 January 2013 at 22:11:59 UTC, H. S. Teoh wrote:

 We could either:

 (1) Have an actual branch called 'staging', where changes made 
 to master
 since the last merge are merged in immediately after release, 
 and when
 things have sufficiently stabilized, we tag/branch that 
 particular
 commit as release 2.X.Y;

 (2) Or, we can follow Walter's suggestion of branching master 
 into 2.X.Y
 after a release and just tag it as a release when it's 
 sufficiently
 stable. In this case, the latest 2.X.Y branch acts as 'staging'.

 Part of the draw of (1) is that betatesters and adventurous 
 people have
 a fixed place to go to, to get the latest and greatest 
 features, but
 without the instability of master.


(1) is what I was advocating precisely so that beta testers could 
use it shortly after a release and up to the final next release. 
There are a few benefits to this that may be under appreciated. 
There's plenty of benefits to having a lot of beta testers, so we 
want to draw them in and encourage them to use the latest beta 
version as soon as possible. Real world programmers do not want 
to make use of something that is too unstable, so they will tend 
to avoid the master development branch and will prefer the latest 
stable release instead. The only reason they will try out the 
latest beta version, is to gain access to a bug fix or new 
feature that they really need for their real world programming 
tasks, but they will only try it out if they think that the beta 
is stable enough and is not going to get horribly broken by 
anything radical being merged in without proper oversight and 
planning (that's what master and the related feature branches are 
for).

If we immediately move master into the next staging after a new 
release, and freeze the introduction of new features, only to 
concentrate on stabilizing it for the next release, then we're 
more likely to attract beta testers from the real world, these 
are the guys who will do testing under the most realistically 
conditions possible, something that in invaluable in gaining feed 
back and routing our difficult to discover bugs.


 I think (2) is more commonly what is actually done by software 
 companies
 and projects. In this case, 'staging' isn't so much as an actual
 physical branch, as the general concept of having a branch 
 (2.X.Y, which
 changes per release) to stabilize things for a release, while
 development on master continues unhindered.

 The Linux kernel also uses a similar kind of process: X.Y.Z is a
 development release (can be thought of as 'master') when Y is 
 odd, and a
 stable release (can be thought of as 'staging' or Walter's 
 release
 branches) when Y is even.

 Basically, the idea is that master should be able to receive 
 the latest
 and greatest features, completely unhindered, while stabilizing 
 the code
 for a release is done in a separate branch which doesn't 
 receive new
 features, but only bugfixes, prior to an actual release.


I'm not certain what the major difference is between (1) and (2), 
because they both sound very nearly identical. In both cases, 
they seem to provide the same goals of stabilizing the next 
release, while allowing development to proceed unhindered, so if 
(2) can be made to achieve the goal I described for attracting 
real world beta testers, it may not matter which is used, however 
I would suggest that we always focus on serving the needs of the 
end user as a top priority as much as is possible. D requires 
plenty of significant end users working on significant projects.

You did mention that (1) is more suitable for beta testers, so I 
ask how is (2) not as suitable?

--rt
Jan 03 2013
next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 07:59:05 +0100
schrieb "Rob T" <rob ucora.com>:


 
 You did mention that (1) is more suitable for beta testers, so I 
 ask how is (2) not as suitable?


With (1) you can always use staging, it'll always be the latest beta.
With (2) you'll have to figure out the most recent branch (2.061,
2.062, 2.063).

But I think we can start with (2) and change to (1) later if we think
it's useful. Staging complicates the concept(on a cognitive level) and
it needs additional "human resources" to do the merges into it.


I think I understand the whole concept now. I seems to be a good
solution but it requires some discipline (basing changes on the correct
branches) so we should document it as clearly as possible. I'd strongly
recommend to only base regression fixes on release branches and usual
bug fixes on master (see my other message).
Jan 04 2013
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 07:59:05 +0100
schrieb "Rob T" <rob ucora.com>:


 
 I'm not certain what the major difference is between (1) and (2), 
 because they both sound very nearly identical. In both cases, 
 they seem to provide the same goals of stabilizing the next 
 release, while allowing development to proceed unhindered, so if 
 (2) can be made to achieve the goal I described for attracting 
 real world beta testers, it may not matter which is used, however 
 I would suggest that we always focus on serving the needs of the 
 end user as a top priority as much as is possible. D requires 
 plenty of significant end users working on significant projects.
 
 You did mention that (1) is more suitable for beta testers, so I 
 ask how is (2) not as suitable?
 
 --rt


http://wiki.dlang.org/User:Jpf/Release_Process

I updated the page with some more git commands, ascii art and color so
it's hopefully easier to understand now. Please tell me if I got
anything wrong. The only thing I (intended to change) changed is that I
removed staging.

Although I can see the benefits of staging as you describe it, we can
get most of those by just creating a new release/version branch as soon
as the current release is released. This removes some complexity from
the workflow.

AFAIK it should be easy enough to introduce staging later on, when
everyone is comfortable with the new workflow. (Please consider that
many of us are not git experts - so if we can avoid some complexity now
and add it later if needed, I'd prefer that)
Jan 04 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 04, 2013 at 03:00:10PM +0100, Johannes Pfau wrote:

 Am Fri, 04 Jan 2013 07:59:05 +0100
 schrieb "Rob T" <rob ucora.com>:

 
 I'm not certain what the major difference is between (1) and (2), 
 because they both sound very nearly identical.




That's because they are essentially two ways of implementing the same
concept.



 In both cases, they seem to provide the same goals of stabilizing
 the next release, while allowing development to proceed unhindered,
 so if (2) can be made to achieve the goal I described for attracting
 real world beta testers, it may not matter which is used, however I
 would suggest that we always focus on serving the needs of the end
 user as a top priority as much as is possible. D requires plenty of
 significant end users working on significant projects.
 
 You did mention that (1) is more suitable for beta testers, so I ask
 how is (2) not as suitable?




I don't think the difference is that serious. (1) is just more
_convenient_ for beta testers, because there's an actual branch called
'staging', so they just have to grab that. (2) requires that they know
which version branch is the latest, which again, isn't that big of a
deal.


[...]

 http://wiki.dlang.org/User:Jpf/Release_Process
 
 I updated the page with some more git commands, ascii art and color so
 it's hopefully easier to understand now. Please tell me if I got
 anything wrong. The only thing I (intended to change) changed is that I
 removed staging.


I like what you did with the page. I think it makes it a lot clearer. I
read through "Release Schedule" and "Branching model", and I think it
pretty much captures what I described as approach (2).



 Although I can see the benefits of staging as you describe it, we can
 get most of those by just creating a new release/version branch as
 soon as the current release is released. This removes some complexity
 from the workflow.


If this approach is more suitable for the core devs, then I think we
should go with it.



 AFAIK it should be easy enough to introduce staging later on, when
 everyone is comfortable with the new workflow. (Please consider that
 many of us are not git experts - so if we can avoid some complexity
 now and add it later if needed, I'd prefer that)


Whether or not one is a git expert ought not matter, if we define the
process sufficiently clearly that one can simply look up the wiki page
and type in the commands as-is, like a script, as Walter puts it. The
important point is to agree on a single implementation of the process,
and supply complete and unambiguous series of git commands to carry out
the process.


T

-- 
The fact that anyone still uses AOL shows that even the presence of
options doesn't stop some people from picking the pessimal one. - Mike
Ellis
Jan 04 2013
parent Johannes Pfau <nospam example.com> writes:
Am Fri, 4 Jan 2013 07:46:55 -0800
schrieb "H. S. Teoh" <hsteoh quickfur.ath.cx>:


 
 I like what you did with the page. I think it makes it a lot clearer.
 I read through "Release Schedule" and "Branching model", and I think
 it pretty much captures what I described as approach (2).
 


Yes. The original workflow described in the wiki should be very
similar, except that I removed staging.


 If this approach is more suitable for the core devs, then I think we
 should go with it.


I also created a version with staging:
http://wiki.dlang.org/User:Jpf/Release_Process_Staging

It occurred to me that there is one detil where staging is really
useful: If we have bugfix pull requests which target a version branch
which is stabilizing (e.g 2.062) that's fine. But if those aren't
merged when 2.062 is released, all open pull requests will still target
2.062 but they should target 2.063 instead. So we have to cycle through
all pull requests and ask the contributors to retarget 2.063.

With staging bug fixes can just always target staging and this is not
an issue.


I think in the end the core devs should choose. But it seems like
an extra staging is more difficult to understand but can avoid some
additional work in long term.


 Whether or not one is a git expert ought not matter, if we define the
 process sufficiently clearly that one can simply look up the wiki page
 and type in the commands as-is, like a script, as Walter puts it. The
 important point is to agree on a single implementation of the process,
 and supply complete and unambiguous series of git commands to carry
 out the process.


I agree. I try to do that on the wiki page, but I can't 
guarantee that it's complete.
Jan 04 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/3/13 3:17 PM, Rob T wrote:

 On Thursday, 3 January 2013 at 19:19:49 UTC, Andrei Alexandrescu wrote:

 On 1/3/13 1:58 PM, Walter Bright wrote:

 As I suggested to Jacob, if the wiki lists git command sequences, it
 should be complete (like a script), and not full of assumptions about
 other commands that need to be inserted.


 I think this is a pertinent point - the process proposed at github is
 incomplete and scantily motivated. Can the experts make one more pass
 through it?

 Thanks,

 Andrei


 I'm rather concerned when I see comments that suggest that the purpose
 of the staging branch was not understood in the slightest. There's a lot
 of discussion on the issue here
 http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com

 The people who are attempting to use the new process have to at least
 take some time to read through the process discussion threads, certainly
 that is not ideal, and the wiki should explain everything much better.


That is correct. The discussion was the crucible and the wiki page 
should be the result of what boiled down into it. There continues to be 
a bit a debate on the exact details of the procedure to be followed, 
which has naturally led to a scant wiki page.

That all is natural and expected - no need for it to cause concern.


 The reason for a less than adequate process description, is that the
 process was not yet fully completed and was difficult to complete
 without a dry run to work out the bugs and oversights etc. Instead of a
 dry run, we're experiencing the real thing, started in the middle of the
 holidays to boot.

 We have to expect some problems of course!


There we go :o).


 It would be nice to contain everything under one discussion thread so
 that the same people who started this whole thing are more likely to
 notice the complaints and misunderstandings and so forth.


It's funny - that was my exact intent when I started the thread "Next 
focus: PROCESS"!


Andrei
Jan 03 2013
parent "Rob T" <rob ucora.com> writes:
On Thursday, 3 January 2013 at 22:16:07 UTC, Andrei Alexandrescu 
wrote:

 It's funny - that was my exact intent when I started the thread 
 "Next focus: PROCESS"!


 Andrei


I know that was your intent and we tried hard to stay focused in 
that thread, and it was working reasonably well, but now we have 
another thread to deal with that is disconnected from the first 
thread, resulting in people repeating themselves all over again. 
I know it was rejected a thousand times before, but solving the 
disconnection problem was one reason for asking to move the forum 
into a web based version that could be managed (ie this type of 
forum cannot be managed at all it seems). The other alternative 
was to discuss everything in the wiki page, but people found that 
to be not so good as can be seen by looking at the wiki as well 
as the comments concerning its use for that purpose.

It would be nice to have a better solution for the community to 
perform cooperative problem solving than what we have now, which 
also leads on to resolving other problems such as the endless 
repeating disconnected discussions on other subjects of 
importance. I don't want to stray off topic, so just keep in mind 
that there may be a more significant issue that should be 
resolved immediately after we get past the current one.

--rt
Jan 03 2013
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/3/2013 12:17 PM, Rob T wrote:

 I'm rather concerned when I see comments that suggest that the purpose of the
 staging branch was not understood in the slightest. There's a lot of discussion
 on the issue here http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com

 The people who are attempting to use the new process have to at least take some
 time to read through the process discussion threads, certainly that is not
 ideal, and the wiki should explain everything much better.


I agree that the pertinent rationales must be in the wiki.
Jan 03 2013
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/3/2013 11:19 AM, Andrei Alexandrescu wrote:

 I think this is a pertinent point - the process proposed at github is
incomplete
 and scantily motivated. Can the experts make one more pass through it?



One feature I really liked about github was when you created a new repository, 
it would give you an exact list of corresponding git commands to set up your 
local version.
Jan 03 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 3 January 2013 at 18:58:51 UTC, Walter Bright wrote:

 Turn that around - what's the benefit of keeping it? It's just 
 clutter.


It has been discussed here before, and many people agreed that 
stagging can go away, as changes made in the proposal made it 
quite redundant. That sounds the reasonable thing to do.
Jan 04 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 15:55:01 +0100
schrieb "deadalnix" <deadalnix gmail.com>:


 On Thursday, 3 January 2013 at 18:58:51 UTC, Walter Bright wrote:

 Turn that around - what's the benefit of keeping it? It's just 
 clutter.


 
 It has been discussed here before, and many people agreed that 
 stagging can go away, as changes made in the proposal made it 
 quite redundant. That sounds the reasonable thing to do.


I just thought about something - maybe staging isn't 100% redundant.
I assume my conclusions on
http://wiki.dlang.org/User:Jpf/Release_Process are correct:

If we want release stabilization time == time between 2 releases i.e.
we start a new release/version branch after every major release: 

Let's say 2.062 is in stabilization phase and we use the 2.062 branch.
Then bugfixes are based on / merged into 2.062 branch. So someone opens
a pull request targeting the 2.062 branch with a bugfix. For some
reason (lack of time) this request isn't merged till 2.062 is released.
After the release only regression fixes should go into 2.062.
But we now have a bugfix (not regression) pull request targeting the
2.062 branch!

Staging completely avoids that issue. So is this reason enough to have
staging?
Jan 04 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 04, 2013 at 04:14:00PM +0100, Johannes Pfau wrote:
[...]

 I just thought about something - maybe staging isn't 100% redundant.
 I assume my conclusions on
 http://wiki.dlang.org/User:Jpf/Release_Process are correct:
 
 If we want release stabilization time == time between 2 releases i.e.
 we start a new release/version branch after every major release: 
 
 Let's say 2.062 is in stabilization phase and we use the 2.062 branch.
 Then bugfixes are based on / merged into 2.062 branch. So someone
 opens a pull request targeting the 2.062 branch with a bugfix. For
 some reason (lack of time) this request isn't merged till 2.062 is
 released.  After the release only regression fixes should go into
 2.062.  But we now have a bugfix (not regression) pull request
 targeting the 2.062 branch!
 
 Staging completely avoids that issue. So is this reason enough to have
 staging?


I'm not sure, but in my experience at my day jobs, any bugfix that *can*
be applied to master (i.e., the code hasn't diverged so much that the
fix doesn't apply anymore), *should* be. For this reason, it's best to
always base bugfixes on master, unless the code has diverged so much the
fix only applies to a particular release branch. Otherwise you end up
with the problem somebody mentioned, that the bug is fixed in a previous
release, but is "unfixed" in a later release because we neglected to
merge the fix into master.

Of course, it's possible to base the fix on the release branch and merge
it back to master, but the latter step is more prone to be forgotten.
But then again, post-release regression fixes should probably be based
on the release branch, so that we know that the fix works, as master may
have diverged significantly since the original release branch was made.
So either way, we have to make sure fixes are applied back to master.


T

-- 
May you live all the days of your life. -- Jonathan Swift
Jan 04 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 4 Jan 2013 08:20:00 -0800
schrieb "H. S. Teoh" <hsteoh quickfur.ath.cx>:


 
 I'm not sure, but in my experience at my day jobs, any bugfix that
 *can* be applied to master (i.e., the code hasn't diverged so much
 that the fix doesn't apply anymore), *should* be. For this reason,
 it's best to always base bugfixes on master, unless the code has
 diverged so much the fix only applies to a particular release branch.
 Otherwise you end up with the problem somebody mentioned, that the
 bug is fixed in a previous release, but is "unfixed" in a later
 release because we neglected to merge the fix into master.


But if you base bug fixes on master you have to merge master into the
release branch to get the bug fixes. But master may contain newly added
features which you don't want in the release branch.

The more I think about it, staging really seems to be useful to avoid
some corner cases.
Jan 04 2013
parent reply "Rob T" <rob ucora.com> writes:
On Friday, 4 January 2013 at 17:11:54 UTC, Johannes Pfau wrote:

 The more I think about it, staging really seems to be useful to 
 avoid
 some corner cases.


I haven't had time to study the newly proposed process in enough 
detail to properly comment, but it always seemed to me that you 
cannot get away without a staging branch. You can try very hard 
to do without, but it means something else will suffer for the 
lack of it, so it seems you are coming to the same conclusion 
which is good (unless I'm wrong).

If we look at how the current 2.061.0 release went down, there 
was clearly no staging at all, it went straight out just as it 
did in the past, and we're already seeing complaints about broken 
code and bug fixes that should have gone into the release that 
were left out. We're also finding bugs discovered right after the 
release was made, this is not good and should not happen.

The whole point of having a process has been defeated in terms of 
the goal of creating highly stable releases. What has been 
released is in effect a beta release or the same thing as a 
staging release, it is *not* a stable release! If it was a stable 
release, we would not see so many immediate problems.

I realize this was our fist pass through with an incomplete 
process, and almost no buy in from the devs who were to use it, 
so with that said we have clearly made some significant 
improvements, and I do not want to take away from that, however I 
hope we can all refocus on the goal of releasing only high 
quality software, not half baked betas.

Thanks!

--rt
Jan 04 2013
next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 21:01:41 Rob T wrote:

 If we look at how the current 2.061.0 release went down, there
 was clearly no staging at all, it went straight out just as it
 did in the past, and we're already seeing complaints about broken
 code and bug fixes that should have gone into the release that
 were left out. We're also finding bugs discovered right after the
 release was made, this is not good and should not happen.


That's what the beta is for, which we did. Clearly, not enough people tried it 
out, or those regressions would have been caught. Creating a separate branch 
would not fix that.

- Jonathan M Davis
Jan 04 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-04 21:11, Jonathan M Davis wrote:


 That's what the beta is for, which we did. Clearly, not enough people tried it
 out, or those regressions would have been caught. Creating a separate branch
 would not fix that.


It fixes other problems, like merging a pull request that should have 
been merge just because someone has missed a beta was out.

-- 
/Jacob Carlborg
Jan 05 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-01-05 12:26, Jacob Carlborg wrote:

 On 2013-01-04 21:11, Jonathan M Davis wrote:


 That's what the beta is for, which we did. Clearly, not enough people
 tried it
 out, or those regressions would have been caught. Creating a separate
 branch
 would not fix that.


 It fixes other problems, like merging a pull request that should have
 been merge just because someone has missed a beta was out.


should _not_.

-- 
/Jacob Carlborg
Jan 05 2013
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
05-Jan-2013 00:01, Rob T пишет:

 On Friday, 4 January 2013 at 17:11:54 UTC, Johannes Pfau wrote:

 The more I think about it, staging really seems to be useful to avoid
 some corner cases.


 I haven't had time to study the newly proposed process in enough detail
 to properly comment, but it always seemed to me that you cannot get away
 without a staging branch. You can try very hard to do without, but it
 means something else will suffer for the lack of it, so it seems you are
 coming to the same conclusion which is good (unless I'm wrong).

 If we look at how the current 2.061.0 release went down, there was
 clearly no staging at all, it went straight out just as it did in the
 past, and we're already seeing complaints about broken code and bug
 fixes that should have gone into the release that were left out. We're
 also finding bugs discovered right after the release was made, this is
 not good and should not happen.


Staging branch was there, so did the betas.
What could have helped is announcing beta at D.announce instead of 
separate list. Plus at holidays there was not much of beta test coverage.


-- 
Dmitry Olshansky
Jan 04 2013
parent reply "Rob T" <rob ucora.com> writes:
On Friday, 4 January 2013 at 20:41:47 UTC, Dmitry Olshansky wrote:

 Staging branch was there, so did the betas.
 What could have helped is announcing beta at D.announce instead 
 of separate list. Plus at holidays there was not much of beta 
 test coverage.


Yes, being the first pass through with an incomplete and poorly 
understood process didn't help, along with the timing, so the 
result is what the process produced in its current form.

We're definitely making progress however, and hopefully the 
criticisms are taken constructively as they should be. We just 
want to see things improve, that's all.

One improvement for next pass through, is to release installable 
snapshots of the next beta directly on the download page (with 
appropriate labeling) so that it is trivially easy for end users 
to download and install the latest beta for testing.

Also, as I mentioned in my previous post, we ought to define what 
"stable" means so that we have a much better way to judge if the 
current beta is ready for release or not. As it sits, we have no 
definition, so it's not surprising that some people thought the 
current version was ready, while others are not so sure of that.

--rt
Jan 04 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Jan 05, 2013 at 12:02:39AM +0100, Rob T wrote:
[...]

 We're definitely making progress however, and hopefully the
 criticisms are taken constructively as they should be. We just want
 to see things improve, that's all.
 
 One improvement for next pass through, is to release installable
 snapshots of the next beta directly on the download page (with
 appropriate labeling) so that it is trivially easy for end users to
 download and install the latest beta for testing.


[...]

+1. More beta test coverage can only be a good thing.


T

-- 
Life is complex. It consists of real and imaginary parts. -- YHL
Jan 04 2013
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 21:01:41 +0100
schrieb "Rob T" <rob ucora.com>:


 On Friday, 4 January 2013 at 17:11:54 UTC, Johannes Pfau wrote:

 The more I think about it, staging really seems to be useful to 
 avoid
 some corner cases.


 
 I haven't had time to study the newly proposed process in enough 
 detail to properly comment, but it always seemed to me that you 
 cannot get away without a staging branch. You can try very hard 
 to do without, but it means something else will suffer for the 
 lack of it, so it seems you are coming to the same conclusion 
 which is good (unless I'm wrong).


Yep. As we have lost most attention in this thread(most people have
probably set "ignore this thread") we should check if we all agree on
the process as outlined on the updated wiki page (I've updated the
original page now) and improve it, if necessary, do some proof reading /
spell checking and then open a new thread, asking for comments from the
main devs.
Jan 05 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Jan 05, 2013 at 11:42:23AM +0100, Johannes Pfau wrote:

 Am Fri, 04 Jan 2013 21:01:41 +0100
 schrieb "Rob T" <rob ucora.com>:
 

 On Friday, 4 January 2013 at 17:11:54 UTC, Johannes Pfau wrote:

 The more I think about it, staging really seems to be useful to
 avoid some corner cases.


 
 I haven't had time to study the newly proposed process in enough 
 detail to properly comment, but it always seemed to me that you 
 cannot get away without a staging branch. You can try very hard 
 to do without, but it means something else will suffer for the 
 lack of it, so it seems you are coming to the same conclusion 
 which is good (unless I'm wrong).


 
 Yep. As we have lost most attention in this thread(most people have
 probably set "ignore this thread") we should check if we all agree on
 the process as outlined on the updated wiki page (I've updated the
 original page now) and improve it, if necessary, do some proof reading
 / spell checking and then open a new thread, asking for comments from
 the main devs.


I like the new page!!

One question, though: in the git workflow section, it says that a
non-regression bugfix should go directly into staging, but there's no
mention of merging it back to master. Is this deliberate? It seems to me
to be kinda important that we merge all fixes back to master so that
bugs don't get "unfixed". Or am I missing something here?


T

-- 
Why waste time learning, when ignorance is instantaneous? -- Hobbes, from
Calvin & Hobbes
Jan 05 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Sat, 5 Jan 2013 10:21:30 -0800
schrieb "H. S. Teoh" <hsteoh quickfur.ath.cx>:


 
 One question, though: in the git workflow section, it says that a
 non-regression bugfix should go directly into staging, but there's no
 mention of merging it back to master. Is this deliberate? It seems to
 me to be kinda important that we merge all fixes back to master so
 that bugs don't get "unfixed". Or am I missing something here?
 


It's mentioned in the official branches section and in the summary
"Regression fixes and bug fixes will be merged from staging into
master." and "Merge staging branch into master regularly", but it
should be added to the workflow as well.

I don't think it's urgent to immediately merge it into master though
and bugs can't really get 'unfixed' anyway: Releases are made from
staging so the fix will be part of releases either way. It's enough to
sporadically merge staging to master so that the feature branches and
master also get the bug fixes.
Jan 05 2013
parent reply "Rob T" <rob ucora.com> writes:
One quick note, is that we may want to change the title and 
description, since I think we're taking about a "Development and 
Release Process" rather than only a "Release Process".

The new wiki, and even the process page should be linked in on 
the main page to make it easy for the devs to get access to it.

--rt
Jan 05 2013
parent Johannes Pfau <nospam example.com> writes:
Am Sun, 06 Jan 2013 00:18:16 +0100
schrieb "Rob T" <rob ucora.com>:


 One quick note, is that we may want to change the title and 
 description,


Done.
http://wiki.dlang.org/Development_and_Release_Process

I also verified and corrected the git commands. They should now all work
as expected and can be used like a script.

Do you agree with the process as described on that page now? Would be
great to also get some feedback from other contributors of the original
wiki page.

I wonder whether a section should be added to the wiki explaining the
transition from current workflow to the new workflow. For the first
release we should do an additional merge from master into staging in
~4weeks as otherwise there won't be any new features in 2.062.

And I'd recommend merging staging into master, then deleting staging
and creating it again as we don't know whether staging already has
cherry picked commits etc?

If we want to maintain the current release we might have to use
cherry-picking this one time as I think there are already regression
fixes only pushed to master / staging which should be in the version
branch.
Jan 06 2013
prev sibling parent reply "Jesse Phillips" <Jessekphillips+D gmail.com> writes:
On Thursday, 3 January 2013 at 17:56:05 UTC, Johannes Pfau wrote:

 What changes? All changes should be made in master, then 
 applied to
 staging via cherry picking. So there should never be changes in 
 staging
 that are not in master so there's nothing that could be 
 overwritten?


I strongly disagree here. We want pull requests to be made 
against the oldest supported/relevant branch. This could be 
merged to both staging and master (which can simply be done by 
merging staging into master) there is no need to complicate 
things with cherry picking.

As for history, I think that clearly shows what the history is, 
staging gets fixes and master gets staging that is what we want.
Jan 09 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 09 Jan 2013 18:37:15 +0100
schrieb "Jesse Phillips" <Jessekphillips+D gmail.com>:


 On Thursday, 3 January 2013 at 17:56:05 UTC, Johannes Pfau wrote:

 What changes? All changes should be made in master, then 
 applied to
 staging via cherry picking. So there should never be changes in 
 staging
 that are not in master so there's nothing that could be 
 overwritten?


 
 I strongly disagree here. We want pull requests to be made 
 against the oldest supported/relevant branch. This could be 
 merged to both staging and master (which can simply be done by 
 merging staging into master) there is no need to complicate 
 things with cherry picking.
 
 As for history, I think that clearly shows what the history is, 
 staging gets fixes and master gets staging that is what we want.


You replied to one of my oldest posts in this discussion ;-)

I didn't understand the staging concept when I wrote that - I think
I've understood most of it now, though. I think the old release process
wiki page wasn't clear enough.

I've updated the http://wiki.dlang.org/Development_and_Release_Process
page last week (I even added a "do not cherry pick from one official
branch into another" rule), so you won't have to convince me ;-)

Do you think the wiki page is good / correct? I've verified that all
git commands work, so if everyone agrees with the workflow (should be
the same as on the old page) we can probably start a new thread and ask
for comments from the core devs.
Jan 09 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jan 09, 2013 at 09:40:59PM +0100, Johannes Pfau wrote:
[...]

 I've updated the http://wiki.dlang.org/Development_and_Release_Process
 page last week (I even added a "do not cherry pick from one official
 branch into another" rule), so you won't have to convince me ;-)
 
 Do you think the wiki page is good / correct? I've verified that all
 git commands work, so if everyone agrees with the workflow (should be
 the same as on the old page) we can probably start a new thread and
 ask for comments from the core devs.


The earlier we can get input from the core devs, the better. Otherwise
we're just building castles in the air.


T

-- 
It only takes one twig to burn down a forest.
Jan 09 2013
prev sibling next sibling parent "alex" <info alexanderbothe.com> writes:
Couldn't we just make an auto-publishing cron-job service or so 
which periodically (once a day or even just after a push/merge) 
compiles dmd and everything, packs it and makes this archive 
downloadable then?

Of course this literally rolling release should be marked as 
unstable/nightly build. Anyway, if a halfway stable state has 
been reached or if the community thinks that it's enough for a 
version bump,
the latest build is simply tagged and marked as latest stable 
version.
How easy is that?
Jan 03 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-01-03 01:58, Walter Bright wrote:

 As always, when I try to do a release, problems crop up. For example,
 the github procedure agreed upon and outlined here:


Don't know if this is the exact model we want to follow but here are 
some commands:

https://github.com/nvie/gitflow

These are based on "A successful Git branching model" that has been 
talked about before:

http://nvie.com/posts/a-successful-git-branching-model/

-- 
/Jacob Carlborg
Jan 04 2013
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 3 January 2013 at 00:59:04 UTC, Walter Bright wrote:

 As always, when I try to do a release, problems crop up. For 
 example,
 the github procedure agreed upon and outlined here:

 http://wiki.dlang.org/Proposed_new_D_development_process#Release_a_new_version_of_D

 Issues:

 1. you cannot have a tag and a branch with the same name. At 
 least, you cannot push them with:

     git push origin 2.N+1

 because it's ambiguous. So I prepended a v to the tag name.


Let's get a very practical case here. 2.061 have been released, 
and Kenji already fixed a bug in it (unreachable statement issue).

Some people (including me, but I'm not the only one) would be 
interested by a new revision of dmd with that fix.

It is why the proposal include branches as 2.N and revision as 
2.N.M . So the same version of D, with bugfixes can be published.

The branch has a 2.N form, the tag has a 2.N.M form.
Jan 04 2013
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 15:26:20 deadalnix wrote:

 On Thursday, 3 January 2013 at 00:59:04 UTC, Walter Bright wrote:

 As always, when I try to do a release, problems crop up. For
 example,
 the github procedure agreed upon and outlined here:
 
 http://wiki.dlang.org/Proposed_new_D_development_process#Release_a_new_ver
 sion_of_D
 
 Issues:
 
 1. you cannot have a tag and a branch with the same name. At
 
 least, you cannot push them with:
     git push origin 2.N+1
 
 because it's ambiguous. So I prepended a v to the tag name.


 
 Let's get a very practical case here. 2.061 have been released,
 and Kenji already fixed a bug in it (unreachable statement issue).
 
 Some people (including me, but I'm not the only one) would be
 interested by a new revision of dmd with that fix.
 
 It is why the proposal include branches as 2.N and revision as
 2.N.M . So the same version of D, with bugfixes can be published.
 
 The branch has a 2.N form, the tag has a 2.N.M form.


And why would that particular bug get a release as opposed to another? Why 
wouldit be special? Or are you proposing that we do a 2.N.M release for every 
bug fix? That seems insane. Why wouldn't you just use master if you want the 
latest?

- Jonathan M Davis
Jan 04 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 4 January 2013 at 16:19:45 UTC, Jonathan M Davis wrote:

 And why would that particular bug get a release as opposed to 
 another? Why
 wouldit be special? Or are you proposing that we do a 2.N.M 
 release for every
 bug fix? That seems insane. Why wouldn't you just use master if 
 you want the
 latest?


That is not latest, that is bug fixes. That is the whole point of 
changing process : being able to publish bug fixes without new 
features, so users don't need to wait mounth to get their bug 
fixed and get bunch of unrelated stuff with the fix that may 
introduce bugs as well.

I took this precise example, but I'm not suggesting that a 
revision is made for EVERY bug fix. Only for severe bugs 
(regressions, corrupted codegen, . . .) or when the quantity is 
judged enough.
Jan 04 2013
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 04, 2013 at 08:18:52AM -0800, Jonathan M Davis wrote:

 On Friday, January 04, 2013 15:26:20 deadalnix wrote:


[...]

 Let's get a very practical case here. 2.061 have been released, and
 Kenji already fixed a bug in it (unreachable statement issue).
 
 Some people (including me, but I'm not the only one) would be
 interested by a new revision of dmd with that fix.
 
 It is why the proposal include branches as 2.N and revision as
 2.N.M . So the same version of D, with bugfixes can be published.
 
 The branch has a 2.N form, the tag has a 2.N.M form.


 
 And why would that particular bug get a release as opposed to another?


It doesn't. It just gets merged in to the 2.N branch post-release, and
once enough fixes have accumulated, we make a 2.N.M release.



 Why wouldit be special? Or are you proposing that we do a 2.N.M
 release for every bug fix? That seems insane.


Some bugs are important enough to push out to the release branch, e.g.
regressions or blocker bugs. Nobody is suggesting we make a new release
for *every* bugfix pushed to the release branch -- that *would* be
insane. The release manager, whoever that person is, makes the judgment
call on when a release branch has enough bugfixes to warrant another
2.N.M release. That can be just a single bug, if it's a critical blocker
bug, or it can be 20 bugs, if none of them are too important, but, taken
together, constitute a significant improvement over the original
release.



 Why wouldn't you just use master if you want the latest?


Because master potentially has breaking changes that one may not want to
deal with. E.g. property enforcement, or breaking syntax changes, etc..
Enterprises who adopt D will want to stay with a stable release that
won't randomly break code all over the place just because a particular
bugfix was needed. That's why it's important to push regressions and
blockers to release branches, without the new features in master.


T

-- 
Doubt is a self-fulfilling prophecy.
Jan 04 2013
parent Johannes Pfau <nospam example.com> writes:
Am Fri, 4 Jan 2013 08:31:57 -0800
schrieb "H. S. Teoh" <hsteoh quickfur.ath.cx>:


 On Fri, Jan 04, 2013 at 08:18:52AM -0800, Jonathan M Davis wrote:

 On Friday, January 04, 2013 15:26:20 deadalnix wrote:

 It is why the proposal include branches as 2.N and revision as
 2.N.M . So the same version of D, with bugfixes can be published.
 
 The branch has a 2.N form, the tag has a 2.N.M form.


 
 And why would that particular bug get a release as opposed to
 another?




I would propose to only merge regression fixes into minor releases. At
least for dmd every bug fix also has potential to break user code. I
think if 2.061 compiled some code, 2.061.1 should as well and the only
way to ensure that is by only merging regression fixes (regressions
which started to occur in 2.061).

The decision what fixed bug / regression justifies an immediate release
is up to the release manager. But to make things simple we can just
have a fixed schedule for less urgent fixes: release a update every x
weeks.
Jan 04 2013
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 08:23:58 H. S. Teoh wrote:

 On Fri, Jan 04, 2013 at 08:15:34AM -0800, Jonathan M Davis wrote:

 On Friday, January 04, 2013 15:20:02 Johannes Pfau wrote:

 And most important: Were the main developers part of that discussion
 / have they agreed? While I agree LTS would be nice it needs someone
 to do it. You also have to consider the available man power.


 
 For the most part, I don't think that they were. I think that Andrei
 was really the only one of us involved at all, though I'd have to look
 at the thread again to be sure.


 
 [...]
 
 We solicited for core dev involvement in that thread, but AFAIK, there
 was only limited participation. It felt like we were just talking about
 castles in the air, because the people who would actually be
 implementing the process didn't say very much at all. So I wasn't at all
 surprised at the misunderstandings in this thread.
 
 Still, I think we're finally converging on the same page here, so that's
 a good thing.


The thread was way too big, and I had too little time to get into that 
conversation, which is why I wasn't involved.

- Jonathan M Davis
Jan 04 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 08:35:17 -0800
schrieb Jonathan M Davis <jmdavisProg gmx.com>:


 The thread was way too big, and I had too little time to get into
 that conversation, which is why I wasn't involved.
 
 - Jonathan M Davis


I've restructured the wiki page here:
http://wiki.dlang.org/User:Jpf/Release_Process_Staging

Do you think that's clearer?

(To my understanding it's the same process as proposed on the
"official" wiki page)
Jan 04 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 4 January 2013 at 17:26:02 UTC, Johannes Pfau wrote:

 Am Fri, 04 Jan 2013 08:35:17 -0800
 schrieb Jonathan M Davis <jmdavisProg gmx.com>:


 The thread was way too big, and I had too little time to get 
 into
 that conversation, which is why I wasn't involved.
 
 - Jonathan M Davis


 I've restructured the wiki page here:
 http://wiki.dlang.org/User:Jpf/Release_Process_Staging

 Do you think that's clearer?


This is very good, however, some thing are still unclear.

First, regression fix don't make any sense to me. You suggest to 
fix bug in master and fix regression in older branches. This 
should be the opposite IMO.

A regression is something that used to work, but don't work 
anymore. So Correcting them in older version seems kind of 
contradictory, especially when other bugs aren't.

I don't think separating those 2 type of bugs is really 
beneficial and I would keep only the process for regression fixes.

Secondly, in every git commands block you have all the commands 
to set-up the repository. I think this belong to its own 
paragraph or even its own page. Then I'd replace them with git 
remote update in git command block.

Finally, it is weird that we have v2.062 -> v2.062.1 . I'd 
prefers v2.062.0 for the first one.

I like very much the ASCII art on top, which make thing really 
clear. Overall, it is better that the actual version on the wiki.
Jan 04 2013
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 04, 2013 at 07:35:17PM +0100, deadalnix wrote:

 On Friday, 4 January 2013 at 17:26:02 UTC, Johannes Pfau wrote:


[...]

I've restructured the wiki page here:
http://wiki.dlang.org/User:Jpf/Release_Process_Staging

Do you think that's clearer?




[...]

 I like very much the ASCII art on top, which make thing really
 clear. Overall, it is better that the actual version on the wiki.


I think we should replace the actual version with this one. No point
working on it as a user page when it's what's going to be implemented
anyway. Wiki history will let us go back to review the old version if
necessary.


T

-- 
Life is too short to run proprietary software. -- Bdale Garbee
Jan 04 2013
prev sibling next sibling parent reply "Rob T" <rob ucora.com> writes:
On Friday, 4 January 2013 at 18:35:18 UTC, deadalnix wrote:

 On Friday, 4 January 2013 at 17:26:02 UTC, Johannes Pfau wrote:

 Am Fri, 04 Jan 2013 08:35:17 -0800
 schrieb Jonathan M Davis <jmdavisProg gmx.com>:


 The thread was way too big, and I had too little time to get 
 into
 that conversation, which is why I wasn't involved.
 
 - Jonathan M Davis






OK, but it would have been a lot shorter had you and the other 
devs become involved ;)


 First, regression fix don't make any sense to me. You suggest 
 to fix bug in master and fix regression in older branches. This 
 should be the opposite IMO.

 A regression is something that used to work, but don't work 
 anymore. So Correcting them in older version seems kind of 
 contradictory, especially when other bugs aren't.

 I don't think separating those 2 type of bugs is really 
 beneficial and I would keep only the process for regression 
 fixes.


Terminology aside, we need to differentiate between bug fixes 
that do not introduce new bugs vs bug fixes that may introduce 
new bugs *or* break existing code for the current release 
version. We also *have* to prevent new features and structural 
changes (destabilizers) from leaking into a stable release.


 Secondly, in every git commands block you have all the commands 
 to set-up the repository. I think this belong to its own 
 paragraph or even its own page. Then I'd replace them with git 
 remote update in git command block.

 Finally, it is weird that we have v2.062 -> v2.062.1 . I'd 
 prefers v2.062.0 for the first one.


Absolutely! Otherwise someone is going to think v2.062 is greater 
than v2.062.1. Guaranteed. I already got semi-confused looking at 
the latest download page and I know what's going on far more than 
Joe Smith who walks in tomorrow checking out D for the first time.


 I like very much the ASCII art on top, which make thing really 
 clear. Overall, it is better that the actual version on the 
 wiki.


Looks good yes.

--rt
Jan 04 2013
next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 20:30:28 Rob T wrote:

 Absolutely! Otherwise someone is going to think v2.062 is greater
 than v2.062.1. Guaranteed. I already got semi-confused looking at
 the latest download page and I know what's going on far more than
 Joe Smith who walks in tomorrow checking out D for the first time.


Really? Why on earth would you think that 2.062 was greater than 2.062.1? 
Also, I believe that it's very common with Linux packages (and probably the 
projects themselves) to do that sort of versioning where there's never a .0 
and the last part only gets added when you actually get a .1.

- Jonathan M Davis
Jan 04 2013
next sibling parent reply "Rob T" <rob ucora.com> writes:
On Friday, 4 January 2013 at 19:59:19 UTC, Jonathan M Davis wrote:

 Really? Why on earth would you think that 2.062 was greater 
 than 2.062.1?


I was asking for clarity so that no one can possibly get confused.

If you look at the download page, the .0 is missing on some of 
the packages, but shows up as a -0 on some of the others, and 
that is simply confusing and totally unnecessary. If it is 
necessary for some reason, then it needs to be explained.


 Also, I believe that it's very common with Linux packages (and 
 probably the
 projects themselves) to do that sort of versioning where 
 there's never a .0
 and the last part only gets added when you actually get a .1.


There's no law that states that we must follow old conventions.

--rt
Jan 04 2013
next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 21:10:32 Rob T wrote:

 On Friday, 4 January 2013 at 19:59:19 UTC, Jonathan M Davis wrote:

 Really? Why on earth would you think that 2.062 was greater
 than 2.062.1?


 
 I was asking for clarity so that no one can possibly get confused.
 
 If you look at the download page, the .0 is missing on some of
 the packages, but shows up as a -0 on some of the others, and
 that is simply confusing and totally unnecessary. If it is
 necessary for some reason, then it needs to be explained.
 

 Also, I believe that it's very common with Linux packages (and
 probably the
 projects themselves) to do that sort of versioning where
 there's never a .0
 and the last part only gets added when you actually get a .1.


 
 There's no law that states that we must follow old conventions.


True, but you also shouldn't do something different just to do something 
different. You need a good reason.

I think that it's pretty ugly to have 2.062.0, and in my experience, that's a 
very abnormal thing to do. You have 2.062 followed by 2.062.1 if you ever have 
any point releases, but you don't start with .0. I don't recall ever seeing 
that before.

And the fact that with have -0 on the latest release on the download page is
downright bizarre too. I don't think that I've ever seen -0 before either. I'd
expect you to add the -1 if you ever need one but not start with -0. And
actually, looking at my Linux box now, it looks like Arch starts with -1, not
-0. I don't know what other distros do.

Regardless, I'm not at all in favor of having .0 on any release. Only add
the minor versions to releases which are minor versions, not to the major
ones.

- Jonathan M Davis
Jan 04 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 4 January 2013 at 20:48:34 UTC, Jonathan M Davis wrote:

 On Friday, January 04, 2013 21:10:32 Rob T wrote:

 On Friday, 4 January 2013 at 19:59:19 UTC, Jonathan M Davis 
 wrote:

 Really? Why on earth would you think that 2.062 was greater
 than 2.062.1?


 
 I was asking for clarity so that no one can possibly get 
 confused.
 
 If you look at the download page, the .0 is missing on some of
 the packages, but shows up as a -0 on some of the others, and
 that is simply confusing and totally unnecessary. If it is
 necessary for some reason, then it needs to be explained.
 

 Also, I believe that it's very common with Linux packages 
 (and
 probably the
 projects themselves) to do that sort of versioning where
 there's never a .0
 and the last part only gets added when you actually get a .1.


 
 There's no law that states that we must follow old conventions.


 True, but you also shouldn't do something different just to do 
 something
 different. You need a good reason.

 I think that it's pretty ugly to have 2.062.0, and in my 
 experience, that's a
 very abnormal thing to do. You have 2.062 followed by 2.062.1 
 if you ever have
 any point releases, but you don't start with .0. I don't recall 
 ever seeing
 that before.

 And the fact that with have -0 on the latest release on the 
 download page is
 downright bizarre too. I don't think that I've ever seen -0 
 before either. I'd
 expect you to add the -1 if you ever need one but not start 
 with -0. And
 actually, looking at my Linux box now, it looks like Arch 
 starts with -1, not
 -0. I don't know what other distros do.

 Regardless, I'm not at all in favor of having .0 on any 
 release. Only add
 the minor versions to releases which are minor versions, not to 
 the major
 ones.

 - Jonathan M Davis


I don't think anybody really care if this start with 0 or 1. What 
is weird is that you'll find 2 numbers versions and 3 numbers 
one, which is confusing (and I never saw that in any software).
Jan 04 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 4 January 2013 at 21:02:45 UTC, deadalnix wrote:

 I don't think anybody really care if this start with 0 or 1. 
 What is weird is that you'll find 2 numbers versions and 3 
 numbers one, which is confusing (and I never saw that in any 
 software).


Looking at several software and what they do, it seems that 
starting with 0 and with 1 are both fairly common.

WE ARE DEV, WE START COUNTING AT 0 !
Jan 04 2013
parent reply "Rob T" <rob ucora.com> writes:
On Friday, 4 January 2013 at 21:05:42 UTC, deadalnix wrote:

 On Friday, 4 January 2013 at 21:02:45 UTC, deadalnix wrote:

 I don't think anybody really care if this start with 0 or 1. 
 What is weird is that you'll find 2 numbers versions and 3 
 numbers one, which is confusing (and I never saw that in any 
 software).


 Looking at several software and what they do, it seems that 
 starting with 0 and with 1 are both fairly common.

 WE ARE DEV, WE START COUNTING AT 0 !


Even better is to also identify in the version sequence, what is 
a beta release and what is not.

AFAIAC the current 2.061 release is in a beta stage because it is 
not yet "stable". The question though, is what does "stable" mean?

For a definition, I propose something like:

All known critical bugs have been resolved, and a certain 
percentage [to be determined] of all known non-critical bugs have 
been resolved, or some function thereof. We can settle on 
something I'm sure, but right now we have no definition of what 
stable means, so that's perhaps one reason why new releases are 
more buggy than I would expect them to be. But what does that 
mean? It means that I would *not* use the new release for 
anything that mattered in a production environment, not until it 
stabilized to a much higher standard than it currently is at.

--rt
Jan 04 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 04, 2013 at 11:54:32PM +0100, Rob T wrote:
[...]

 Even better is to also identify in the version sequence, what is a
 beta release and what is not.
 
 AFAIAC the current 2.061 release is in a beta stage because it is
 not yet "stable". The question though, is what does "stable" mean?
 
 For a definition, I propose something like:
 
 All known critical bugs have been resolved, and a certain percentage
 [to be determined] of all known non-critical bugs have been
 resolved, or some function thereof. We can settle on something I'm
 sure, but right now we have no definition of what stable means, so
 that's perhaps one reason why new releases are more buggy than I
 would expect them to be. But what does that mean? It means that I
 would *not* use the new release for anything that mattered in a
 production environment, not until it stabilized to a much higher
 standard than it currently is at.


[...]

The problem is, what constitutes "stable" is a judgment call, because
which bugs are critical (or, in this case, release-critical) are also a
judgment call.

So we would need a delegated Release Manager who decides when a
particular release branch is ready to be released, and who holds high
standards of what constitutes "release-ready".

Or have some kind of voting system such that some fixed percentage of
the top-voted bugs must be fixed before something can be released.  The
bug tracker already has a voting system, but (1) people don't pay
attention to it, so the number of votes a bug gets doesn't seem to
correlate with its likelihood of getting fixed; and (2) the number of
votes you can cast is arbitrarily limited to 10, which discourages
people from using the system.


T

-- 
Don't get stuck in a closet---wear yourself out.
Jan 04 2013
parent "Rob T" <rob ucora.com> writes:
On Saturday, 5 January 2013 at 01:00:28 UTC, H. S. Teoh wrote:

 The problem is, what constitutes "stable" is a judgment call, 
 because
 which bugs are critical (or, in this case, release-critical) 
 are also a
 judgment call.

 So we would need a delegated Release Manager who decides when a
 particular release branch is ready to be released, and who 
 holds high
 standards of what constitutes "release-ready".

 [...]


Good points.

As it stands right now, someone *has* to make a decision when to 
release, so we in effect have a "Release Manager" already, 
although who is making the decision and how the decision is being 
made is not clear at all. Personally, I would not want to be in 
that position because I would have nothing to guide me when 
deciding to make a release (or not), and if I made a mistake and 
released too soon or too late, then all the abuse for the mistake 
will be directed at me personally. I'd much rather have angry 
people place blame on a less than adequate process instead.

Processes are much easier to improve on and no one but the 
process itself is too blame when a process fails to deliver.

The point is, any definitions we come up with will be better than 
absolutely no definitions at all. For example the process as it 
is being defined, is making a positive impact even in its 
embryonic state. It allows us to look back at past 
less-than-perfect results, and move ahead with further 
incremental improvements. No one is to blame for the results, so 
we focus on the process improvements instead of pointing fingers 
at people.

--rt
Jan 04 2013
prev sibling parent "Jesse Phillips" <Jessekphillips+D gmail.com> writes:
On Friday, 4 January 2013 at 20:10:33 UTC, Rob T wrote:

 If you look at the download page, the .0 is missing on some of 
 the packages, but shows up as a -0 on some of the others, and 
 that is simply confusing and totally unnecessary. If it is 
 necessary for some reason, then it needs to be explained.


The -0 is a packaging version, if you repackage 2.63 (fixing 
dependencies) then the package is 2.63-1.
Jan 09 2013
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 4 January 2013 at 19:59:19 UTC, Jonathan M Davis wrote:

 On Friday, January 04, 2013 20:30:28 Rob T wrote:

 Absolutely! Otherwise someone is going to think v2.062 is 
 greater
 than v2.062.1. Guaranteed. I already got semi-confused looking 
 at
 the latest download page and I know what's going on far more 
 than
 Joe Smith who walks in tomorrow checking out D for the first 
 time.


 Really? Why on earth would you think that 2.062 was greater 
 than 2.062.1?
 Also, I believe that it's very common with Linux packages (and 
 probably the
 projects themselves) to do that sort of versioning where 
 there's never a .0
 and the last part only gets added when you actually get a .1.

 - Jonathan M Davis


Both debian and ubuntu uses version.of.software-packageversion

For instance :

$ apt-cache policy audacious
audacious:
   Installé : 3.2.4-1
   Candidat : 3.2.4-1
  Table de version :
      3.3.3-2 0
          10 http://ftp.fr.debian.org/debian/ experimental/main 
amd64 Packages
  *** 3.2.4-1 0
         990 http://ftp.fr.debian.org/debian/ testing/main amd64 
Packages
         150 http://ftp.fr.debian.org/debian/ unstable/main amd64 
Packages
         100 /var/lib/dpkg/status
      2.3-2 0
         800 http://ftp.fr.debian.org/debian/ stable/main amd64 
Packages

3.2.4 is the version of audacious. 1 is the version of the 
package. Distro rarely interfers with the version of the software 
itself and simply resuse what software devs choose, adding their 
own system on top of it.
Jan 04 2013
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 4 January 2013 at 19:30:29 UTC, Rob T wrote:

 Terminology aside, we need to differentiate between bug fixes 
 that do not introduce new bugs vs bug fixes that may introduce 
 new bugs *or* break existing code for the current release 
 version. We also *have* to prevent new features and structural 
 changes (destabilizers) from leaking into a stable release.


Understood. The things is that it is more dependent of the code 
than of the bug itself.

Basically, it say that we should refuse risky bug fixes in 
released versions, and fix them into master. This probably should 
be specified at some point, but this is quite badly defined now, 
and I'm not sure this is really important to define this before 
actually using the process.

I'd go for a use you judgment here, or some very generic 
guideline like « if the bug fix require important refactoring, or 
change the behavior of an existing feature, then it should only 
be fixed in master ».
Jan 04 2013
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 19:35:17 +0100
schrieb "deadalnix" <deadalnix gmail.com>:



 
 First, regression fix don't make any sense to me. You suggest to 
 fix bug in master and fix regression in older branches. This 
 should be the opposite IMO.
 
 A regression is something that used to work, but don't work 
 anymore. So Correcting them in older version seems kind of 
 contradictory, especially when other bugs aren't.


I added a "Definitions" section to explain what's meant by regression
in that context. I also added a rationale, but already explained it
well. This was written assuming dmd though, were bugfixes can easily
cause regressions. For phobos/druntime we could use less strict rules,
I added a not about that on the wiki.

Now why we fix regressions in the release branches:
Let's say we have dmd 2.061, 2.062, 2.062.1. A regression was
introduced in 2.062 and my code that compiled fine in 2.061 doesn't work
in 2.062 anymore. But then the regression fix is pushed to the 2.062
release branch, a minor release is made and 2.062.1 successfully
compiles that code again.

If the regression already occured in 2.061 should it be fixed in 2.062?
This is a different question, but that situation can't happen as long
as we only support 1 release and only fix newly introduced regressions.


 Secondly, in every git commands block you have all the commands 
 to set-up the repository. I think this belong to its own 
 paragraph or even its own page. Then I'd replace them with git 
 remote update in git command block.


Yep, will fix that.
Jan 05 2013
parent Johannes Pfau <nospam example.com> writes:
Am Sat, 5 Jan 2013 11:51:43 +0100
schrieb Johannes Pfau <nospam example.com>:


 Am Fri, 04 Jan 2013 19:35:17 +0100
 schrieb "deadalnix" <deadalnix gmail.com>:

 Secondly, in every git commands block you have all the commands 
 to set-up the repository. I think this belong to its own 
 paragraph or even its own page. Then I'd replace them with git 
 remote update in git command block.


 
 Yep, will fix that.
 


http://wiki.dlang.org/Release_Process

Fixed, there's a "Local repository setup" section now. The instructions
should be useable like a script now, although it would be good if
someone proof-read them.

BTW: especially
http://wiki.dlang.org/Release_Process#Local_repository_setup should be
proof-read. I'm not sure if it's the best solution, but I wanted
something which works fine if the staging branch already exists on
origin(new forks) and if it doesn't already exist.
Jan 05 2013