www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Lieutenant needed: build and release process

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Andrew Edwards has done a great job with the recent release, but needs 
to step down because he's busy with other pursuits.

We need a release lieutenant who would carry us through the release 
process. Please tender your application by replying to this.


Thanks,

Andrei
Sep 07 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 8 September 2014 at 01:30:02 UTC, Andrei Alexandrescu 
wrote:
 Andrew Edwards has done a great job with the recent release, 
 but needs to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the 
 release process. Please tender your application by replying to 
 this.


 Thanks,

 Andrei
Did Andrew leave any kind of notes about the process he ended up with? (If it is on wiki, link may be helpful)
Sep 08 2014
next sibling parent "Joakim" <dlang joakim.airpost.net> writes:
On Monday, 8 September 2014 at 09:29:48 UTC, Dicebot wrote:
 On Monday, 8 September 2014 at 01:30:02 UTC, Andrei 
 Alexandrescu wrote:
 Andrew Edwards has done a great job with the recent release, 
 but needs to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the 
 release process. Please tender your application by replying to 
 this.


 Thanks,

 Andrei
Did Andrew leave any kind of notes about the process he ended up with? (If it is on wiki, link may be helpful)
Yes, describing or documenting the process would help people decide if they want to do it.
Sep 08 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/8/14, 2:29 AM, Dicebot wrote:
 On Monday, 8 September 2014 at 01:30:02 UTC, Andrei Alexandrescu wrote:
 Andrew Edwards has done a great job with the recent release, but needs
 to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the release
 process. Please tender your application by replying to this.


 Thanks,

 Andrei
Did Andrew leave any kind of notes about the process he ended up with? (If it is on wiki, link may be helpful)
Thanks for your interest. Forwarded the question to Andrew to make sure he doesn't miss it. I take it you're on the cusp of volunteering? :o) Andrei
Sep 08 2014
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 8 September 2014 at 16:59:55 UTC, Andrei Alexandrescu 
wrote:
 Did Andrew leave any kind of notes about the process he ended 
 up with?
 (If it is on wiki, link may be helpful)
Thanks for your interest. Forwarded the question to Andrew to make sure he doesn't miss it. I take it you're on the cusp of volunteering? :o) Andrei
I have already mentioned in the mail list that I can't afford such commitment for the time being but if there are simple instructions will try find time to help tih 2.066.1 I am primarily interested with information about various platform build box access and most up do date version of packaging scripts - git part is simple.
Sep 08 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/8/14, 2:29 AM, Dicebot wrote:
 On Monday, 8 September 2014 at 01:30:02 UTC, Andrei Alexandrescu wrote:
 Andrew Edwards has done a great job with the recent release, but needs
 to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the release
 process. Please tender your application by replying to this.


 Thanks,

 Andrei
Did Andrew leave any kind of notes about the process he ended up with? (If it is on wiki, link may be helpful)
I'm not positive but I think he followed these: http://wiki.dlang.org/Development_and_Release_Process http://wiki.dlang.org/Beta_Testing Andrei
Sep 08 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 03:56:12 UTC, Andrei Alexandrescu 
wrote:
 Did Andrew leave any kind of notes about the process he ended 
 up with?
 (If it is on wiki, link may be helpful)
I'm not positive but I think he followed these: http://wiki.dlang.org/Development_and_Release_Process http://wiki.dlang.org/Beta_Testing Andrei
Question about build boxes stands. Is it supposed to be done via auto-tester slaves or lieutenant needs to come with own set of build machines for all platforms? :)
Sep 08 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/8/14, 9:22 PM, Dicebot wrote:
 On Tuesday, 9 September 2014 at 03:56:12 UTC, Andrei Alexandrescu wrote:
 Did Andrew leave any kind of notes about the process he ended up with?
 (If it is on wiki, link may be helpful)
I'm not positive but I think he followed these: http://wiki.dlang.org/Development_and_Release_Process http://wiki.dlang.org/Beta_Testing Andrei
Question about build boxes stands. Is it supposed to be done via auto-tester slaves or lieutenant needs to come with own set of build machines for all platforms? :)
The lieutenant uses the existing infrastructure. -- Andrei
Sep 08 2014
parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/8/2014 9:36 PM, Andrei Alexandrescu via Digitalmars-d wrote:
 On 9/8/14, 9:22 PM, Dicebot wrote:
 On Tuesday, 9 September 2014 at 03:56:12 UTC, Andrei Alexandrescu wrote:
 Did Andrew leave any kind of notes about the process he ended up with?
 (If it is on wiki, link may be helpful)
I'm not positive but I think he followed these: http://wiki.dlang.org/Development_and_Release_Process http://wiki.dlang.org/Beta_Testing Andrei
Question about build boxes stands. Is it supposed to be done via auto-tester slaves or lieutenant needs to come with own set of build machines for all platforms? :)
The lieutenant uses the existing infrastructure. -- Andrei
Oh? The releases to date have NOT used the auto-tester systems at all. They're not really setup for others to use or for builds that need to be portable to other systems. Do you mean Walter's machines? Again, I don't think so since he's also never set them up for external use as far as I know. I believe that some vm images were created by someone that Andrew used, and that's likely the best course to continue right now, imho.
Sep 08 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 04:57:03 UTC, Brad Roberts via 
Digitalmars-d wrote:
 On 9/8/2014 9:36 PM, Andrei Alexandrescu via Digitalmars-d 
 wrote:
 On 9/8/14, 9:22 PM, Dicebot wrote:
Oh? The releases to date have NOT used the auto-tester systems at all. They're not really setup for others to use or for builds that need to be portable to other systems. Do you mean Walter's machines? Again, I don't think so since he's also never set them up for external use as far as I know. I believe that some vm images were created by someone that Andrew used, and that's likely the best course to continue right now, imho.
That also means that initial time costs to prepare everything for release maintenance are rather high making it non-trivial commitment. I'll see what I can do later this week but no promises.
Sep 08 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/8/14, 9:56 PM, Brad Roberts via Digitalmars-d wrote:
 On 9/8/2014 9:36 PM, Andrei Alexandrescu via Digitalmars-d wrote:
 On 9/8/14, 9:22 PM, Dicebot wrote:
 On Tuesday, 9 September 2014 at 03:56:12 UTC, Andrei Alexandrescu wrote:
 Did Andrew leave any kind of notes about the process he ended up with?
 (If it is on wiki, link may be helpful)
I'm not positive but I think he followed these: http://wiki.dlang.org/Development_and_Release_Process http://wiki.dlang.org/Beta_Testing Andrei
Question about build boxes stands. Is it supposed to be done via auto-tester slaves or lieutenant needs to come with own set of build machines for all platforms? :)
The lieutenant uses the existing infrastructure. -- Andrei
Oh? The releases to date have NOT used the auto-tester systems at all. They're not really setup for others to use or for builds that need to be portable to other systems. Do you mean Walter's machines? Again, I don't think so since he's also never set them up for external use as far as I know. I believe that some vm images were created by someone that Andrew used, and that's likely the best course to continue right now, imho.
s/uses/should use/ Andrei
Sep 08 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 09/09/14 06:56, Brad Roberts via Digitalmars-d wrote:

 I believe that some vm images were created by someone that Andrew used,
 and that's likely the best course to continue right now, imho.
I think Martin Nowak created the VM images using Vagrant. This file [1] contains some Vagrant related code/config. [1] https://github.com/D-Programming-Language/installer/blob/master/create_dmd_release/build_all.d -- /Jacob Carlborg
Sep 08 2014
prev sibling next sibling parent reply "Dragos Carp" <dragoscarp gmail.com> writes:
On Monday, 8 September 2014 at 01:30:02 UTC, Andrei Alexandrescu
wrote:
 Andrew Edwards has done a great job with the recent release, 
 but needs to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the 
 release process. Please tender your application by replying to 
 this.


 Thanks,

 Andrei
It may sound a bit radical, but I think the release process can be simplified a lot, if the 3 github repositories (dmd, druntime, and phobos) would be merged. For example, personally I find it quite annoying working with 3 different repositories for simple tasks as compiling a previous phobos library with debug symbols or such. The simple existence of tools like dvm or digger is a proof that more people have the same problems. I know that these tools do more than building sources from the 3 repositories, but on the other hand there are also a lot of other tasks that these tools does not support. Speaking about a new release lieutenant, I can imagine that working with 3 repositories instead of 1 is quite of a burden. Think about the beta and release candidate phase... instead of one list of open items you have 3 of them, each with own specialties, maybe some correlated. It could work on paper, but it does not fit in one's head (at least not in mine). I have collected a few pros/cons about merging the repositories: Pro: - simplified release tagging and branching - atomic commit of cross-repository changes - easier to experiment with cross-repository feature branches - single pull request queue offering a better overview about the project - easier grep, easier build - simplified build documentation Cons: - migration effort (documentation, merge scripts) - current work-flow adjustments - the resulted repo history could be sometimes confusing - lost github pull-request history What do you think about this? Would it be worth the effort? Destroy!
Sep 08 2014
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Monday, 8 September 2014 at 19:51:58 UTC, Dragos Carp wrote:
 I have collected a few pros/cons about merging the repositories:
This topic has been discussed in the past. Some more points that I can think of:
 Pro:
 - simplified release tagging and branching
 - atomic commit of cross-repository changes
 - easier to experiment with cross-repository feature branches
 - single pull request queue offering a better overview about the
 project
 - easier grep, easier build
 - simplified build documentation
- easier to run the entire test suite - much easier "git bisect"
 Cons:
 - migration effort (documentation, merge scripts)
 - current work-flow adjustments
 - the resulted repo history could be sometimes confusing
 - lost github pull-request history
- more difficult to assign ownership/responsibility - forking just one component becomes more difficult - more mixing of free and non-free source code in the same repository (although I heard splitting the DMD repo into two (frontend and backend) repositories was being discussed)
Sep 08 2014
parent reply "Dragos Carp" <dragoscarp gmail.com> writes:
On Monday, 8 September 2014 at 20:56:51 UTC, Vladimir Panteleev
wrote:
 On Monday, 8 September 2014 at 19:51:58 UTC, Dragos Carp wrote:
 I have collected a few pros/cons about merging the 
 repositories:
This topic has been discussed in the past. Some more points that I can think of:
 Pro:
 - simplified release tagging and branching
 - atomic commit of cross-repository changes
 - easier to experiment with cross-repository feature branches
 - single pull request queue offering a better overview about 
 the
 project
 - easier grep, easier build
 - simplified build documentation
- easier to run the entire test suite - much easier "git bisect"
 Cons:
 - migration effort (documentation, merge scripts)
 - current work-flow adjustments
 - the resulted repo history could be sometimes confusing
 - lost github pull-request history
- more difficult to assign ownership/responsibility
The directory structure will still be present. I don't think that would be a problem, it works pretty well for bigger projects.
 - forking just one component becomes more difficult
Forking a component is a seldom event, working with all three is the rule and we should strive to optimize it.
 - more mixing of free and non-free source code in the same 
 repository (although I heard splitting the DMD repo into two 
 (frontend and backend) repositories was being discussed)
Sep 08 2014
parent Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
Personally, I've never found the multiple repositories inconvenient. 
About the only place they are are when simultaneous changes are required 
to more than one of the parts.  That's INTENDED to be rare since it 
directly implies a non backwards compatible change.  Those changes tend 
to hurt people and making it more obvious that it's happening and making 
the bar a little higher to leap isn't a bad thing.

About the only pro listed below that I can agree with is the atomic 
commit issue (see above for my feelings on those) and the regression 
hunt.  There's already tooling to deal with the latter, and it works 
pretty well.  The rest fall into the "yes, you're right, but the benefit 
it minimal at best.

The con's are more significant to me.  The permissions issues.  The 
enforced layers.  The fact that the cost of change isn't small and the 
benefits from the change are small.

I vote no.

If I had to do it all over again, I'd still split them up.

On 9/8/2014 2:24 PM, Dragos Carp via Digitalmars-d wrote:
 On Monday, 8 September 2014 at 20:56:51 UTC, Vladimir Panteleev
 wrote:
 On Monday, 8 September 2014 at 19:51:58 UTC, Dragos Carp wrote:
 I have collected a few pros/cons about merging the repositories:
This topic has been discussed in the past. Some more points that I can think of:
 Pro:
 - simplified release tagging and branching
 - atomic commit of cross-repository changes
 - easier to experiment with cross-repository feature branches
 - single pull request queue offering a better overview about the
 project
 - easier grep, easier build
 - simplified build documentation
- easier to run the entire test suite - much easier "git bisect"
 Cons:
 - migration effort (documentation, merge scripts)
 - current work-flow adjustments
 - the resulted repo history could be sometimes confusing
 - lost github pull-request history
- more difficult to assign ownership/responsibility
The directory structure will still be present. I don't think that would be a problem, it works pretty well for bigger projects.
 - forking just one component becomes more difficult
Forking a component is a seldom event, working with all three is the rule and we should strive to optimize it.
 - more mixing of free and non-free source code in the same repository
 (although I heard splitting the DMD repo into two (frontend and
 backend) repositories was being discussed)
Sep 08 2014
prev sibling next sibling parent reply Jeremy Powers via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, Sep 8, 2014 at 12:51 PM, Dragos Carp via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 It may sound a bit radical, but I think the release process can
 be simplified a lot, if the 3 github repositories (dmd, druntime,
 and phobos) would be merged.
...
 Destroy!
These are separate projects/products, that just happen to be released together. Merging things together would obfuscate this (even more than it is now). How would you consume phobos with a different compiler, for example?
Sep 08 2014
parent "Dragos Carp" <dragoscarp gmail.com> writes:
On Monday, 8 September 2014 at 21:03:07 UTC, Jeremy Powers via
Digitalmars-d wrote:
 On Mon, Sep 8, 2014 at 12:51 PM, Dragos Carp via Digitalmars-d <
 digitalmars-d puremagic.com> wrote:

 It may sound a bit radical, but I think the release process can
 be simplified a lot, if the 3 github repositories (dmd, 
 druntime,
 and phobos) would be merged.
...
 Destroy!
These are separate projects/products, that just happen to be released together. Merging things together would obfuscate this (even more than it is now). How would you consume phobos with a different compiler, for example?
This would be the job of an installer build step... or just take the phobos directory.
Sep 08 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/8/14, 12:51 PM, Dragos Carp wrote:
 It may sound a bit radical, but I think the release process can
 be simplified a lot, if the 3 github repositories (dmd, druntime,
 and phobos) would be merged.
I'd support that if there's consensus. -- Andrei
Sep 08 2014
parent reply "Trass3r" <un known.com> writes:
You could also use submodules (or subtrees, haven't tried them 
yet).
Sep 08 2014
next sibling parent reply "Dragos Carp" <dragoscarp gmail.com> writes:
On Monday, 8 September 2014 at 21:55:47 UTC, Trass3r wrote:
 You could also use submodules (or subtrees, haven't tried them 
 yet).
AFAIK the tags and branches does not work over submodules/subtrees and you would still have 3 github repos with 3 pull request queues.
Sep 08 2014
parent reply "Trass3r" <un known.com> writes:
 with 3 pull request queues
Good argument for the separation :)
Sep 08 2014
parent Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/8/2014 3:12 PM, Trass3r via Digitalmars-d wrote:
 with 3 pull request queues
Good argument for the separation :)
And they're visible together via the auto-tester which happens to keep the lists concatenated. I don't see the separation to be an issue either.
Sep 08 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/8/14, 2:55 PM, Trass3r wrote:
 You could also use submodules (or subtrees, haven't tried them yet).
A git expert told me submodules are fail. Is that true? -- Andrei
Sep 08 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 00:28:38 UTC, Andrei Alexandrescu 
wrote:
 On 9/8/14, 2:55 PM, Trass3r wrote:
 You could also use submodules (or subtrees, haven't tried them 
 yet).
A git expert told me submodules are fail. Is that true? -- Andrei
As any powerful feature it can be used to great success or great failure :) However it is probably a bad case for deeply connected projects that needs to be updated in sync like it is with dmd and phobos. Those are more useful as a git-only alternative to dub dependencies in my opinion - static relations that don't change often. `git subtree` may be a betetr match here but I am not familiar with those. Going to check the docs now.
Sep 08 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 00:51:25 UTC, Dicebot wrote:
 `git subtree` may be a betetr match here but I am not familiar 
 with those. Going to check the docs now.
Yeah those definitely look more interesting - as far as I can see subtree is bound to a branch and not to specific commit hash.
Sep 08 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 09/09/14 02:56, Dicebot wrote:

 Yeah those definitely look more interesting - as far as I can see
 subtree is bound to a branch and not to specific commit hash.
Since Git 1.8.2 you can bound a submodule to a branch. -- /Jacob Carlborg
Sep 08 2014
parent "Trass3r" <un known.com> writes:
 Since Git 1.8.2 you can bound a submodule to a branch.
Ah cool didn't know that.
Sep 09 2014
prev sibling next sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Monday, 8 September 2014 at 19:51:58 UTC, Dragos Carp wrote:
 Cons:
 - migration effort (documentation, merge scripts)
 - current work-flow adjustments
 - the resulted repo history could be sometimes confusing
 - lost github pull-request history
It would also be a pretty serious change for LDC. We are currently using druntime/Phobos as Git submodules. David
Sep 08 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 8 September 2014 at 19:51:58 UTC, Dragos Carp wrote:
 I have collected a few pros/cons about merging the repositories:

 Pro:
 - simplified release tagging and branching
 - atomic commit of cross-repository changes
 - easier to experiment with cross-repository feature branches
 - single pull request queue offering a better overview about the
 project
 - easier grep, easier build
 - simplified build documentation

 Cons:
 - migration effort (documentation, merge scripts)
 - current work-flow adjustments
 - the resulted repo history could be sometimes confusing
 - lost github pull-request history

 What do you think about this? Would it be worth the effort?
 Destroy!
ne important (in my opinion) disadvantage misssing: considerably harder PR maintenance (missing automatic categorization and separation between phobos / dmd teams). Being forced to look ar DMD pull requests to do Phobos maintenance will add lot of context overhead for me. I also don't feel like it will help much for release preparation. Bisection and history investigation - undoubtedly. But for release management building stuff is one of the easier parts.
Sep 08 2014
next sibling parent Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/8/2014 3:51 PM, Dicebot via Digitalmars-d wrote:
 I also don't feel like it will help much for release preparation.
 Bisection and history investigation - undoubtedly. But for release
 management building stuff is one of the easier parts.
I totally agree with this. Anyone that believes that the checkout, build, and test steps are anywhere near hard or the hard part of the release process really doesn't have a good understanding of what it takes. Those are the _easy_ steps, and so thoroughly automated/automatable that it's just a non-issue.
Sep 08 2014
prev sibling parent reply "Dragos Carp" <dragoscarp gmail.com> writes:
 ne important (in my opinion) disadvantage misssing: 
 considerably harder PR maintenance (missing automatic 
 categorization and separation between phobos / dmd teams). 
 Being forced to look ar DMD pull requests to do Phobos 
 maintenance will add lot of context overhead for me.
This kind of attitude worries me: every team works happily in its own cozy repository and it doesn't feel responsible about the other repos. The consequence is that, while breeding a new release, the release manager MUST take most of the burden about coordinating the teams and their work. This is a very complex task which requires insight knowledge in all 3 repos. Having 3 repos, one team is not encouraged to help the other team to get the release out the door. This is because one team have a higher bar to help the other: it needs to checkout the right version of the foreign repo, manage own branches in the foreign repo, etc. Maybe this is the reason of multiple regression in the last release or the "virtual keyword surprise" (at least for some people). I think that a 2 phase process similar to the Linux kernel release process [1] would work much better and motivates all the teams to help with the release: Phase 1. Merge window: merge PRs containing new features Phase 2. Test and stabilization: merge bugfix PRs only Ideally the first phase is quite short (1 week for example) and it can be close supervised (or even executed??) by Walter and/or Andrei. The second phase will continue until the acceptance criteria are met, the teams having a focus on the release (they get no new goodies till the test phase is done). The normal development happens somehow offline by preparing quality PRs to be merged in phase 1.
 I also don't feel like it will help much for release 
 preparation. Bisection and history investigation - undoubtedly. 
 But for release management building stuff is one of the easier 
 parts.
Agree, the technical build steps of the release process are easy. Just the casual experimenting with D is somehow clumsy. [1] - https://www.kernel.org/doc/Documentation/development-process/2.Process
Sep 09 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 09:59:37 UTC, Dragos Carp wrote:
 ne important (in my opinion) disadvantage misssing: 
 considerably harder PR maintenance (missing automatic 
 categorization and separation between phobos / dmd teams). 
 Being forced to look ar DMD pull requests to do Phobos 
 maintenance will add lot of context overhead for me.
This kind of attitude worries me: every team works happily in its own cozy repository and it doesn't feel responsible about the other repos. The consequence is that, while breeding a new release, the release manager MUST take most of the burden about coordinating the teams and their work. This is a very complex task which requires insight knowledge in all 3 repos.
This is not about happiness or zone of comfort - I am simply not competent to judge even most trivial DMD pull request as opposed to Phobos ones. Team separation exists for a reason - skills and knowledge required are totally different. Some developers do both but I doubt it is even the majority of active maintainers. By merging repos you are likely to discourage me from paying attention at all, not to invest huge amount of effort in increasing DMD proficiency. I am also not sure what what coordination you are speaking about. Integrity of 3 repos is verified by auto-teser. Cross-repo issues are rare and usually handled by those who has merge access to both. Release manager shouldn't normally need any detailed knowledge of repo internal development and just tag certain branch heads. If this is not the case we have some hidden problem I am not aware of.
 Having 3 repos, one team is not encouraged to help the other 
 team to get the release out the door. This is because one team 
 have a higher bar to help the other: it needs to checkout the 
 right version of the foreign repo, manage own branches in the 
 foreign repo, etc. Maybe this is the reason of multiple 
 regression in the last release or the "virtual keyword 
 surprise" (at least for some people).
It feels like you have imagined some process that does not really match reality :( Explicit tracking of matching version is never needed - it is either branch head for all repos, or identically named tags. I have already explained above some of the real reasons that make closer collaboration close to impossible.
 I think that a 2 phase process similar to the Linux kernel 
 release process [1] would work much better and motivates all 
 the teams to help with the release:

 Phase 1. Merge window: merge PRs containing new features
 Phase 2. Test and stabilization: merge bugfix PRs only

 Ideally the first phase is quite short (1 week for example) and 
 it can be close supervised (or even executed??) by Walter 
 and/or Andrei.

 The second phase will continue until the acceptance criteria 
 are met, the teams having a focus on the release (they get no 
 new goodies till the test phase is done).

 The normal development happens somehow offline by preparing 
 quality PRs to be merged in phase 1.
This all sounds like solving the problem that doesn't exist instead of addressing the real ones (lack of clear documented rules for release packaging and awkward cherry-pick based release branch maintenance)
Sep 09 2014
parent reply "Dicebot" <public dicebot.lv> writes:
Also it sounds as if you think that someone actually does any 
coordination about what must go into release. As far as I am 
aware there is no such thing, even http://wiki.dlang.org/Agenda 
is just a convention. Currently releases are based exclusively on 
a time frame + regression list (all that was in master goes to 
the release branch and is kept there until known regressions are 
fixed, repeat for the next cycle).
Sep 09 2014
parent reply "Dragos Carp" <dragoscarp gmail.com> writes:
On Tuesday, 9 September 2014 at 12:31:26 UTC, Dicebot wrote:
 Also it sounds as if you think that someone actually does any 
 coordination about what must go into release. As far as I am 
 aware there is no such thing, even http://wiki.dlang.org/Agenda 
 is just a convention. Currently releases are based exclusively 
 on a time frame + regression list (all that was in master goes 
 to the release branch and is kept there until known regressions 
 are fixed, repeat for the next cycle).
Are you satisfied with the current process? Let me summarize some important drawbacks of the current workflow: 1. No clear defined deadline for preparing a merge-able PR. 2. Unorganized PR merge campaigns. The people merging the PR are doing a great job, but they do this triggered by arbitrary events: too many open PRs, a cool new PR appears, somebody poke them on forum, or simply have some time for this kind of work. 3. Somehow arbitrary merge criteria. Having a defined merge window, when some people do just PR merges, will implicitly produce more predictable and uniform acceptance criteria. 4. Lack of focus during test phase. Maybe this is the main reason for the v2.066 regressions. Some people keep merging new PRs, before the old ones are proven done during the test phase. Even Walter was annoyed a couple of times by the multitude of versions that the people are simultaneously working on. 5. Rotting old PRs. The "merge window" phase would be a defined recurrent occasion to review and decide about those.
Sep 09 2014
next sibling parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/9/2014 6:54 AM, Dragos Carp via Digitalmars-d wrote:
 On Tuesday, 9 September 2014 at 12:31:26 UTC, Dicebot wrote:
 Also it sounds as if you think that someone actually does any
 coordination about what must go into release. As far as I am aware
 there is no such thing, even http://wiki.dlang.org/Agenda is just a
 convention. Currently releases are based exclusively on a time frame +
 regression list (all that was in master goes to the release branch and
 is kept there until known regressions are fixed, repeat for the next
 cycle).
Are you satisfied with the current process? Let me summarize some important drawbacks of the current workflow: 1. No clear defined deadline for preparing a merge-able PR. 2. Unorganized PR merge campaigns. The people merging the PR are doing a great job, but they do this triggered by arbitrary events: too many open PRs, a cool new PR appears, somebody poke them on forum, or simply have some time for this kind of work. 3. Somehow arbitrary merge criteria. Having a defined merge window, when some people do just PR merges, will implicitly produce more predictable and uniform acceptance criteria. 4. Lack of focus during test phase. Maybe this is the main reason for the v2.066 regressions. Some people keep merging new PRs, before the old ones are proven done during the test phase. Even Walter was annoyed a couple of times by the multitude of versions that the people are simultaneously working on. 5. Rotting old PRs. The "merge window" phase would be a defined recurrent occasion to review and decide about those.
Of course the process can be better. But NONE of those are a result of the repository split that you're advocating removing.
Sep 09 2014
parent "Dragos Carp" <dragoscarp gmail.com> writes:
On Tuesday, 9 September 2014 at 18:56:16 UTC, Brad Roberts via
Digitalmars-d wrote:
 Of course the process can be better.  But NONE of those are a 
 result of the repository split that you're advocating removing.
I think these are related. It will be a real challenge to keep in sync three in parallel running release processes. Keeping in mind that each team has own development pace, man-power, or undertakings, how do you avoid that the three release processes does not diverge between the teams?
Sep 09 2014
prev sibling next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 9/9/2014 9:54 AM, Dragos Carp wrote:
 Are you satisfied with the current process?

 Let me summarize some important drawbacks of the current workflow:

 1. No clear defined deadline for preparing a merge-able PR.

 2. Unorganized PR merge campaigns. The people merging the PR are doing a
 great job, but they do this triggered by arbitrary events: too many open
 PRs, a cool new PR appears, somebody poke them on forum, or simply have
 some time for this kind of work.

 3. Somehow arbitrary merge criteria. Having a defined merge window, when
 some people do just PR merges, will implicitly produce more predictable
 and uniform acceptance criteria.

 4. Lack of focus during test phase. Maybe this is the main reason for
 the v2.066 regressions. Some people keep merging new PRs, before the old
 ones are proven done during the test phase. Even Walter was annoyed a
 couple of times by the multitude of versions that the people are
 simultaneously working on.

 5. Rotting old PRs. The "merge window" phase would be a defined
 recurrent occasion to review and decide about those.
Most of that is unavoidable without salaried devs on it. D is 100% volunteer, deadlines and such aren't realistically feasible.
Sep 09 2014
parent reply "Dragos Carp" <dragoscarp gmail.com> writes:
On Tuesday, 9 September 2014 at 22:41:39 UTC, Nick Sabalausky
wrote:
 Most of that is unavoidable without salaried devs on it. D is 
 100% volunteer, deadlines and such aren't realistically 
 feasible.
I'm talking about the deadlines for your own PRs. If you miss one release, you will have your chance in the next one. Of course you will still freely choose for yourself the tasks that suit you best. Beside that, having a deadline it is a great motivational factor to do even some unpleasant work (for me at least).
Sep 09 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/9/14, 4:06 PM, Dragos Carp wrote:
 On Tuesday, 9 September 2014 at 22:41:39 UTC, Nick Sabalausky
 wrote:
 Most of that is unavoidable without salaried devs on it. D is 100%
 volunteer, deadlines and such aren't realistically feasible.
I'm talking about the deadlines for your own PRs. If you miss one release, you will have your chance in the next one. Of course you will still freely choose for yourself the tasks that suit you best. Beside that, having a deadline it is a great motivational factor to do even some unpleasant work (for me at least).
Yah, we should have deadlines. -- Andrei
Sep 10 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 10 September 2014 at 07:43:22 UTC, Andrei 
Alexandrescu wrote:
 Yah, we should have deadlines. -- Andrei
We have them already, it just happens that those are never realistically executed. http://wiki.dlang.org/Release_Process#Release_Schedule
Sep 10 2014
parent Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/10/14, Dicebot via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Wednesday, 10 September 2014 at 07:43:22 UTC, Andrei
 Alexandrescu wrote:
 Yah, we should have deadlines. -- Andrei
We have them already, it just happens that those are never realistically executed. http://wiki.dlang.org/Release_Process#Release_Schedule
“I love deadlines. I love the whooshing noise they make as they go by.” - Douglas Adams
Sep 10 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 13:54:15 UTC, Dragos Carp wrote:
 Are you satisfied with the current process?
Yes. I am not satisfied with certain parts of implementation of the process but the concept overall is a good as it can possibly be given manpower available and lack of any centralized collaboration. Think about it - despite the fact that there is no one actually responsible for release planning and no fully commited developers we still somehow manage to get those release out. This is quite an achievment if you think about it.
 Let me summarize some important drawbacks of the current 
 workflow:

 1. No clear defined deadline for preparing a merge-able PR.
.. which allows maintainers to merge pull requests when they seem ready without any synchronization "locks" and does not interfere with release branches.
 2. Unorganized PR merge campaigns. The people merging the PR 
 are doing a great job, but they do this triggered by arbitrary 
 events: too many open PRs, a cool new PR appears, somebody poke 
 them on forum, or simply have some time for this kind of work.
There is no such thing as "PR merge campaign" and there is not point in having one. It is everlasting process - stuff gets merged when its done, it is completly irrelevant to release planning. "simply have some time for this kind of work" is the only realistic criteria for any sort of maintenance activity here, you can't get any other without bunch of devs on salary. It is a process which is almost exclusively blocked by maintainer time. It is quite telling that situation with Phobos (which has easier learning curve and more active maintainers overall) is much better than with DMD.
 3. Somehow arbitrary merge criteria. Having a defined merge 
 window, when some people do just PR merges, will implicitly 
 produce more predictable and uniform acceptance criteria.
What makes you think so? It will only introduce arbitrary limitation on a time when you can merge pull requests. Merge criteria is an orthogonal topic - if it is possible to formalize one, it can also be done without introducing merge windows. Implicit uniform criteria is much better defined by the simple fact that those are the very same people doing merges (and there aren't many) Only thing merge window will achieve is making pull reqeuests stockpile even more because developers won't be allowed to merge them when they have time to do so.
 4. Lack of focus during test phase. Maybe this is the main 
 reason for the v2.066 regressions. Some people keep merging new 
 PRs, before the old ones are proven done during the test phase. 
 Even Walter was annoyed a couple of times by the multitude of 
 versions that the people are simultaneously working on
Main reason is that there are only few developers capable of fixing DMD regressions. If release beta happens when either of them is busy it will result in immediate slowdown. This is especially true about Kenji. Do you realize that fixing release and merging new pull requests happens in _different_ branches? And merging even 1000 new features doesn't affect testing of the release beta? There were certain bad cases when major refactorings in master make it hard to port bug fixes to release branch but those were rare.
 5. Rotting old PRs. The "merge window" phase would be a defined 
 recurrent occasion to review and decide about those.
There are no rotting old PRs now in Phobos despite the maintenance / release process is the same. Guess why? With recent addition of bunch of new maintainers we simply got the resources to clean and categorize those and this just immediately happened. Unfortunately, there is no magic process can just create new developers out of the air :(
Sep 09 2014
prev sibling parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Dragos Carp"  wrote in message news:qjcqyddvsaopicjmeuoy forum.dlang.org...

 It may sound a bit radical, but I think the release process can
 be simplified a lot, if the 3 github repositories (dmd, druntime,
 and phobos) would be merged.
Even if there were no downsides, it still wouldn't be worth the disruption.
Sep 08 2014
prev sibling parent reply Andrew Edwards <ridimz yahoo.com> writes:
On 9/8/14, 10:30 AM, Andrei Alexandrescu wrote:
 Andrew Edwards has done a great job with the recent release, but needs
 to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the release
 process. Please tender your application by replying to this.


 Thanks,

 Andrei
Follow these steps to prepare your build environment: 1. Install VirtualBox: https://www.virtualbox.org/wiki/Downloads 2. Install Vagrant: https://www.vagrantup.com/downloads.html 3. Clone installer: git clone --recursive https://github.com/D-Programming-Language/installer.git Note 1: MartinNowak already prepared boxes for FreeBSD and Debian which are directly imported through the build script. However, you must prepare your own OSX and Windows 7 boxes. Instructions for preparing doing so are located here: OSX => https://gist.github.com/MartinNowak/8156507 Windows7 => https://gist.github.com/MartinNowak/8270666 Steps for building are outlined here: http://wiki.dlang.org/Simplified_Release_Process_Proposal Tracking is done here: http://wiki.dlang.org/Beta_Testing Note 2: I am not vacating the post of Build Master. I have some personal matters that require a significant amount of my time over the next couple of months. I cannot promise to be on time with the releases before October 15. Until then, I will do my best to prepare releases as time permits. As it stands: If there are no pending merges for the 2.066.1 branch, I will be able to prepare a RC before morning (time now is 12:58 AM JST).
Sep 09 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 15:58:45 UTC, Andrew Edwards 
wrote:
 On 9/8/14, 10:30 AM, Andrei Alexandrescu wrote:
 Andrew Edwards has done a great job with the recent release, 
 but needs
 to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the 
 release
 process. Please tender your application by replying to this.


 Thanks,

 Andrei
Follow these steps to prepare your build environment: 1. Install VirtualBox: https://www.virtualbox.org/wiki/Downloads 2. Install Vagrant: https://www.vagrantup.com/downloads.html 3. Clone installer: git clone --recursive https://github.com/D-Programming-Language/installer.git Note 1: MartinNowak already prepared boxes for FreeBSD and Debian which are directly imported through the build script. However, you must prepare your own OSX and Windows 7 boxes. Instructions for preparing doing so are located here: OSX => https://gist.github.com/MartinNowak/8156507 Windows7 => https://gist.github.com/MartinNowak/8270666 Steps for building are outlined here: http://wiki.dlang.org/Simplified_Release_Process_Proposal Tracking is done here: http://wiki.dlang.org/Beta_Testing Note 2: I am not vacating the post of Build Master. I have some personal matters that require a significant amount of my time over the next couple of months. I cannot promise to be on time with the releases before October 15. Until then, I will do my best to prepare releases as time permits. As it stands: If there are no pending merges for the 2.066.1 branch, I will be able to prepare a RC before morning (time now is 12:58 AM JST).
Thanks Andrew! This is most helpful. I will try this out this weekend to be able to act as backup build master for 2.067
Sep 09 2014
prev sibling next sibling parent "NVolcz" <volcz kth.se> writes:
On Tuesday, 9 September 2014 at 15:58:45 UTC, Andrew Edwards 
wrote:
 On 9/8/14, 10:30 AM, Andrei Alexandrescu wrote:
 Andrew Edwards has done a great job with the recent release, 
 but needs
 to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the 
 release
 process. Please tender your application by replying to this.


 Thanks,

 Andrei
Note 2: I am not vacating the post of Build Master. I have some personal matters that require a significant amount of my time over the next couple of months. I cannot promise to be on time with the releases before October 15. Until then, I will do my best to prepare releases as time permits. As it stands: If there are no pending merges for the 2.066.1 branch, I will be able to prepare a RC before morning (time now is 12:58 AM JST).
How much of your free time has the Build Mater/release lieutenant role taken?
Sep 10 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 15:58:45 UTC, Andrew Edwards 
wrote:
 On 9/8/14, 10:30 AM, Andrei Alexandrescu wrote:
 Andrew Edwards has done a great job with the recent release, 
 but needs
 to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the 
 release
 process. Please tender your application by replying to this.


 Thanks,

 Andrei
Follow these steps to prepare your build environment: 1. Install VirtualBox: https://www.virtualbox.org/wiki/Downloads 2. Install Vagrant: https://www.vagrantup.com/downloads.html 3. Clone installer: git clone --recursive https://github.com/D-Programming-Language/installer.git Note 1: MartinNowak already prepared boxes for FreeBSD and Debian which are directly imported through the build script. However, you must prepare your own OSX and Windows 7 boxes. Instructions for preparing doing so are located here: OSX => https://gist.github.com/MartinNowak/8156507 Windows7 => https://gist.github.com/MartinNowak/8270666 Steps for building are outlined here: http://wiki.dlang.org/Simplified_Release_Process_Proposal Tracking is done here: http://wiki.dlang.org/Beta_Testing
Trying this right now, got through Win7 box setup but where one is supposed to get "InstallESD.dmg" mentioned in installation instructions?
Sep 20 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-09-20 16:44, Dicebot wrote:

 Trying this right now, got through Win7 box setup but where one is
 supposed to get "InstallESD.dmg" mentioned in installation instructions?
As the instructions say, from the "Install Mac OS X Mountain Lion.app" bundle. This bundle is downloaded from the Mac App Store. For Mavericks it's located in "Install OS X Mavericks.app/Contents/SharedSupport/InstallESD.dmg". If you download it form the app store it's will be placed in /Applications. BTW you should only do this on a Mac. Technically it's possible to do on another platform. -- /Jacob Carlborg
Sep 20 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 15:16:20 UTC, Jacob Carlborg 
wrote:
 BTW you should only do this on a Mac. Technically it's possible 
 to do on another platform.
Ah I see. Guess this is the point where I should say "Thanks, no, I am not going to volunteer". Can do linux-only releases if anyone is interested ;)
Sep 20 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 8:23 AM, Dicebot wrote:
 On Saturday, 20 September 2014 at 15:16:20 UTC, Jacob Carlborg wrote:
 BTW you should only do this on a Mac. Technically it's possible to do
 on another platform.
Ah I see. Guess this is the point where I should say "Thanks, no, I am not going to volunteer". Can do linux-only releases if anyone is interested ;)
Any help is welcome, but we do need a point of contact. Who wants to be our build lieutenant? Thanks, Andrei
Sep 20 2014