www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Lieutenant needed: build and release process

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Andrew Edwards has done a great job with the recent release, but needs 
to step down because he's busy with other pursuits.

We need a release lieutenant who would carry us through the release 
process. Please tender your application by replying to this.


Thanks,

Andrei
Sep 07 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 8 September 2014 at 01:30:02 UTC, Andrei Alexandrescu 
wrote:

 Andrew Edwards has done a great job with the recent release, 
 but needs to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the 
 release process. Please tender your application by replying to 
 this.


 Thanks,

 Andrei


Did Andrew leave any kind of notes about the process he ended up 
with? (If it is on wiki, link may be helpful)
Sep 08 2014
next sibling parent "Joakim" <dlang joakim.airpost.net> writes:
On Monday, 8 September 2014 at 09:29:48 UTC, Dicebot wrote:

 On Monday, 8 September 2014 at 01:30:02 UTC, Andrei 
 Alexandrescu wrote:

 Andrew Edwards has done a great job with the recent release, 
 but needs to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the 
 release process. Please tender your application by replying to 
 this.


 Thanks,

 Andrei


 Did Andrew leave any kind of notes about the process he ended 
 up with? (If it is on wiki, link may be helpful)


Yes, describing or documenting the process would help people 
decide if they want to do it.
Sep 08 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/8/14, 2:29 AM, Dicebot wrote:

 On Monday, 8 September 2014 at 01:30:02 UTC, Andrei Alexandrescu wrote:

 Andrew Edwards has done a great job with the recent release, but needs
 to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the release
 process. Please tender your application by replying to this.


 Thanks,

 Andrei


 Did Andrew leave any kind of notes about the process he ended up with?
 (If it is on wiki, link may be helpful)


Thanks for your interest. Forwarded the question to Andrew to make sure 
he doesn't miss it. I take it you're on the cusp of volunteering? :o)

Andrei
Sep 08 2014
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 8 September 2014 at 16:59:55 UTC, Andrei Alexandrescu 
wrote:

 Did Andrew leave any kind of notes about the process he ended 
 up with?
 (If it is on wiki, link may be helpful)


 Thanks for your interest. Forwarded the question to Andrew to 
 make sure he doesn't miss it. I take it you're on the cusp of 
 volunteering? :o)

 Andrei


I have already mentioned in the mail list that I can't afford 
such commitment for  the time being but if there are simple 
instructions will try find time to help tih 2.066.1

I am primarily interested with information about various platform 
build box access and most up do date version of packaging scripts 
- git part is simple.
Sep 08 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/8/14, 2:29 AM, Dicebot wrote:

 On Monday, 8 September 2014 at 01:30:02 UTC, Andrei Alexandrescu wrote:

 Andrew Edwards has done a great job with the recent release, but needs
 to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the release
 process. Please tender your application by replying to this.


 Thanks,

 Andrei


 Did Andrew leave any kind of notes about the process he ended up with?
 (If it is on wiki, link may be helpful)


I'm not positive but I think he followed these:

http://wiki.dlang.org/Development_and_Release_Process
http://wiki.dlang.org/Beta_Testing


Andrei
Sep 08 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 03:56:12 UTC, Andrei Alexandrescu 
wrote:

 Did Andrew leave any kind of notes about the process he ended 
 up with?
 (If it is on wiki, link may be helpful)


 I'm not positive but I think he followed these:

 http://wiki.dlang.org/Development_and_Release_Process
 http://wiki.dlang.org/Beta_Testing


 Andrei


Question about build boxes stands. Is it supposed to be done via 
auto-tester slaves or lieutenant needs to come with own set of 
build machines for all platforms? :)
Sep 08 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/8/14, 9:22 PM, Dicebot wrote:

 On Tuesday, 9 September 2014 at 03:56:12 UTC, Andrei Alexandrescu wrote:

 Did Andrew leave any kind of notes about the process he ended up with?
 (If it is on wiki, link may be helpful)


 I'm not positive but I think he followed these:

 http://wiki.dlang.org/Development_and_Release_Process
 http://wiki.dlang.org/Beta_Testing


 Andrei


 Question about build boxes stands. Is it supposed to be done via
 auto-tester slaves or lieutenant needs to come with own set of build
 machines for all platforms? :)


The lieutenant uses the existing infrastructure. -- Andrei
Sep 08 2014
parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/8/2014 9:36 PM, Andrei Alexandrescu via Digitalmars-d wrote:

 On 9/8/14, 9:22 PM, Dicebot wrote:

 On Tuesday, 9 September 2014 at 03:56:12 UTC, Andrei Alexandrescu wrote:

 Did Andrew leave any kind of notes about the process he ended up with?
 (If it is on wiki, link may be helpful)


 I'm not positive but I think he followed these:

 http://wiki.dlang.org/Development_and_Release_Process
 http://wiki.dlang.org/Beta_Testing


 Andrei


 Question about build boxes stands. Is it supposed to be done via
 auto-tester slaves or lieutenant needs to come with own set of build
 machines for all platforms? :)


 The lieutenant uses the existing infrastructure. -- Andrei


Oh?  The releases to date have NOT used the auto-tester systems at all. 
  They're not really setup for others to use or for builds that need to 
be portable to other systems.  Do you mean Walter's machines?  Again, I 
don't think so since he's also never set them up for external use as far 
as I know.

I believe that some vm images were created by someone that Andrew used, 
and that's likely the best course to continue right now, imho.
Sep 08 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 04:57:03 UTC, Brad Roberts via 
Digitalmars-d wrote:

 On 9/8/2014 9:36 PM, Andrei Alexandrescu via Digitalmars-d 
 wrote:

 On 9/8/14, 9:22 PM, Dicebot wrote:


 Oh?  The releases to date have NOT used the auto-tester systems 
 at all.
  They're not really setup for others to use or for builds that 
 need to be portable to other systems.  Do you mean Walter's 
 machines?  Again, I don't think so since he's also never set 
 them up for external use as far as I know.

 I believe that some vm images were created by someone that 
 Andrew used, and that's likely the best course to continue 
 right now, imho.


That also means that initial time costs to prepare everything for 
release maintenance are rather high making it non-trivial 
commitment. I'll see what I can do later this week but no 
promises.
Sep 08 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/8/14, 9:56 PM, Brad Roberts via Digitalmars-d wrote:

 On 9/8/2014 9:36 PM, Andrei Alexandrescu via Digitalmars-d wrote:

 On 9/8/14, 9:22 PM, Dicebot wrote:

 On Tuesday, 9 September 2014 at 03:56:12 UTC, Andrei Alexandrescu wrote:

 Did Andrew leave any kind of notes about the process he ended up with?
 (If it is on wiki, link may be helpful)


 I'm not positive but I think he followed these:

 http://wiki.dlang.org/Development_and_Release_Process
 http://wiki.dlang.org/Beta_Testing


 Andrei


 Question about build boxes stands. Is it supposed to be done via
 auto-tester slaves or lieutenant needs to come with own set of build
 machines for all platforms? :)


 The lieutenant uses the existing infrastructure. -- Andrei


 Oh?  The releases to date have NOT used the auto-tester systems at all.
   They're not really setup for others to use or for builds that need to
 be portable to other systems.  Do you mean Walter's machines?  Again, I
 don't think so since he's also never set them up for external use as far
 as I know.

 I believe that some vm images were created by someone that Andrew used,
 and that's likely the best course to continue right now, imho.


s/uses/should use/

Andrei
Sep 08 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 09/09/14 06:56, Brad Roberts via Digitalmars-d wrote:


 I believe that some vm images were created by someone that Andrew used,
 and that's likely the best course to continue right now, imho.


I think Martin Nowak created the VM images using Vagrant. This file [1] 
contains some Vagrant related code/config.

[1] 
https://github.com/D-Programming-Language/installer/blob/master/create_dmd_release/build_all.d

-- 
/Jacob Carlborg
Sep 08 2014
prev sibling next sibling parent reply "Dragos Carp" <dragoscarp gmail.com> writes:
On Monday, 8 September 2014 at 01:30:02 UTC, Andrei Alexandrescu
wrote:

 Andrew Edwards has done a great job with the recent release, 
 but needs to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the 
 release process. Please tender your application by replying to 
 this.


 Thanks,

 Andrei


It may sound a bit radical, but I think the release process can
be simplified a lot, if the 3 github repositories (dmd, druntime,
and phobos) would be merged.

For example, personally I find it quite annoying working with 3
different repositories for simple tasks as compiling a previous
phobos library with debug symbols or such.

The simple existence of tools like dvm or digger is a proof that
more people have the same problems. I know that these tools do
more than building sources from the 3 repositories, but on the
other hand there are also a lot of other tasks that these tools
does not support.

Speaking about a new release lieutenant, I can imagine that
working with 3 repositories instead of 1 is quite of a burden.
Think about the beta and release candidate phase... instead of
one list of open items you have 3 of them, each with own
specialties, maybe some correlated. It could work on paper, but
it does not fit in one's head (at least not in mine).

I have collected a few pros/cons about merging the repositories:

Pro:
- simplified release tagging and branching
- atomic commit of cross-repository changes
- easier to experiment with cross-repository feature branches
- single pull request queue offering a better overview about the
project
- easier grep, easier build
- simplified build documentation

Cons:
- migration effort (documentation, merge scripts)
- current work-flow adjustments
- the resulted repo history could be sometimes confusing
- lost github pull-request history

What do you think about this? Would it be worth the effort?
Destroy!
Sep 08 2014
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Monday, 8 September 2014 at 19:51:58 UTC, Dragos Carp wrote:

 I have collected a few pros/cons about merging the repositories:


This topic has been discussed in the past. Some more points that 
I can think of:


 Pro:
 - simplified release tagging and branching
 - atomic commit of cross-repository changes
 - easier to experiment with cross-repository feature branches
 - single pull request queue offering a better overview about the
 project
 - easier grep, easier build
 - simplified build documentation


- easier to run the entire test suite
- much easier "git bisect"


 Cons:
 - migration effort (documentation, merge scripts)
 - current work-flow adjustments
 - the resulted repo history could be sometimes confusing
 - lost github pull-request history


- more difficult to assign ownership/responsibility
- forking just one component becomes more difficult
- more mixing of free and non-free source code in the same 
repository (although I heard splitting the DMD repo into two 
(frontend and backend) repositories was being discussed)
Sep 08 2014
parent reply "Dragos Carp" <dragoscarp gmail.com> writes:
On Monday, 8 September 2014 at 20:56:51 UTC, Vladimir Panteleev
wrote:

 On Monday, 8 September 2014 at 19:51:58 UTC, Dragos Carp wrote:

 I have collected a few pros/cons about merging the 
 repositories:


 This topic has been discussed in the past. Some more points 
 that I can think of:


 Pro:
 - simplified release tagging and branching
 - atomic commit of cross-repository changes
 - easier to experiment with cross-repository feature branches
 - single pull request queue offering a better overview about 
 the
 project
 - easier grep, easier build
 - simplified build documentation


 - easier to run the entire test suite
 - much easier "git bisect"


 Cons:
 - migration effort (documentation, merge scripts)
 - current work-flow adjustments
 - the resulted repo history could be sometimes confusing
 - lost github pull-request history


 - more difficult to assign ownership/responsibility


The directory structure will still be present. I don't think that
would be a problem, it works pretty well for bigger projects.


 - forking just one component becomes more difficult


Forking a component is a seldom event, working with all three is
the rule and we should strive to optimize it.


 - more mixing of free and non-free source code in the same 
 repository (although I heard splitting the DMD repo into two 
 (frontend and backend) repositories was being discussed)
Sep 08 2014
parent Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
Personally, I've never found the multiple repositories inconvenient. 
About the only place they are are when simultaneous changes are required 
to more than one of the parts.  That's INTENDED to be rare since it 
directly implies a non backwards compatible change.  Those changes tend 
to hurt people and making it more obvious that it's happening and making 
the bar a little higher to leap isn't a bad thing.

About the only pro listed below that I can agree with is the atomic 
commit issue (see above for my feelings on those) and the regression 
hunt.  There's already tooling to deal with the latter, and it works 
pretty well.  The rest fall into the "yes, you're right, but the benefit 
it minimal at best.

The con's are more significant to me.  The permissions issues.  The 
enforced layers.  The fact that the cost of change isn't small and the 
benefits from the change are small.

I vote no.

If I had to do it all over again, I'd still split them up.

On 9/8/2014 2:24 PM, Dragos Carp via Digitalmars-d wrote:

 On Monday, 8 September 2014 at 20:56:51 UTC, Vladimir Panteleev
 wrote:

 On Monday, 8 September 2014 at 19:51:58 UTC, Dragos Carp wrote:

 I have collected a few pros/cons about merging the repositories:


 This topic has been discussed in the past. Some more points that I can
 think of:


 Pro:
 - simplified release tagging and branching
 - atomic commit of cross-repository changes
 - easier to experiment with cross-repository feature branches
 - single pull request queue offering a better overview about the
 project
 - easier grep, easier build
 - simplified build documentation


 - easier to run the entire test suite
 - much easier "git bisect"


 Cons:
 - migration effort (documentation, merge scripts)
 - current work-flow adjustments
 - the resulted repo history could be sometimes confusing
 - lost github pull-request history


 - more difficult to assign ownership/responsibility


 The directory structure will still be present. I don't think that
 would be a problem, it works pretty well for bigger projects.


 - forking just one component becomes more difficult


 Forking a component is a seldom event, working with all three is
 the rule and we should strive to optimize it.


 - more mixing of free and non-free source code in the same repository
 (although I heard splitting the DMD repo into two (frontend and
 backend) repositories was being discussed)
Sep 08 2014
prev sibling next sibling parent reply Jeremy Powers via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, Sep 8, 2014 at 12:51 PM, Dragos Carp via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 It may sound a bit radical, but I think the release process can
 be simplified a lot, if the 3 github repositories (dmd, druntime,
 and phobos) would be merged.


 ...


 Destroy!


These are separate projects/products, that just happen to be released
together.  Merging things together would obfuscate this (even more than it
is now).  How would you consume phobos with a different compiler, for
example?
Sep 08 2014
parent "Dragos Carp" <dragoscarp gmail.com> writes:
On Monday, 8 September 2014 at 21:03:07 UTC, Jeremy Powers via
Digitalmars-d wrote:

 On Mon, Sep 8, 2014 at 12:51 PM, Dragos Carp via Digitalmars-d <
 digitalmars-d puremagic.com> wrote:


 It may sound a bit radical, but I think the release process can
 be simplified a lot, if the 3 github repositories (dmd, 
 druntime,
 and phobos) would be merged.


  ...


 Destroy!


 These are separate projects/products, that just happen to be 
 released
 together.  Merging things together would obfuscate this (even 
 more than it
 is now).  How would you consume phobos with a different 
 compiler, for
 example?


This would be the job of an installer build step... or just take
the phobos directory.
Sep 08 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/8/14, 12:51 PM, Dragos Carp wrote:

 It may sound a bit radical, but I think the release process can
 be simplified a lot, if the 3 github repositories (dmd, druntime,
 and phobos) would be merged.


I'd support that if there's consensus. -- Andrei
Sep 08 2014
parent reply "Trass3r" <un known.com> writes:
You could also use submodules (or subtrees, haven't tried them 
yet).
Sep 08 2014
next sibling parent reply "Dragos Carp" <dragoscarp gmail.com> writes:
On Monday, 8 September 2014 at 21:55:47 UTC, Trass3r wrote:

 You could also use submodules (or subtrees, haven't tried them 
 yet).


AFAIK the tags and branches does not work over
submodules/subtrees and you would still have 3 github repos with 
3 pull request queues.
Sep 08 2014
parent reply "Trass3r" <un known.com> writes:
 with 3 pull request queues


Good argument for the separation :)
Sep 08 2014
parent Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/8/2014 3:12 PM, Trass3r via Digitalmars-d wrote:

 with 3 pull request queues


 Good argument for the separation :)


And they're visible together via the auto-tester which happens to keep 
the lists concatenated.  I don't see the separation to be an issue either.
Sep 08 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/8/14, 2:55 PM, Trass3r wrote:

 You could also use submodules (or subtrees, haven't tried them yet).


A git expert told me submodules are fail. Is that true? -- Andrei
Sep 08 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 00:28:38 UTC, Andrei Alexandrescu 
wrote:

 On 9/8/14, 2:55 PM, Trass3r wrote:

 You could also use submodules (or subtrees, haven't tried them 
 yet).


 A git expert told me submodules are fail. Is that true? -- 
 Andrei


As any powerful feature it can be used to great success or great 
failure :) However it is probably a bad case for deeply connected 
projects that needs to be updated in sync like it is with dmd and 
phobos. Those are more useful as a git-only alternative to dub 
dependencies in my opinion - static relations that don't change 
often.

`git subtree` may be a betetr match here but I am not familiar 
with those. Going to check the docs now.
Sep 08 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 00:51:25 UTC, Dicebot wrote:

 `git subtree` may be a betetr match here but I am not familiar 
 with those. Going to check the docs now.


Yeah those definitely look more interesting - as far as I can see 
subtree is bound to a branch and not to specific commit hash.
Sep 08 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 09/09/14 02:56, Dicebot wrote:


 Yeah those definitely look more interesting - as far as I can see
 subtree is bound to a branch and not to specific commit hash.


Since Git 1.8.2 you can bound a submodule to a branch.

-- 
/Jacob Carlborg
Sep 08 2014
parent "Trass3r" <un known.com> writes:
 Since Git 1.8.2 you can bound a submodule to a branch.


Ah cool didn't know that.
Sep 09 2014
prev sibling next sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Monday, 8 September 2014 at 19:51:58 UTC, Dragos Carp wrote:

 Cons:
 - migration effort (documentation, merge scripts)
 - current work-flow adjustments
 - the resulted repo history could be sometimes confusing
 - lost github pull-request history


It would also be a pretty serious change for LDC. We are 
currently using druntime/Phobos as Git submodules.

David
Sep 08 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 8 September 2014 at 19:51:58 UTC, Dragos Carp wrote:

 I have collected a few pros/cons about merging the repositories:

 Pro:
 - simplified release tagging and branching
 - atomic commit of cross-repository changes
 - easier to experiment with cross-repository feature branches
 - single pull request queue offering a better overview about the
 project
 - easier grep, easier build
 - simplified build documentation

 Cons:
 - migration effort (documentation, merge scripts)
 - current work-flow adjustments
 - the resulted repo history could be sometimes confusing
 - lost github pull-request history

 What do you think about this? Would it be worth the effort?
 Destroy!


ne important (in my opinion) disadvantage misssing: considerably 
harder PR maintenance (missing automatic categorization and 
separation between phobos / dmd teams). Being forced to look ar 
DMD pull requests to do Phobos maintenance will add lot of 
context overhead for me.

I also don't feel like it will help much for release preparation. 
Bisection and history investigation - undoubtedly. But for 
release management building stuff is one of the easier parts.
Sep 08 2014
next sibling parent Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/8/2014 3:51 PM, Dicebot via Digitalmars-d wrote:

 I also don't feel like it will help much for release preparation.
 Bisection and history investigation - undoubtedly. But for release
 management building stuff is one of the easier parts.


I totally agree with this.  Anyone that believes that the checkout, 
build, and test steps are anywhere near hard or the hard part of the 
release process really doesn't have a good understanding of what it 
takes.  Those are the _easy_ steps, and so thoroughly 
automated/automatable that it's just a non-issue.
Sep 08 2014
prev sibling parent reply "Dragos Carp" <dragoscarp gmail.com> writes:
 ne important (in my opinion) disadvantage misssing: 
 considerably harder PR maintenance (missing automatic 
 categorization and separation between phobos / dmd teams). 
 Being forced to look ar DMD pull requests to do Phobos 
 maintenance will add lot of context overhead for me.


This kind of attitude worries me: every team works happily in its 
own cozy repository and it doesn't feel responsible about the 
other repos. The consequence is that, while breeding a new 
release, the release manager MUST take most of the burden about 
coordinating the teams and their work. This is a very complex 
task which requires insight knowledge in all 3 repos.

Having 3 repos, one team is not encouraged to help the other team 
to get the release out the door. This is because one team have a 
higher bar to help the other: it needs to checkout the right 
version of the foreign repo, manage own branches in the foreign 
repo, etc. Maybe this is the reason of multiple regression in the 
last release or the "virtual keyword surprise" (at least for some 
people).

I think that a 2 phase process similar to the Linux kernel 
release process [1] would work much better and motivates all the 
teams to help with the release:

Phase 1. Merge window: merge PRs containing new features
Phase 2. Test and stabilization: merge bugfix PRs only

Ideally the first phase is quite short (1 week for example) and 
it can be close supervised (or even executed??) by Walter and/or 
Andrei.

The second phase will continue until the acceptance criteria are 
met, the teams having a focus on the release (they get no new 
goodies till the test phase is done).

The normal development happens somehow offline by preparing 
quality PRs to be merged in phase 1.


 I also don't feel like it will help much for release 
 preparation. Bisection and history investigation - undoubtedly. 
 But for release management building stuff is one of the easier 
 parts.


Agree, the technical build steps of the release process are easy. 
Just the casual experimenting with D is somehow clumsy.

[1] - 
https://www.kernel.org/doc/Documentation/development-process/2.Process
Sep 09 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 09:59:37 UTC, Dragos Carp wrote:

 ne important (in my opinion) disadvantage misssing: 
 considerably harder PR maintenance (missing automatic 
 categorization and separation between phobos / dmd teams). 
 Being forced to look ar DMD pull requests to do Phobos 
 maintenance will add lot of context overhead for me.


 This kind of attitude worries me: every team works happily in 
 its own cozy repository and it doesn't feel responsible about 
 the other repos. The consequence is that, while breeding a new 
 release, the release manager MUST take most of the burden about 
 coordinating the teams and their work. This is a very complex 
 task which requires insight knowledge in all 3 repos.


This is not about happiness or zone of comfort - I am simply not 
competent to judge even most trivial DMD pull request as opposed 
to Phobos ones. Team separation exists for a reason - skills and 
knowledge required are totally different. Some developers do both 
but I doubt it is even the majority of active maintainers. By 
merging repos you are likely to discourage me from paying 
attention at all, not to invest huge amount of effort in 
increasing DMD proficiency.

I am also not sure what what coordination you are speaking about. 
Integrity of 3 repos is verified by auto-teser. Cross-repo issues 
are rare and usually handled by those who has merge access to 
both. Release manager shouldn't normally need any detailed 
knowledge of repo internal development and just tag certain 
branch heads. If this is not the case we have some hidden problem 
I am not aware of.


 Having 3 repos, one team is not encouraged to help the other 
 team to get the release out the door. This is because one team 
 have a higher bar to help the other: it needs to checkout the 
 right version of the foreign repo, manage own branches in the 
 foreign repo, etc. Maybe this is the reason of multiple 
 regression in the last release or the "virtual keyword 
 surprise" (at least for some people).


It feels like you have imagined some process that does not really 
match reality :( Explicit tracking of matching version is never 
needed - it is either branch head for all repos, or identically 
named tags.

I have already explained above some of the real reasons that make 
closer collaboration close to impossible.


 I think that a 2 phase process similar to the Linux kernel 
 release process [1] would work much better and motivates all 
 the teams to help with the release:

 Phase 1. Merge window: merge PRs containing new features
 Phase 2. Test and stabilization: merge bugfix PRs only

 Ideally the first phase is quite short (1 week for example) and 
 it can be close supervised (or even executed??) by Walter 
 and/or Andrei.

 The second phase will continue until the acceptance criteria 
 are met, the teams having a focus on the release (they get no 
 new goodies till the test phase is done).

 The normal development happens somehow offline by preparing 
 quality PRs to be merged in phase 1.



This all sounds like solving the problem that doesn't exist 
instead of addressing the real ones (lack of clear documented 
rules for release packaging and awkward cherry-pick based release 
branch maintenance)
Sep 09 2014
parent reply "Dicebot" <public dicebot.lv> writes:
Also it sounds as if you think that someone actually does any 
coordination about what must go into release. As far as I am 
aware there is no such thing, even http://wiki.dlang.org/Agenda 
is just a convention. Currently releases are based exclusively on 
a time frame + regression list (all that was in master goes to 
the release branch and is kept there until known regressions are 
fixed, repeat for the next cycle).
Sep 09 2014
parent reply "Dragos Carp" <dragoscarp gmail.com> writes:
On Tuesday, 9 September 2014 at 12:31:26 UTC, Dicebot wrote:

 Also it sounds as if you think that someone actually does any 
 coordination about what must go into release. As far as I am 
 aware there is no such thing, even http://wiki.dlang.org/Agenda 
 is just a convention. Currently releases are based exclusively 
 on a time frame + regression list (all that was in master goes 
 to the release branch and is kept there until known regressions 
 are fixed, repeat for the next cycle).


Are you satisfied with the current process?

Let me summarize some important drawbacks of the current workflow:

1. No clear defined deadline for preparing a merge-able PR.

2. Unorganized PR merge campaigns. The people merging the PR are 
doing a great job, but they do this triggered by arbitrary 
events: too many open PRs, a cool new PR appears, somebody poke 
them on forum, or simply have some time for this kind of work.

3. Somehow arbitrary merge criteria. Having a defined merge 
window, when some people do just PR merges, will implicitly 
produce more predictable and uniform acceptance criteria.

4. Lack of focus during test phase. Maybe this is the main reason 
for the v2.066 regressions. Some people keep merging new PRs, 
before the old ones are proven done during the test phase. Even 
Walter was annoyed a couple of times by the multitude of versions 
that the people are simultaneously working on.

5. Rotting old PRs. The "merge window" phase would be a defined 
recurrent occasion to review and decide about those.
Sep 09 2014
next sibling parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/9/2014 6:54 AM, Dragos Carp via Digitalmars-d wrote:

 On Tuesday, 9 September 2014 at 12:31:26 UTC, Dicebot wrote:

 Also it sounds as if you think that someone actually does any
 coordination about what must go into release. As far as I am aware
 there is no such thing, even http://wiki.dlang.org/Agenda is just a
 convention. Currently releases are based exclusively on a time frame +
 regression list (all that was in master goes to the release branch and
 is kept there until known regressions are fixed, repeat for the next
 cycle).


 Are you satisfied with the current process?

 Let me summarize some important drawbacks of the current workflow:

 1. No clear defined deadline for preparing a merge-able PR.

 2. Unorganized PR merge campaigns. The people merging the PR are doing a
 great job, but they do this triggered by arbitrary events: too many open
 PRs, a cool new PR appears, somebody poke them on forum, or simply have
 some time for this kind of work.

 3. Somehow arbitrary merge criteria. Having a defined merge window, when
 some people do just PR merges, will implicitly produce more predictable
 and uniform acceptance criteria.

 4. Lack of focus during test phase. Maybe this is the main reason for
 the v2.066 regressions. Some people keep merging new PRs, before the old
 ones are proven done during the test phase. Even Walter was annoyed a
 couple of times by the multitude of versions that the people are
 simultaneously working on.

 5. Rotting old PRs. The "merge window" phase would be a defined
 recurrent occasion to review and decide about those.


Of course the process can be better.  But NONE of those are a result of 
the repository split that you're advocating removing.
Sep 09 2014
parent "Dragos Carp" <dragoscarp gmail.com> writes:
On Tuesday, 9 September 2014 at 18:56:16 UTC, Brad Roberts via
Digitalmars-d wrote:

 Of course the process can be better.  But NONE of those are a 
 result of the repository split that you're advocating removing.


I think these are related. It will be a real challenge to keep in
sync three in parallel running release processes. Keeping in mind
that each team has own development pace, man-power, or
undertakings, how do you avoid that the three release processes
does not diverge between the teams?
Sep 09 2014
prev sibling next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 9/9/2014 9:54 AM, Dragos Carp wrote:

 Are you satisfied with the current process?

 Let me summarize some important drawbacks of the current workflow:

 1. No clear defined deadline for preparing a merge-able PR.

 2. Unorganized PR merge campaigns. The people merging the PR are doing a
 great job, but they do this triggered by arbitrary events: too many open
 PRs, a cool new PR appears, somebody poke them on forum, or simply have
 some time for this kind of work.

 3. Somehow arbitrary merge criteria. Having a defined merge window, when
 some people do just PR merges, will implicitly produce more predictable
 and uniform acceptance criteria.

 4. Lack of focus during test phase. Maybe this is the main reason for
 the v2.066 regressions. Some people keep merging new PRs, before the old
 ones are proven done during the test phase. Even Walter was annoyed a
 couple of times by the multitude of versions that the people are
 simultaneously working on.

 5. Rotting old PRs. The "merge window" phase would be a defined
 recurrent occasion to review and decide about those.


Most of that is unavoidable without salaried devs on it. D is 100% 
volunteer, deadlines and such aren't realistically feasible.
Sep 09 2014
parent reply "Dragos Carp" <dragoscarp gmail.com> writes:
On Tuesday, 9 September 2014 at 22:41:39 UTC, Nick Sabalausky
wrote:

 Most of that is unavoidable without salaried devs on it. D is 
 100% volunteer, deadlines and such aren't realistically 
 feasible.


I'm talking about the deadlines for your own PRs. If you miss one
release, you will have your chance in the next one. Of course you
will still freely choose for yourself the tasks that suit you
best. Beside that, having a deadline it is a great motivational
factor to do even some unpleasant work (for me at least).
Sep 09 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/9/14, 4:06 PM, Dragos Carp wrote:

 On Tuesday, 9 September 2014 at 22:41:39 UTC, Nick Sabalausky
 wrote:

 Most of that is unavoidable without salaried devs on it. D is 100%
 volunteer, deadlines and such aren't realistically feasible.


 I'm talking about the deadlines for your own PRs. If you miss one
 release, you will have your chance in the next one. Of course you
 will still freely choose for yourself the tasks that suit you
 best. Beside that, having a deadline it is a great motivational
 factor to do even some unpleasant work (for me at least).


Yah, we should have deadlines. -- Andrei
Sep 10 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 10 September 2014 at 07:43:22 UTC, Andrei 
Alexandrescu wrote:

 Yah, we should have deadlines. -- Andrei


We have them already, it just happens that those are never 
realistically executed.

http://wiki.dlang.org/Release_Process#Release_Schedule
Sep 10 2014
parent Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/10/14, Dicebot via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Wednesday, 10 September 2014 at 07:43:22 UTC, Andrei
 Alexandrescu wrote:

 Yah, we should have deadlines. -- Andrei


 We have them already, it just happens that those are never
 realistically executed.

 http://wiki.dlang.org/Release_Process#Release_Schedule


“I love deadlines. I love the whooshing noise they make as they go
by.”  - Douglas Adams
Sep 10 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 13:54:15 UTC, Dragos Carp wrote:

 Are you satisfied with the current process?


Yes. I am not satisfied with certain parts of implementation of 
the process but the concept overall is a good as it can possibly 
be given manpower available and lack of any centralized 
collaboration. Think about it - despite the fact that there is no 
one actually responsible for release planning and no fully 
commited developers we still somehow manage to get those release 
out. This is quite an achievment if you think about it.


 Let me summarize some important drawbacks of the current 
 workflow:

 1. No clear defined deadline for preparing a merge-able PR.


.. which allows maintainers to merge pull requests when they seem 
ready without any synchronization "locks" and does not interfere 
with release branches.


 2. Unorganized PR merge campaigns. The people merging the PR 
 are doing a great job, but they do this triggered by arbitrary 
 events: too many open PRs, a cool new PR appears, somebody poke 
 them on forum, or simply have some time for this kind of work.


There is no such thing as "PR merge campaign" and there is not 
point in having one. It is everlasting process - stuff gets 
merged when its done, it is completly irrelevant to release 
planning. "simply have some time for this kind of work" is the 
only realistic criteria for any sort of maintenance activity 
here, you can't get any other without bunch of devs on salary.

It is a process which is almost exclusively blocked by maintainer 
time. It is quite telling that situation with Phobos (which has 
easier learning curve and more active maintainers overall) is 
much better than with DMD.


 3. Somehow arbitrary merge criteria. Having a defined merge 
 window, when some people do just PR merges, will implicitly 
 produce more predictable and uniform acceptance criteria.


What makes you think so? It will only introduce arbitrary 
limitation on a time when you can merge pull requests. Merge 
criteria is an orthogonal topic - if it is possible to formalize 
one, it can also be done without introducing merge windows. 
Implicit uniform criteria is much better defined by the simple 
fact that those are the very same people doing merges (and there 
aren't many)

Only thing merge window will achieve is making pull reqeuests 
stockpile even more because developers won't be allowed to merge 
them when they have time to do so.


 4. Lack of focus during test phase. Maybe this is the main 
 reason for the v2.066 regressions. Some people keep merging new 
 PRs, before the old ones are proven done during the test phase. 
 Even Walter was annoyed a couple of times by the multitude of 
 versions that the people are simultaneously working on


Main reason is that there are only few developers capable of 
fixing DMD regressions. If release beta happens when either of 
them is busy it will result in immediate slowdown. This is 
especially true about Kenji.

Do you realize that fixing release and merging new pull requests 
happens in _different_ branches? And merging even 1000 new 
features doesn't affect testing of the release beta? There were 
certain bad cases when major refactorings in master make it hard 
to port bug fixes to release branch but those were rare.


 5. Rotting old PRs. The "merge window" phase would be a defined 
 recurrent occasion to review and decide about those.


There are no rotting old PRs now in Phobos despite the 
maintenance / release process is the same. Guess why? With recent 
addition of bunch of new maintainers we simply got the resources 
to clean and categorize those and this just immediately happened.

Unfortunately, there is no magic process can just create new 
developers out of the air :(
Sep 09 2014
prev sibling parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Dragos Carp"  wrote in message news:qjcqyddvsaopicjmeuoy forum.dlang.org...


 It may sound a bit radical, but I think the release process can
 be simplified a lot, if the 3 github repositories (dmd, druntime,
 and phobos) would be merged.


Even if there were no downsides, it still wouldn't be worth the disruption.
Sep 08 2014
prev sibling parent reply Andrew Edwards <ridimz yahoo.com> writes:
On 9/8/14, 10:30 AM, Andrei Alexandrescu wrote:

 Andrew Edwards has done a great job with the recent release, but needs
 to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the release
 process. Please tender your application by replying to this.


 Thanks,

 Andrei



Follow these steps to prepare your build environment:

     1. Install VirtualBox: https://www.virtualbox.org/wiki/Downloads
     2. Install Vagrant: https://www.vagrantup.com/downloads.html
     3. Clone installer: git clone --recursive 
https://github.com/D-Programming-Language/installer.git

Note 1: MartinNowak already prepared boxes for FreeBSD and Debian which 
are directly imported through the build script. However, you must 
prepare your own OSX and Windows 7 boxes. Instructions for preparing 
doing so are located here:

	OSX => https://gist.github.com/MartinNowak/8156507
	Windows7 => https://gist.github.com/MartinNowak/8270666

Steps for building are outlined here:
	http://wiki.dlang.org/Simplified_Release_Process_Proposal

Tracking is done here:
	http://wiki.dlang.org/Beta_Testing

Note 2: I am not vacating the post of Build Master. I have some personal 
matters that require a significant amount of my time over the next 
couple of months. I cannot promise to be on time with the releases 
before October 15. Until then, I will do my best to prepare releases as 
time permits.

As it stands: If there are no pending merges for the 2.066.1 branch, I 
will be able to prepare a RC before morning (time now is 12:58 AM JST).
Sep 09 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 15:58:45 UTC, Andrew Edwards 
wrote:

 On 9/8/14, 10:30 AM, Andrei Alexandrescu wrote:

 Andrew Edwards has done a great job with the recent release, 
 but needs
 to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the 
 release
 process. Please tender your application by replying to this.


 Thanks,

 Andrei



 Follow these steps to prepare your build environment:

     1. Install VirtualBox: 
 https://www.virtualbox.org/wiki/Downloads
     2. Install Vagrant: https://www.vagrantup.com/downloads.html
     3. Clone installer: git clone --recursive 
 https://github.com/D-Programming-Language/installer.git

 Note 1: MartinNowak already prepared boxes for FreeBSD and 
 Debian which are directly imported through the build script. 
 However, you must prepare your own OSX and Windows 7 boxes. 
 Instructions for preparing doing so are located here:

 	OSX => https://gist.github.com/MartinNowak/8156507
 	Windows7 => https://gist.github.com/MartinNowak/8270666

 Steps for building are outlined here:
 	http://wiki.dlang.org/Simplified_Release_Process_Proposal

 Tracking is done here:
 	http://wiki.dlang.org/Beta_Testing

 Note 2: I am not vacating the post of Build Master. I have some 
 personal matters that require a significant amount of my time 
 over the next couple of months. I cannot promise to be on time 
 with the releases before October 15. Until then, I will do my 
 best to prepare releases as time permits.

 As it stands: If there are no pending merges for the 2.066.1 
 branch, I will be able to prepare a RC before morning (time now 
 is 12:58 AM JST).


Thanks Andrew! This is most helpful. I will try this out this 
weekend to be able to act as backup build master for 2.067
Sep 09 2014
prev sibling next sibling parent "NVolcz" <volcz kth.se> writes:
On Tuesday, 9 September 2014 at 15:58:45 UTC, Andrew Edwards 
wrote:

 On 9/8/14, 10:30 AM, Andrei Alexandrescu wrote:

 Andrew Edwards has done a great job with the recent release, 
 but needs
 to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the 
 release
 process. Please tender your application by replying to this.


 Thanks,

 Andrei


 Note 2: I am not vacating the post of Build Master. I have some 
 personal matters that require a significant amount of my time 
 over the next couple of months. I cannot promise to be on time 
 with the releases before October 15. Until then, I will do my 
 best to prepare releases as time permits.

 As it stands: If there are no pending merges for the 2.066.1 
 branch, I will be able to prepare a RC before morning (time now 
 is 12:58 AM JST).


How much of your free time has the Build Mater/release lieutenant 
role taken?
Sep 10 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 9 September 2014 at 15:58:45 UTC, Andrew Edwards 
wrote:

 On 9/8/14, 10:30 AM, Andrei Alexandrescu wrote:

 Andrew Edwards has done a great job with the recent release, 
 but needs
 to step down because he's busy with other pursuits.

 We need a release lieutenant who would carry us through the 
 release
 process. Please tender your application by replying to this.


 Thanks,

 Andrei



 Follow these steps to prepare your build environment:

     1. Install VirtualBox: 
 https://www.virtualbox.org/wiki/Downloads
     2. Install Vagrant: https://www.vagrantup.com/downloads.html
     3. Clone installer: git clone --recursive 
 https://github.com/D-Programming-Language/installer.git

 Note 1: MartinNowak already prepared boxes for FreeBSD and 
 Debian which are directly imported through the build script. 
 However, you must prepare your own OSX and Windows 7 boxes. 
 Instructions for preparing doing so are located here:

 	OSX => https://gist.github.com/MartinNowak/8156507
 	Windows7 => https://gist.github.com/MartinNowak/8270666

 Steps for building are outlined here:
 	http://wiki.dlang.org/Simplified_Release_Process_Proposal

 Tracking is done here:
 	http://wiki.dlang.org/Beta_Testing


Trying this right now, got through Win7 box setup but where one 
is supposed to get "InstallESD.dmg" mentioned in installation 
instructions?
Sep 20 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-09-20 16:44, Dicebot wrote:


 Trying this right now, got through Win7 box setup but where one is
 supposed to get "InstallESD.dmg" mentioned in installation instructions?


As the instructions say, from the "Install Mac OS X Mountain Lion.app" 
bundle. This bundle is downloaded from the Mac App Store. For Mavericks 
it's located in "Install OS X 
Mavericks.app/Contents/SharedSupport/InstallESD.dmg". If you download it 
form the app store it's will be placed in /Applications.

BTW you should only do this on a Mac. Technically it's possible to do on 
another platform.

-- 
/Jacob Carlborg
Sep 20 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 15:16:20 UTC, Jacob Carlborg 
wrote:

 BTW you should only do this on a Mac. Technically it's possible 
 to do on another platform.


Ah I see. Guess this is the point where I should say "Thanks, no, 
I am not going to volunteer".

Can do linux-only releases if anyone is interested ;)
Sep 20 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 8:23 AM, Dicebot wrote:

 On Saturday, 20 September 2014 at 15:16:20 UTC, Jacob Carlborg wrote:

 BTW you should only do this on a Mac. Technically it's possible to do
 on another platform.


 Ah I see. Guess this is the point where I should say "Thanks, no, I am
 not going to volunteer".

 Can do linux-only releases if anyone is interested ;)


Any help is welcome, but we do need a point of contact.

Who wants to be our build lieutenant?


Thanks,

Andrei
Sep 20 2014