www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - GCs are complicated beasts: .NET Core, 35000 lines

reply "Tourist" <gravatar gravatar.com> writes:
Source code:
https://raw.githubusercontent.com/dotnet/coreclr/master/src/gc/gc.cpp

Origin:
http://www.reddit.com/r/programming/comments/2unrll/net_core_is_on_github/coa2uot
Feb 03 2015
parent reply "Orvid King" <blah38621 gmail.com> writes:
On Tuesday, 3 February 2015 at 22:19:08 UTC, Tourist wrote:
 Source code:
 https://raw.githubusercontent.com/dotnet/coreclr/master/src/gc/gc.cpp

 Origin:
 http://www.reddit.com/r/programming/comments/2unrll/net_core_is_on_github/coa2uot
The thing to realize about the .net GC is that it is precise, generational, compacting, and asynchronous. It also has a low-latency mode which is primarily used for ASP.net, which limits the maximum time spent in performing a single collection. It also supporting pinning allocations. In comparison, D's GC is simple, as are most GC implementations.
Feb 03 2015
next sibling parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 3 February 2015 at 22:27:38 UTC, Orvid King wrote:
 On Tuesday, 3 February 2015 at 22:19:08 UTC, Tourist wrote:
 Source code:
 https://raw.githubusercontent.com/dotnet/coreclr/master/src/gc/gc.cpp

 Origin:
 http://www.reddit.com/r/programming/comments/2unrll/net_core_is_on_github/coa2uot
The thing to realize about the .net GC is that it is precise, generational, compacting, and asynchronous. It also has a low-latency mode which is primarily used for ASP.net, which limits the maximum time spent in performing a single collection. It also supporting pinning allocations. In comparison, D's GC is simple, as are most GC implementations.
And I bet commercial JVMs even have more LOCs given the pluggable set of GCs they come with. Oracle is planning to drop some of them by Java 9 exactly because of maintenance costs. http://openjdk.java.net/jeps/214 -- Paulo
Feb 03 2015
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2015-02-03 at 22:40 +0000, Paulo Pinto via Digitalmars-d wrote:
 […]
 
 And I bet commercial JVMs even have more LOCs given the pluggable 
 set of GCs they come with.
On the other hand does anyone use anything other than G1 these days?
 Oracle is planning to drop some of them by Java 9 exactly because of 
 maintenance costs.
 
 http://openjdk.java.net/jeps/214
And because G1 provably does the job. BTW The Go team have completely rewritten the Go GC 1.4 → 1.5. Of course they have also completely rewritten the entire Go toolchain. Compiler, linker, runtime all written in Go now, no more C. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Feb 04 2015
next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Wednesday, 4 February 2015 at 12:09:36 UTC, Russel Winder 
wrote:
 On Tue, 2015-02-03 at 22:40 +0000, Paulo Pinto via 
 Digitalmars-d wrote:
 […]
 
 And I bet commercial JVMs even have more LOCs given the 
 pluggable set of GCs they come with.
On the other hand does anyone use anything other than G1 these days?
Yes. You just have to witness a discussion on HN or Reddit to see how the majority of developers that don't optimize for performance are unaware of GC options across the Java world.
 Oracle is planning to drop some of them by Java 9 exactly 
 because of maintenance costs.
 
 http://openjdk.java.net/jeps/214
And because G1 provably does the job. BTW The Go team have completely rewritten the Go GC 1.4 → 1.5. Of course they have also completely rewritten the entire Go toolchain. Compiler, linker, runtime all written in Go now, no more C.
I really like that they decided to do that. First because I always enjoyed following the bootstrap route when playing with compiler development back in my degree days. Second, because it gets rid of C so it earns brownie points. Many times language designers use C due to multiple reasons, some of them make good sense like using it as portable assembler or fast bootstrap of the runtime, and not because it isn't possible to avoid it. However the naysayers without compiler design background, use the C dependency as argument for language X cannot be done without C, ergo C rulez. So looking forward for Go 1.5 and also SDC.
Feb 04 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/4/15 4:09 AM, Russel Winder via Digitalmars-d wrote:
 BTW The Go team have completely rewritten the Go GC 1.4 → 1.5. Of
 course they have also completely rewritten the entire Go toolchain.
 Compiler, linker, runtime all written in Go now, no more C.
How does the rewrite compare against the originals? -- Andrei
Feb 04 2015
next sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 2015-02-04 at 08:31 -0800, Andrei Alexandrescu via Digitalmars-d wrote:
 On 2/4/15 4:09 AM, Russel Winder via Digitalmars-d wrote:
 BTW The Go team have completely rewritten the Go GC 1.4 → 1.5. Of 
 course they have also completely rewritten the entire Go 
 toolchain. Compiler, linker, runtime all written in Go now, no 
 more C.
How does the rewrite compare against the originals? -- Andrei
I haven't tried them as yet. I switched from compiling the Go toolchain from Mercurial to just using released distributions (*). I think I will have to clone the Git repository and compile from there (**) (*) Fedora Rawhide packages take about 2-days to be released, Debian Sid is in freeze so they are already so far out of date with Go as to be annoying. (**) Yes you read that right, the Go team have switched from Mercurial to Git. This is not because they were unhappy with Mercurial, it is because they were unhappy with Rietveld and switched to Gerrit for their changeset handing. The Go team do not use pull requests at all, everything goes through a review manager, now that is Gerrit. Mayhap D could follow the Go example? -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Feb 04 2015
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/4/2015 10:31 AM, Russel Winder via Digitalmars-d wrote:
 The Go team do not use pull requests at all,
 everything goes through a review manager, now that is Gerrit. Mayhap D
 could follow the Go example?
We'd need an awful good reason to ditch the PR system we use now.
Feb 04 2015
parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 2015-02-04 at 12:12 -0800, Walter Bright via Digitalmars-d wrote:
 On 2/4/2015 10:31 AM, Russel Winder via Digitalmars-d wrote:
 The Go team do not use pull requests at all,
 everything goes through a review manager, now that is Gerrit.=20
 Mayhap D could follow the Go example?
=20 We'd need an awful good reason to ditch the PR system we use now.
If it works fine. All I was reporting is that The Go team felt the=20 GitHub pull request support was significantly inferior to using Gerrit=20 to manage review and merge of changesets.=20 --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Feb 06 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/4/15 10:31 AM, Russel Winder via Digitalmars-d wrote:
 (**) Yes you read that right, the Go team have switched from Mercurial
 to Git. This is not because they were unhappy with Mercurial, it is
 because they were unhappy with Rietveld and switched to Gerrit for
 their changeset handing. The Go team do not use pull requests at all,
 everything goes through a review manager, now that is Gerrit. Mayhap D
 could follow the Go example?
Not sure what you're proposing. I am positive we are on git and not on mercurial :o). -- Andrei
Feb 04 2015
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 2015-02-04 at 12:17 -0800, Andrei Alexandrescu via Digitalmars-d wr=
ote:
 On 2/4/15 10:31 AM, Russel Winder via Digitalmars-d wrote:
 (**) Yes you read that right, the Go team have switched from=20
 Mercurial to Git. This is not because they were unhappy with=20
 Mercurial, it is because they were unhappy with Rietveld and=20
 switched to Gerrit for their changeset handing. The Go team do not=20
 use pull requests at all, everything goes through a review=20
 manager, now that is Gerrit. Mayhap D could follow the Go example?
=20 Not sure what you're proposing. I am positive we are on git and not=20 on mercurial :o). -- Andrei
All I was was saying was that the Go team, along with other teams in=20 Google, feel the GitHub pull request infrastructure is not good enough=20 to enable the sort of review of changesets that leads to consistently=20 high quality code bases. The bit at the end was really asking whether=20 the issues getting codebase changesets reviewed in the D community=20 might be aided by going the same route. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Feb 06 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 4:36 AM, Russel Winder via Digitalmars-d wrote:
 On Wed, 2015-02-04 at 12:17 -0800, Andrei Alexandrescu via Digitalmars-d wrote:
 On 2/4/15 10:31 AM, Russel Winder via Digitalmars-d wrote:
 (**) Yes you read that right, the Go team have switched from
 Mercurial to Git. This is not because they were unhappy with
 Mercurial, it is because they were unhappy with Rietveld and
 switched to Gerrit for their changeset handing. The Go team do not
 use pull requests at all, everything goes through a review
 manager, now that is Gerrit. Mayhap D could follow the Go example?
Not sure what you're proposing. I am positive we are on git and not on mercurial :o). -- Andrei
All I was was saying was that the Go team, along with other teams in Google, feel the GitHub pull request infrastructure is not good enough to enable the sort of review of changesets that leads to consistently high quality code bases. The bit at the end was really asking whether the issues getting codebase changesets reviewed in the D community might be aided by going the same route.
I see. Not surprising - some teams at Facebook also prefer phabricator over github for reviews (others don't). I'm using both and phabricator is somewhat - but not dramatically - better. As an leader of this community I must make quick decisions with little information at hand on whether something is an issue, and then be ready to actually act on it. My current assessment is that the github infrastructure is not among our bottlenecks for the time being. This assessment may change if a champion pushes the matter forward. Andrei
Feb 06 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-02-06 17:17, Andrei Alexandrescu wrote:

 I see. Not surprising - some teams at Facebook also prefer phabricator
 over github for reviews (others don't). I'm using both and phabricator
 is somewhat - but not dramatically - better.
Any specifics you can point out where phabricator is better? -- /Jacob Carlborg
Feb 07 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/7/15 6:26 AM, Jacob Carlborg wrote:
 On 2015-02-06 17:17, Andrei Alexandrescu wrote:

 I see. Not surprising - some teams at Facebook also prefer phabricator
 over github for reviews (others don't). I'm using both and phabricator
 is somewhat - but not dramatically - better.
Any specifics you can point out where phabricator is better?
E.g. comments are batched together, making one review one cohesive unit. People can't edit one another's comments. -- Andrei
Feb 07 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-02-04 17:31, Andrei Alexandrescu wrote:

 How does the rewrite compare against the originals? -- Andrei
I don't have much practical experience with Go but I think that the custom linker makes the overall build time faster. For D, the linker is a problem, even though the compiler is fast but the overall user experience counts too. -- /Jacob Carlborg
Feb 05 2015
prev sibling parent reply "Muahmmad Adel" <devadel gmail.com> writes:
On a previous reddit thread, Andrei Alexandrescu said that he 
will be rewriting the GC based on the core allocator. However 
there were no more details about this anticipated garbage 
collector features.

So will the garbage collector be a generational one? will it 
allow some sort of control over latency of garbage collection ( 
allowing a low latency option or something similar).

We had this case that we faced with ElasticSearch, where garbage 
collection firing non-deterministically (non deterministic firing 
time and non deterministic garbage collection time) causes one 
node to suddenly be not responding, increasing load over other 
nodes, which in turn when having higher load go into garbage 
collection halt one after the other, which causes the whole 
system to be unresponsive for a good amount of time.

This is an  example of thundering herd anti pattern where one 
event causes all nodes to go busy.

I am not a java developer but I think that java's GC allows more 
control over things and ElasticSearch simply didn't use these 
controls or used them inefficiently.

However, with the increasing trend towards distributed server 
applications, having more control over the garbage collector's 
performance parameters would be crucial for pushing D to this 
area.
Feb 06 2015
next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Friday, 6 February 2015 at 09:38:48 UTC, Muahmmad Adel wrote:
 On a previous reddit thread, Andrei Alexandrescu said that he 
 will be rewriting the GC based on the core allocator. However 
 there were no more details about this anticipated garbage 
 collector features.

 So will the garbage collector be a generational one? will it 
 allow some sort of control over latency of garbage collection ( 
 allowing a low latency option or something similar).

 We had this case that we faced with ElasticSearch, where 
 garbage collection firing non-deterministically (non 
 deterministic firing time and non deterministic garbage 
 collection time) causes one node to suddenly be not responding, 
 increasing load over other nodes, which in turn when having 
 higher load go into garbage collection halt one after the 
 other, which causes the whole system to be unresponsive for a 
 good amount of time.

 This is an  example of thundering herd anti pattern where one 
 event causes all nodes to go busy.

 I am not a java developer but I think that java's GC allows 
 more control over things and ElasticSearch simply didn't use 
 these controls or used them inefficiently.

 However, with the increasing trend towards distributed server 
 applications, having more control over the garbage collector's 
 performance parameters would be crucial for pushing D to this 
 area.
Each JVM (Oracle, IBM, HP-UX, ...) has a different set of knobs to turn. http://docs.oracle.com/javase/8/docs/technotes/tools/windows/java.html#BABFAFAE And visual monitoring tools to go alongside them. In Oracle's case http://visualvm.java.net/ http://www.oracle.com/technetwork/java/javaseproducts/mission-control/java-mission-control-1998576.html However it is true that in such types of applications, there are still a few issues to tackle in Java, which D already handles without problems like real generics, value types, painless integration with C and C++ code. All issues to be fixed latest by Java 10 time-frame (Projects Panama, Valhalla and Sumatra). -- Paulo
Feb 06 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 1:38 AM, Muahmmad Adel wrote:
 On a previous reddit thread, Andrei Alexandrescu said that he will be
 rewriting the GC based on the core allocator. However there were no more
 details about this anticipated garbage collector features.
Rainer Schütze and Martin Nowak are working on significant improvements of D's GC, most of which will be released with 2.067. -- Andrei
Feb 06 2015
parent reply "Paolo Invernizzi" <paolo.invernizzi no.address> writes:
On Friday, 6 February 2015 at 16:05:47 UTC, Andrei Alexandrescu 
wrote:
 On 2/6/15 1:38 AM, Muahmmad Adel wrote:
 On a previous reddit thread, Andrei Alexandrescu said that he 
 will be
 rewriting the GC based on the core allocator. However there 
 were no more
 details about this anticipated garbage collector features.
Rainer Schütze and Martin Nowak are working on significant improvements of D's GC, most of which will be released with 2.067. -- Andrei
Sorry for the silly question: is there a PR, or is some work already merged in head? /P
Feb 06 2015
next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Fri, 06 Feb 2015 16:14:54 +0000, Paolo Invernizzi wrote:

 On Friday, 6 February 2015 at 16:05:47 UTC, Andrei Alexandrescu wrote:
 On 2/6/15 1:38 AM, Muahmmad Adel wrote:
 On a previous reddit thread, Andrei Alexandrescu said that he will be
 rewriting the GC based on the core allocator. However there were no
 more details about this anticipated garbage collector features.
Rainer Sch=C3=BCtze and Martin Nowak are working on significant improvem=
ents
 of D's GC, most of which will be released with 2.067. -- Andrei
=20 Sorry for the silly question: is there a PR, or is some work already merged in head?
there is constant (albeit small) flow of GC-related changes in commit=20 log, so it seems that some work is going on. i think that big changes are=20 not pushing due to big amount of necessary testing and possible code=20 breakage from not so-well-tested commits.=
Feb 06 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 8:14 AM, Paolo Invernizzi wrote:
 On Friday, 6 February 2015 at 16:05:47 UTC, Andrei Alexandrescu wrote:
 On 2/6/15 1:38 AM, Muahmmad Adel wrote:
 On a previous reddit thread, Andrei Alexandrescu said that he will be
 rewriting the GC based on the core allocator. However there were no more
 details about this anticipated garbage collector features.
Rainer Schütze and Martin Nowak are working on significant improvements of D's GC, most of which will be released with 2.067. -- Andrei
Sorry for the silly question: is there a PR, or is some work already merged in head?
A bunch. Just run this query: https://github.com/pulls?q=is%3Apr+user%3AD-Programming-Language+author%3AMar inNowak+is%3Aclosed -- Andrei
Feb 06 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/15 8:25 AM, Andrei Alexandrescu wrote:
 On 2/6/15 8:14 AM, Paolo Invernizzi wrote:
 On Friday, 6 February 2015 at 16:05:47 UTC, Andrei Alexandrescu wrote:
 On 2/6/15 1:38 AM, Muahmmad Adel wrote:
 On a previous reddit thread, Andrei Alexandrescu said that he will be
 rewriting the GC based on the core allocator. However there were no
 more
 details about this anticipated garbage collector features.
Rainer Schütze and Martin Nowak are working on significant improvements of D's GC, most of which will be released with 2.067. -- Andrei
Sorry for the silly question: is there a PR, or is some work already merged in head?
A bunch. Just run this query: https://github.com/pulls?q=is%3Apr+user%3AD-Programming-Language+author%3AMartinNowak+is%3Aclosed -- Andrei
Apparently choosing two authors doesn't work, here are Rainer's commits: https://github.com/pulls?q=is%3Apr+user%3AD-Programming-Language+author%3 rainers+is%3Aclosed -- Andrei
Feb 06 2015