www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Status of nogc with the runtime

reply Peter Campbell <peter spcampbell.co.uk> writes:
Hi everyone, it's been almost a year since I used D but I was 
wanted to get back into it. I was checking the 2017 vision pages 
and was wondering why there hasn't been a 2018H1 vision page but 
also what the current status of being able to use D without a 
garbage collector is?

It was noted in the H1 and H2 pages for 2017 but looking through 
the compiler changelog doesn't indicate that we're there yet? 
https://wiki.dlang.org/Vision/2017H2

Thanks.
Feb 17 2018
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Saturday, 17 February 2018 at 12:18:28 UTC, Peter Campbell 
wrote:
 I was checking the 2017 vision pages and was wondering why 
 there hasn't been a 2018H1 vision page but also what the 
 current status of being able to use D without a garbage 
 collector is?
There are a number of ways to use D without the garbage collector. However, one of the easiest is with the -betterC flag. You'll want to read the two blog articles here (https://dlang.org/blog/category/betterc/) for more information about that. 2.079, the upcoming release will also have a way to use D without the runtime and without -betterC. What this allows you to do is partially implement just the runtime features you wish to support. You can read more about it in the nightly changelog here (https://dlang.org/changelog/pending.html#minimal_runtime) You might also be interested in this article: https://wiki.dlang.org/Memory_Management Mike
Feb 17 2018
parent reply Peter Campbell <peter spcampbell.co.uk> writes:
On Saturday, 17 February 2018 at 12:32:04 UTC, Mike Franklin 
wrote:
 There are a number of ways to use D without the garbage 
 collector.  However, one of the easiest is with the -betterC 
 flag.  You'll want to read the two blog articles here 
 (https://dlang.org/blog/category/betterc/) for more information 
 about that.

 2.079, the upcoming release will also have a way to use D 
 without the runtime and without -betterC.  What this allows you 
 to do is partially implement just the runtime features you wish 
 to support.  You can read more about it in the nightly 
 changelog here 
 (https://dlang.org/changelog/pending.html#minimal_runtime)

 You might also be interested in this article: 
 https://wiki.dlang.org/Memory_Management

 Mike
I've read about BetterC and I'm interested in the minimal runtime feature, that seems like it'll be really cool. But these methods requiring not using certain D features which rely on the GC. My understanding from the vision documents and what Andrei mentioned at his DConf presentations is that the runtime itself will be modified to not rely on the GC, allowing for you to continue using features such as associative arrays and array concatenation, but without requiring the garbage collector. Is my understanding correct? If so is work still being done to make this happen and is there an easy way to follow the progress? On Saturday, 17 February 2018 at 12:44:35 UTC, Eugene Wissner wrote:
 Let me do a bit advertising :) I'm writing a library for  nogc 
 D programming, tanya. It has different aims than -betterC: I 
 use the full set of D features, just except features that 
 require GC. It isn't complete, I'm using it only in personal 
 projects and it is also a big learn project for me, so don't 
 wonder that it reimplements a lot of features "from scratch" 
 and almost doesn't depend on phobos. But if you like to play 
 with it, you're welcome:
 https://github.com/caraus-ecms/tanya
That's an interesting project, I'll be sure to take a look! On Saturday, 17 February 2018 at 14:25:41 UTC, Seb wrote:
 On Saturday, 17 February 2018 at 12:18:28 UTC, Peter Campbell 
 wrote:
 The garbage collector in druntime is now lazily initialized on 
 the first use:

 https://github.com/dlang/druntime/pull/2057

 (in master since a few days)
That's really nice to hear 2.079 is looking really cool!
Feb 17 2018
next sibling parent reply bachmeier <no spam.net> writes:
On Saturday, 17 February 2018 at 18:03:44 UTC, Peter Campbell 
wrote:

 My understanding from the vision documents and what Andrei 
 mentioned at his DConf presentations is that the runtime itself 
 will be modified to not rely on the GC, allowing for you to 
 continue using features such as associative arrays and array 
 concatenation, but without requiring the garbage collector. Is 
 my understanding correct? If so is work still being done to 
 make this happen
Andrei recently said something about an announcement coming, but I don't have any details. https://www.reddit.com/r/programming/comments/7vw0gj/vanquish_forever_these_bugs_that_blasted_your/dtwb6fv/
Feb 17 2018
parent Peter Campbell <peter spcampbell.co.uk> writes:
On Saturday, 17 February 2018 at 19:32:50 UTC, bachmeier wrote:
 On Saturday, 17 February 2018 at 18:03:44 UTC, Peter Campbell 
 wrote:

 My understanding from the vision documents and what Andrei 
 mentioned at his DConf presentations is that the runtime 
 itself will be modified to not rely on the GC, allowing for 
 you to continue using features such as associative arrays and 
 array concatenation, but without requiring the garbage 
 collector. Is my understanding correct? If so is work still 
 being done to make this happen
Andrei recently said something about an announcement coming, but I don't have any details. https://www.reddit.com/r/programming/comments/7vw0gj/vanquish_forever_these_bugs_that_blasted_your/dtwb6fv/
What a tease, but that sounds promising :)
Feb 17 2018
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 17 February 2018 at 18:03:44 UTC, Peter Campbell 
wrote:
 Andrei mentioned at his DConf presentations is that the runtime 
 itself will be modified to not rely on the GC, allowing for you 
 to continue using features such as associative arrays and array 
 concatenation, but without requiring the garbage collector. Is 
 my understanding correct?
I haven't heard anything about that, and I doubt it would happen. How would you actually manage the memory? The front-facing api there doesn't offer any access to it. But, of course, it is quite easy to write your own types that do the same stuff (including operator overloading, of course) that do expose memory management, and you can use those with or without the runtime.
Feb 17 2018
parent Peter Campbell <peter spcampbell.co.uk> writes:
On Saturday, 17 February 2018 at 23:43:07 UTC, Adam D. Ruppe 
wrote:
 On Saturday, 17 February 2018 at 18:03:44 UTC, Peter Campbell 
 wrote:
 Andrei mentioned at his DConf presentations is that the 
 runtime itself will be modified to not rely on the GC, 
 allowing for you to continue using features such as 
 associative arrays and array concatenation, but without 
 requiring the garbage collector. Is my understanding correct?
I haven't heard anything about that, and I doubt it would happen. How would you actually manage the memory? The front-facing api there doesn't offer any access to it. But, of course, it is quite easy to write your own types that do the same stuff (including operator overloading, of course) that do expose memory management, and you can use those with or without the runtime.
The first time I heard about this was the DConf 2016 keynote: https://www.youtube.com/watch?v=4oDK91E3VKs&feature=youtu.be&t=2023 Andrei discusses a reference counting based approach however there are requirements which make it difficult to implement in the runtime. It's also mentioned in both the 2017 vision documents on the wiki.
Feb 17 2018
prev sibling next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 02/17/2018 07:03 PM, Peter Campbell wrote:
 My understanding from the vision documents and what Andrei
 mentioned at his DConf presentations is that the runtime itself
 will be modified to not rely on the GC, allowing for you to
 continue using features such as associative arrays and array
 concatenation, but without requiring the garbage collector. Is my
 understanding correct? If so is work still being done to make this
 happen and is there an easy way to follow the progress?
nogc containers will most likely come in the form of a library. We plan to move built-in hashes to a templated druntime implementation (https://github.com/dlang/druntime/pull/1282), and at best make array and hash literal syntax available for user defined types (w/ an approach similar to C++'s initializer_list). Our current std.container package does not use the GC (https://dlang.org/phobos/std_container.html), but isn't really up to par. Your best options atm. would be [emsi_containers](http://code.dlang.org/packages/emsi_containers) for containers and [automem](http://code.dlang.org/packages/automem) for smart pointers. Both libraries support custom allocators (https://dlang.org/phobos/std_experimental_allocator.html). Writing containers and smart pointers isn't rocket science, so it's the least of our worries. While heading for nogc we've decided that we don't want to give up safe-ty (remember that GC based allocations don't suffer from memory corruptions). Walter has spent a lot of effort on preventing escaping of aliases (DIP1000). I'm currently working on a proposal (DIP) and implementation called "limited lifetimes" that is targeted at preventing use-after-free bugs at compile time without impeding usability too much. We hopefully will have at least a prototype implementation for this year's DConf in May. This should become the foundation for ` safe nogc` RC/Uniq/Weak primitives, onto which we can layer ` safe nogc` libraries. In parallel Andrei and his students are investigating alternative approaches to reduce GC overhead or prevent use-after-free errors at runtime. We've somewhat stopped to work on our GC, to focus on unique/ref-counted memory management. Though soon the GC implementation can be replaced with custom GCs from our package manager dub (https://github.com/dlang/druntime/pull/1924). This should open the door for low-latency GCs, based on CoW and dirty page flags. We don't want to add write-barriers to the language due to the ~5% overall slowdown, so generational GCs are out of reach, but there is an interesting paper on using type information to perform partial collections ([Connectivity-Based Garbage Collection](https://www.cs.purdue.edu/homes/hosking/690M/cbgc.pdf)). At the moment there is no good way to track progress, as a lot is being worked on behind the scenes by many distributed actors. I think we are the ones most annoyed that this transition takes so long, but our plans are also very ambitious, targeting minimal memory footprint, optimal performance while supporting a safe language subset and advanced features like custom allocators or region based allocations. - -Martin -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEpzRNrTw0HqEtE8TmsnOBFhK7GTkFAlqJ7CsACgkQsnOBFhK7 GTnTyQ//axaUtY8B1VhjW3Y/cMs4GESDmwa68r3IU/3XnyUelZfbMhAnA14aQh23 7y9D9MpPLYbgRQDbsrbUU7BoXMIiguFDQmR/265I7ZlhugVbG8VerQv+cIfXjiz9 fNKNgZHjp6hBPDIcHruiOcFNmqg73ymSg/9VWmEIzS1HUSMtZpBslz9uPWwKCacV axv8a/oPOBo5C8WcHqwtXHPeNZYKiwAcBaE6cXordN9r0dK1OC/JtvbjLRQnbv72 d0Xifdj4IkLM4krJvEZVX7jEPOvq15P48ns/gxMiem8e2qn6pHW+FIPHRx/y9Wv0 K5RGsivkBETDX6xrohwKdSEf5vIx4qQavI1pAA/8NIVsI2AlBs0OYivUnd+pTHA6 mLLI/Ask8NmWoA4FNFdWI0+zC/zTs1e06sTl1nsMs2NHCCaRL3CSTJmExeLVDrYD iz/5XvKMsZUh1wSlFvoqFX4UvmjZH3HseZfh1WhKaYnSculc0RzhDIYzbftCKqJR 1nCOnNyhS8kuxGT881oYZmbTyevoy2uU3tBStmXcGcRTfpHkd9xaFY7vlqF0Aa41 eUI0z6r4SG+ZmDmwHlnZD58kcIvBVvWZaAoo4hz0138LW0LNd9tRiBPTyUfE9PJL qCmEbVvB25nz2yge9iEOGERJYRI8P1fssfUX3bjNcDoeLTMokPo= =wnx7 -----END PGP SIGNATURE-----
Feb 18 2018
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Sunday, 18 February 2018 at 21:12:11 UTC, Martin Nowak wrote:
 [snip]
Interesting reading. Thanks for the update.
Feb 18 2018
parent reply Peter Campbell <peter spcampbell.co.uk> writes:
On Sunday, 18 February 2018 at 21:58:57 UTC, jmh530 wrote:
 On Sunday, 18 February 2018 at 21:12:11 UTC, Martin Nowak wrote:
 [snip]
Interesting reading. Thanks for the update.
Indeed, very interesting read and exactly the information I was looking for! Thanks a lot Martin, I'm excited to see this progress. It's good to know it's still being worked on and progress is being made.
Feb 18 2018
parent Martin Nowak <code dawg.eu> writes:
On Sunday, 18 February 2018 at 22:28:48 UTC, Peter Campbell wrote:
 Indeed, very interesting read and exactly the information I was 
 looking for! Thanks a lot Martin, I'm excited to see this 
 progress. It's good to know it's still being worked on and 
 progress is being made.
Yes, it's just crazy how much work and expertise is involved with a programming language ecosystem, so things always go much slower than intended. In case someone wants to help, I currently have to divert a lot of time into https://github.com/dlang/dub. Package managers are suprisingly complex pieces of software. We have some basic architecture https://github.com/dlang/dub/blob/master/ARCHITECTURE.md and would benefit a lot from a few more contributors getting familiar with the implementation.
Feb 19 2018
prev sibling parent Soren <x x.de> writes:
Thank you for this status report, much appreciated, and exactly 
the kind of thing I hope for.  Don’t be discouraged by the time 
it has taken, it will be worth it.  Good work!
Feb 19 2018
prev sibling parent reply IGotD- <nise nise.com> writes:
On Saturday, 17 February 2018 at 18:03:44 UTC, Peter Campbell 
wrote:
 I've read about BetterC and I'm interested in the minimal 
 runtime feature, that seems like it'll be really cool. But 
 these methods requiring not using certain D features which rely 
 on the GC. My understanding from the vision documents and what 
 Andrei mentioned at his DConf presentations is that the runtime 
 itself will be modified to not rely on the GC, allowing for you 
 to continue using features such as associative arrays and array 
 concatenation, but without requiring the garbage collector. Is 
 my understanding correct? If so is work still being done to 
 make this happen and is there an easy way to follow the 
 progress?
Any more progress regard this? Dynamic arrays and associative arrays are very useful even in a betterC environment. Wouldn't it be better to rewrite the dynamic arrays to use an internal storage struct using atomic reference counting similar to C++ strings (at least in some previous implementations). This way arrays are only dependent malloc/free. Using the reference counting, slicing would be supported just as before if I'm not mistaken. I also don't think this should be special variant of druntime. Reference counted dynamic arrays could go into the full D language as well. This would probably even reduce the memory consumption compared to the current GC implementation.
Oct 06
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 6 October 2019 at 12:50:04 UTC, IGotD- wrote:
 Dynamic arrays and associative arrays are very useful even in a 
 betterC environment. Wouldn't it be better to rewrite the 
 dynamic arrays to use an internal storage struct using atomic 
 reference counting similar to C++ strings
I think that'd be a mistake. D's slices are versatile and lightweight right now, they can be used with gc or malloc or anything else and this change would jeopardize that. It is easy to make a library array type that works like you described, and it can even yield built-in slices when you want that. If anything I'd just let you disable built in dynamic array append/concat so it calls out when you need to put in a wrapper...
Oct 06
prev sibling next sibling parent Eugene Wissner <belka caraus.de> writes:
On Saturday, 17 February 2018 at 12:18:28 UTC, Peter Campbell 
wrote:
 Hi everyone, it's been almost a year since I used D but I was 
 wanted to get back into it. I was checking the 2017 vision 
 pages and was wondering why there hasn't been a 2018H1 vision 
 page but also what the current status of being able to use D 
 without a garbage collector is?

 It was noted in the H1 and H2 pages for 2017 but looking 
 through the compiler changelog doesn't indicate that we're 
 there yet? https://wiki.dlang.org/Vision/2017H2

 Thanks.
Let me do a bit advertising :) I'm writing a library for nogc D programming, tanya. It has different aims than -betterC: I use the full set of D features, just except features that require GC. It isn't complete, I'm using it only in personal projects and it is also a big learn project for me, so don't wonder that it reimplements a lot of features "from scratch" and almost doesn't depend on phobos. But if you like to play with it, you're welcome: https://github.com/caraus-ecms/tanya
Feb 17 2018
prev sibling parent Seb <seb wilzba.ch> writes:
On Saturday, 17 February 2018 at 12:18:28 UTC, Peter Campbell 
wrote:
 Hi everyone, it's been almost a year since I used D but I was 
 wanted to get back into it. I was checking the 2017 vision 
 pages and was wondering why there hasn't been a 2018H1 vision 
 page but also what the current status of being able to use D 
 without a garbage collector is?

 It was noted in the H1 and H2 pages for 2017 but looking 
 through the compiler changelog doesn't indicate that we're 
 there yet? https://wiki.dlang.org/Vision/2017H2

 Thanks.
The garbage collector in druntime is now lazily initialized on the first use: https://github.com/dlang/druntime/pull/2057 (in master since a few days)
Feb 17 2018