www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Status of nogc with the runtime

reply Peter Campbell <peter spcampbell.co.uk> writes:
Hi everyone, it's been almost a year since I used D but I was 
wanted to get back into it. I was checking the 2017 vision pages 
and was wondering why there hasn't been a 2018H1 vision page but 
also what the current status of being able to use D without a 
garbage collector is?

It was noted in the H1 and H2 pages for 2017 but looking through 
the compiler changelog doesn't indicate that we're there yet? 
https://wiki.dlang.org/Vision/2017H2

Thanks.
Feb 17 2018
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Saturday, 17 February 2018 at 12:18:28 UTC, Peter Campbell 
wrote:

 I was checking the 2017 vision pages and was wondering why 
 there hasn't been a 2018H1 vision page but also what the 
 current status of being able to use D without a garbage 
 collector is?


There are a number of ways to use D without the garbage 
collector.  However, one of the easiest is with the -betterC 
flag.  You'll want to read the two blog articles here 
(https://dlang.org/blog/category/betterc/) for more information 
about that.

2.079, the upcoming release will also have a way to use D without 
the runtime and without -betterC.  What this allows you to do is 
partially implement just the runtime features you wish to 
support.  You can read more about it in the nightly changelog 
here (https://dlang.org/changelog/pending.html#minimal_runtime)

You might also be interested in this article: 
https://wiki.dlang.org/Memory_Management

Mike
Feb 17 2018
parent reply Peter Campbell <peter spcampbell.co.uk> writes:
On Saturday, 17 February 2018 at 12:32:04 UTC, Mike Franklin 
wrote:

 There are a number of ways to use D without the garbage 
 collector.  However, one of the easiest is with the -betterC 
 flag.  You'll want to read the two blog articles here 
 (https://dlang.org/blog/category/betterc/) for more information 
 about that.

 2.079, the upcoming release will also have a way to use D 
 without the runtime and without -betterC.  What this allows you 
 to do is partially implement just the runtime features you wish 
 to support.  You can read more about it in the nightly 
 changelog here 
 (https://dlang.org/changelog/pending.html#minimal_runtime)

 You might also be interested in this article: 
 https://wiki.dlang.org/Memory_Management

 Mike


I've read about BetterC and I'm interested in the minimal runtime 
feature, that seems like it'll be really cool. But these methods 
requiring not using certain D features which rely on the GC. My 
understanding from the vision documents and what Andrei mentioned 
at his DConf presentations is that the runtime itself will be 
modified to not rely on the GC, allowing for you to continue 
using features such as associative arrays and array 
concatenation, but without requiring the garbage collector. Is my 
understanding correct? If so is work still being done to make 
this happen and is there an easy way to follow the progress?

On Saturday, 17 February 2018 at 12:44:35 UTC, Eugene Wissner 
wrote:

 Let me do a bit advertising :) I'm writing a library for  nogc 
 D programming, tanya. It has different aims than -betterC: I 
 use the full set of D features, just except features that 
 require GC. It isn't complete, I'm using it only in personal 
 projects and it is also a big learn project for me, so don't 
 wonder that it reimplements a lot of features "from scratch" 
 and almost doesn't depend on phobos. But if you like to play 
 with it, you're welcome:
 https://github.com/caraus-ecms/tanya


That's an interesting project, I'll be sure to take a look!

On Saturday, 17 February 2018 at 14:25:41 UTC, Seb wrote:

 On Saturday, 17 February 2018 at 12:18:28 UTC, Peter Campbell 
 wrote:
 The garbage collector in druntime is now lazily initialized on 
 the first use:

 https://github.com/dlang/druntime/pull/2057

 (in master since a few days)


That's really nice to hear 2.079 is looking really cool!
Feb 17 2018
next sibling parent reply bachmeier <no spam.net> writes:
On Saturday, 17 February 2018 at 18:03:44 UTC, Peter Campbell 
wrote:


 My understanding from the vision documents and what Andrei 
 mentioned at his DConf presentations is that the runtime itself 
 will be modified to not rely on the GC, allowing for you to 
 continue using features such as associative arrays and array 
 concatenation, but without requiring the garbage collector. Is 
 my understanding correct? If so is work still being done to 
 make this happen


Andrei recently said something about an announcement coming, but 
I don't have any details. 
https://www.reddit.com/r/programming/comments/7vw0gj/vanquish_forever_these_bugs_that_blasted_your/dtwb6fv/
Feb 17 2018
parent Peter Campbell <peter spcampbell.co.uk> writes:
On Saturday, 17 February 2018 at 19:32:50 UTC, bachmeier wrote:

 On Saturday, 17 February 2018 at 18:03:44 UTC, Peter Campbell 
 wrote:


 My understanding from the vision documents and what Andrei 
 mentioned at his DConf presentations is that the runtime 
 itself will be modified to not rely on the GC, allowing for 
 you to continue using features such as associative arrays and 
 array concatenation, but without requiring the garbage 
 collector. Is my understanding correct? If so is work still 
 being done to make this happen


 Andrei recently said something about an announcement coming, 
 but I don't have any details. 
 https://www.reddit.com/r/programming/comments/7vw0gj/vanquish_forever_these_bugs_that_blasted_your/dtwb6fv/


What a tease, but that sounds promising :)
Feb 17 2018
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 17 February 2018 at 18:03:44 UTC, Peter Campbell 
wrote:

 Andrei mentioned at his DConf presentations is that the runtime 
 itself will be modified to not rely on the GC, allowing for you 
 to continue using features such as associative arrays and array 
 concatenation, but without requiring the garbage collector. Is 
 my understanding correct?


I haven't heard anything about that, and I doubt it would happen. 
How would you actually manage the memory? The front-facing api 
there doesn't offer any access to it.

But, of course, it is quite easy to write your own types that do 
the same stuff (including operator overloading, of course) that 
do expose memory management, and you can use those with or 
without the runtime.
Feb 17 2018
parent Peter Campbell <peter spcampbell.co.uk> writes:
On Saturday, 17 February 2018 at 23:43:07 UTC, Adam D. Ruppe 
wrote:

 On Saturday, 17 February 2018 at 18:03:44 UTC, Peter Campbell 
 wrote:

 Andrei mentioned at his DConf presentations is that the 
 runtime itself will be modified to not rely on the GC, 
 allowing for you to continue using features such as 
 associative arrays and array concatenation, but without 
 requiring the garbage collector. Is my understanding correct?


 I haven't heard anything about that, and I doubt it would 
 happen. How would you actually manage the memory? The 
 front-facing api there doesn't offer any access to it.

 But, of course, it is quite easy to write your own types that 
 do the same stuff (including operator overloading, of course) 
 that do expose memory management, and you can use those with or 
 without the runtime.


The first time I heard about this was the DConf 2016 keynote: 
https://www.youtube.com/watch?v=4oDK91E3VKs&feature=youtu.be&t=2023

Andrei discusses a reference counting based approach however 
there are requirements which make it difficult to implement in 
the runtime. It's also mentioned in both the 2017 vision 
documents on the wiki.
Feb 17 2018
prev sibling next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 02/17/2018 07:03 PM, Peter Campbell wrote:

 My understanding from the vision documents and what Andrei
 mentioned at his DConf presentations is that the runtime itself
 will be modified to not rely on the GC, allowing for you to
 continue using features such as associative arrays and array
 concatenation, but without requiring the garbage collector. Is my
 understanding correct? If so is work still being done to make this
 happen and is there an easy way to follow the progress?


 nogc containers will most likely come in the form of a library.
We plan to move built-in hashes to a templated druntime implementation
(https://github.com/dlang/druntime/pull/1282), and at best make array
and hash literal syntax available for user defined types (w/ an
approach similar to C++'s initializer_list).
Our current std.container package does not use the GC
(https://dlang.org/phobos/std_container.html), but isn't really up to
par. Your best options atm. would be
[emsi_containers](http://code.dlang.org/packages/emsi_containers) for
containers and [automem](http://code.dlang.org/packages/automem) for
smart pointers.
Both libraries support custom allocators
(https://dlang.org/phobos/std_experimental_allocator.html).
Writing containers and smart pointers isn't rocket science, so it's
the least of our worries.

While heading for  nogc we've decided that we don't want to give up
 safe-ty (remember that GC based allocations don't suffer from memory
corruptions).
Walter has spent a lot of effort on preventing escaping of aliases
(DIP1000).
I'm currently working on a proposal (DIP) and implementation called
"limited lifetimes" that is targeted at preventing use-after-free bugs
at compile time without impeding usability too much.
We hopefully will have at least a prototype implementation for this
year's DConf in May.
This should become the foundation for ` safe  nogc` RC/Uniq/Weak
primitives, onto which we can layer ` safe  nogc` libraries.

In parallel Andrei and his students are investigating alternative
approaches to reduce GC overhead or prevent use-after-free errors at
runtime.

We've somewhat stopped to work on our GC, to focus on
unique/ref-counted memory management. Though soon the GC
implementation can be replaced with custom GCs from our package
manager dub (https://github.com/dlang/druntime/pull/1924). This should
open the door for low-latency GCs, based on CoW and dirty page flags.
We don't want to add write-barriers to the language due to the ~5%
overall slowdown, so generational GCs are out of reach, but there is
an interesting paper on using type information to perform partial
collections ([Connectivity-Based Garbage
Collection](https://www.cs.purdue.edu/homes/hosking/690M/cbgc.pdf)).

At the moment there is no good way to track progress, as a lot is
being worked on behind the scenes by many distributed actors.
I think we are the ones most annoyed that this transition takes so
long, but our plans are also very ambitious, targeting minimal memory
footprint, optimal performance while supporting a  safe language
subset and advanced features like custom allocators or region based
allocations.

- -Martin
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEpzRNrTw0HqEtE8TmsnOBFhK7GTkFAlqJ7CsACgkQsnOBFhK7
GTnTyQ//axaUtY8B1VhjW3Y/cMs4GESDmwa68r3IU/3XnyUelZfbMhAnA14aQh23
7y9D9MpPLYbgRQDbsrbUU7BoXMIiguFDQmR/265I7ZlhugVbG8VerQv+cIfXjiz9
fNKNgZHjp6hBPDIcHruiOcFNmqg73ymSg/9VWmEIzS1HUSMtZpBslz9uPWwKCacV
axv8a/oPOBo5C8WcHqwtXHPeNZYKiwAcBaE6cXordN9r0dK1OC/JtvbjLRQnbv72
d0Xifdj4IkLM4krJvEZVX7jEPOvq15P48ns/gxMiem8e2qn6pHW+FIPHRx/y9Wv0
K5RGsivkBETDX6xrohwKdSEf5vIx4qQavI1pAA/8NIVsI2AlBs0OYivUnd+pTHA6
mLLI/Ask8NmWoA4FNFdWI0+zC/zTs1e06sTl1nsMs2NHCCaRL3CSTJmExeLVDrYD
iz/5XvKMsZUh1wSlFvoqFX4UvmjZH3HseZfh1WhKaYnSculc0RzhDIYzbftCKqJR
1nCOnNyhS8kuxGT881oYZmbTyevoy2uU3tBStmXcGcRTfpHkd9xaFY7vlqF0Aa41
eUI0z6r4SG+ZmDmwHlnZD58kcIvBVvWZaAoo4hz0138LW0LNd9tRiBPTyUfE9PJL
qCmEbVvB25nz2yge9iEOGERJYRI8P1fssfUX3bjNcDoeLTMokPo=
=wnx7
-----END PGP SIGNATURE-----
Feb 18 2018
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Sunday, 18 February 2018 at 21:12:11 UTC, Martin Nowak wrote:

 [snip]


Interesting reading. Thanks for the update.
Feb 18 2018
parent reply Peter Campbell <peter spcampbell.co.uk> writes:
On Sunday, 18 February 2018 at 21:58:57 UTC, jmh530 wrote:

 On Sunday, 18 February 2018 at 21:12:11 UTC, Martin Nowak wrote:

 [snip]


 Interesting reading. Thanks for the update.


Indeed, very interesting read and exactly the information I was 
looking for! Thanks a lot Martin, I'm excited to see this 
progress. It's good to know it's still being worked on and 
progress is being made.
Feb 18 2018
parent Martin Nowak <code dawg.eu> writes:
On Sunday, 18 February 2018 at 22:28:48 UTC, Peter Campbell wrote:

 Indeed, very interesting read and exactly the information I was 
 looking for! Thanks a lot Martin, I'm excited to see this 
 progress. It's good to know it's still being worked on and 
 progress is being made.


Yes, it's just crazy how much work and expertise is involved with 
a programming language ecosystem, so things always go much slower 
than intended.

In case someone wants to help, I currently have to divert a lot 
of time into https://github.com/dlang/dub. Package managers are 
suprisingly complex pieces of software. We have some basic 
architecture 
https://github.com/dlang/dub/blob/master/ARCHITECTURE.md and 
would benefit a lot from a few more contributors getting familiar 
with the implementation.
Feb 19 2018
prev sibling parent Soren <x x.de> writes:
Thank you for this status report, much appreciated, and exactly 
the kind of thing I hope for.  Don’t be discouraged by the time 
it has taken, it will be worth it.  Good work!
Feb 19 2018
prev sibling parent reply IGotD- <nise nise.com> writes:
On Saturday, 17 February 2018 at 18:03:44 UTC, Peter Campbell 
wrote:

 I've read about BetterC and I'm interested in the minimal 
 runtime feature, that seems like it'll be really cool. But 
 these methods requiring not using certain D features which rely 
 on the GC. My understanding from the vision documents and what 
 Andrei mentioned at his DConf presentations is that the runtime 
 itself will be modified to not rely on the GC, allowing for you 
 to continue using features such as associative arrays and array 
 concatenation, but without requiring the garbage collector. Is 
 my understanding correct? If so is work still being done to 
 make this happen and is there an easy way to follow the 
 progress?


Any more progress regard this? Dynamic arrays and associative 
arrays are very useful even in a betterC environment. Wouldn't it 
be better to rewrite the dynamic arrays to use an internal 
storage struct using atomic reference counting similar to C++ 
strings (at least in some previous implementations). This way 
arrays are only dependent malloc/free. Using the reference 
counting, slicing would be supported just as before if I'm not 
mistaken.

I also don't think this should be special variant of druntime. 
Reference counted dynamic arrays could go into the full D 
language as well. This would probably even reduce the memory 
consumption compared to the current GC implementation.
Oct 06 2019
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 6 October 2019 at 12:50:04 UTC, IGotD- wrote:

 Dynamic arrays and associative arrays are very useful even in a 
 betterC environment. Wouldn't it be better to rewrite the 
 dynamic arrays to use an internal storage struct using atomic 
 reference counting similar to C++ strings


I think that'd be a mistake. D's slices are versatile and 
lightweight right now, they can be used with gc or malloc or 
anything else and this change would jeopardize that.

It is easy to make a library array type that works like you 
described, and it can even yield built-in slices when you want 
that.

If anything I'd just let you disable built in dynamic array 
append/concat so it calls out when you need to put in a wrapper...
Oct 06 2019
prev sibling next sibling parent Eugene Wissner <belka caraus.de> writes:
On Saturday, 17 February 2018 at 12:18:28 UTC, Peter Campbell 
wrote:

 Hi everyone, it's been almost a year since I used D but I was 
 wanted to get back into it. I was checking the 2017 vision 
 pages and was wondering why there hasn't been a 2018H1 vision 
 page but also what the current status of being able to use D 
 without a garbage collector is?

 It was noted in the H1 and H2 pages for 2017 but looking 
 through the compiler changelog doesn't indicate that we're 
 there yet? https://wiki.dlang.org/Vision/2017H2

 Thanks.


Let me do a bit advertising :) I'm writing a library for  nogc D 
programming, tanya. It has different aims than -betterC: I use 
the full set of D features, just except features that require GC. 
It isn't complete, I'm using it only in personal projects and it 
is also a big learn project for me, so don't wonder that it 
reimplements a lot of features "from scratch" and almost doesn't 
depend on phobos. But if you like to play with it, you're welcome:
https://github.com/caraus-ecms/tanya
Feb 17 2018
prev sibling parent Seb <seb wilzba.ch> writes:
On Saturday, 17 February 2018 at 12:18:28 UTC, Peter Campbell 
wrote:

 Hi everyone, it's been almost a year since I used D but I was 
 wanted to get back into it. I was checking the 2017 vision 
 pages and was wondering why there hasn't been a 2018H1 vision 
 page but also what the current status of being able to use D 
 without a garbage collector is?

 It was noted in the H1 and H2 pages for 2017 but looking 
 through the compiler changelog doesn't indicate that we're 
 there yet? https://wiki.dlang.org/Vision/2017H2

 Thanks.


The garbage collector in druntime is now lazily initialized on 
the first use:

https://github.com/dlang/druntime/pull/2057

(in master since a few days)
Feb 17 2018