www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - is d-runtime non-gc safe?

reply Benjamin Thaut <code benjamin-thaut.de> writes:
I know phobos is nto usable without a gc but is druntime usable without 
having a GC?

-- 
Kind Regards
Benjamin Thaut
Dec 02 2011
parent reply Sean Kelly <sean invisibleduck.org> writes:
On Dec 2, 2011, at 8:57 AM, Benjamin Thaut wrote:


 I know phobos is nto usable without a gc but is druntime usable =


without having a GC?

You'll leak when threads terminate (they're allocated by the GC and not =
safe for the user to delete), but I think that's it for user-visible =
code.  Some array ops and AA use will leak as well, but there's no way =
around that.  For kicks you might want to try linking gcstub (which =
calls malloc/free) and see how it goes.=
Dec 02 2011
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 02.12.2011 20:29, schrieb Sean Kelly:

 On Dec 2, 2011, at 8:57 AM, Benjamin Thaut wrote:


 I know phobos is nto usable without a gc but is druntime usable without having
a GC?


 You'll leak when threads terminate (they're allocated by the GC and not safe
for the user to delete), but I think that's it for user-visible code.  Some
array ops and AA use will leak as well, but there's no way around that.  For
kicks you might want to try linking gcstub (which calls malloc/free) and see
how it goes.


Thanks, That doesn't sound to bad. If I manage to get a non leaking non 
gc version of d-runtime working would there be any interest in that?

Kind Regards
Benjamin Thaut
Dec 03 2011
next sibling parent maarten van damme <maartenvd1994 gmail.com> writes:
I could really use one. Not for real use but to test a bit with.
So if you really make one, host it at github or something like that.
Dec 03 2011
prev sibling next sibling parent Dejan Lekic <dejan.lekic gmail.com> writes:
 
 Thanks, That doesn't sound to bad. If I manage to get a non leaking non
 gc version of d-runtime working would there be any interest in that?
 


Absolutely, judging by the number of related discussions on IRC there will 
be many developers who would be interested in such project.
Dec 03 2011
prev sibling parent reply "Marco Leise" <Marco.Leise gmx.de> writes:
Am 03.12.2011, 10:01 Uhr, schrieb Benjamin Thaut <code benjamin-thaut.de>:


 Thanks, That doesn't sound to bad. If I manage to get a non leaking non  
 gc version of d-runtime working would there be any interest in that?

 Kind Regards
 Benjamin Thaut


That would be most interesting! How do you imagine this could work though?  
As an example D array slices work on GC memory and you can have dozens of  
slices on the same memory block. Would these memory blocks be reference  
counted then?
Dec 03 2011
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
+1 on interest on having this. Back when I was attempting to port VST
to D I got asked by a Steinberg dev how I can guarantee that D plugins
will work. But I couldn't guarantee it, if a GC collection were to run
the plugin would freeze, the host would crash, and the host company
would likely get the blame. A GC-less environment would be great for
realtime plugins, I'm not sure how arrays/slices would work, but I
guess custom Array refcounted structs could be made or something.
Dec 03 2011
next sibling parent reply "Marco Leise" <Marco.Leise gmx.de> writes:
Am 03.12.2011, 19:05 Uhr, schrieb Andrej Mitrovic  
<andrej.mitrovich gmail.com>:


 +1 on interest on having this. Back when I was attempting to port VST
 to D I got asked by a Steinberg dev how I can guarantee that D plugins
 will work. But I couldn't guarantee it, if a GC collection were to run
 the plugin would freeze, the host would crash, and the host company
 would likely get the blame.


You'll get some angry comments on this one by others. I'll lean back and  
enjoy the show :D
Dec 03 2011
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/3/11, Marco Leise <Marco.Leise gmx.de> wrote:

 Am 03.12.2011, 19:05 Uhr, schrieb Andrej Mitrovic
 <andrej.mitrovich gmail.com>:


 +1 on interest on having this. Back when I was attempting to port VST
 to D I got asked by a Steinberg dev how I can guarantee that D plugins
 will work. But I couldn't guarantee it, if a GC collection were to run
 the plugin would freeze, the host would crash, and the host company
 would likely get the blame.


 You'll get some angry comments on this one by others. I'll lean back and
 enjoy the show :D


I don't follow..?
Dec 03 2011
parent reply "Marco Leise" <Marco.Leise gmx.de> writes:
Am 03.12.2011, 19:47 Uhr, schrieb Andrej Mitrovic  
<andrej.mitrovich gmail.com>:


 On 12/3/11, Marco Leise <Marco.Leise gmx.de> wrote:

 Am 03.12.2011, 19:05 Uhr, schrieb Andrej Mitrovic
 <andrej.mitrovich gmail.com>:


 +1 on interest on having this. Back when I was attempting to port VST
 to D I got asked by a Steinberg dev how I can guarantee that D plugins
 will work. But I couldn't guarantee it, if a GC collection were to run
 the plugin would freeze, the host would crash, and the host company
 would likely get the blame.


 You'll get some angry comments on this one by others. I'll lean back and
 enjoy the show :D


 I don't follow..?


Hmm, and no one commented either. I thought someone would jump in and say  
that you could disable the GC and that malloc isn't realtime either and  
for that reason the host application should not crash when a plugin  
doesn't deliver in time. The audio should stutter and that's it. So: no  
realtime guarantees whatsoever on consumer OSs, but a high probability  
that it will just work.
Dec 04 2011
next sibling parent reply Norbert Nemec <Norbert Nemec-online.de> writes:
On 04.12.2011 09:10, Marco Leise wrote:

 Am 03.12.2011, 19:47 Uhr, schrieb Andrej Mitrovic
 <andrej.mitrovich gmail.com>:


 On 12/3/11, Marco Leise <Marco.Leise gmx.de> wrote:

 Am 03.12.2011, 19:05 Uhr, schrieb Andrej Mitrovic
 <andrej.mitrovich gmail.com>:


 +1 on interest on having this. Back when I was attempting to port VST
 to D I got asked by a Steinberg dev how I can guarantee that D plugins
 will work. But I couldn't guarantee it, if a GC collection were to run
 the plugin would freeze, the host would crash, and the host company
 would likely get the blame.


 You'll get some angry comments on this one by others. I'll lean back and
 enjoy the show :D


 I don't follow..?


 Hmm, and no one commented either. I thought someone would jump in and
 say that you could disable the GC and that malloc isn't realtime either
 and for that reason the host application should not crash when a plugin
 doesn't deliver in time. The audio should stutter and that's it. So: no
 realtime guarantees whatsoever on consumer OSs, but a high probability
 that it will just work.


Indeed, malloc is not real-time safe. It is common wisdom that real-time 
audio code should not handle any memory allocations.

To this point, D is just as safe as C++: don't do any operations that 
may allocate memory in the audio thread, so the GC will never be invoked 
in real-time critical code.

Where it gets more involved, though, is that D garbage collection is not 
thread safe. When it starts, all registered threads are stopped.

Note: the audio-thread is typically created externally with special 
settings, so D will not know about it and the GC will not stop it. As 
long as the audio-thread never does any pointer assignments that are 
relevant to the GC, everything should be safe.

Just make sure that any garbage collected objects that the audio thread 
accesses have live pointer from areas that the GC knows about. That way 
it will not matter whether the GC scans the audio-data or not or whether 
this data is modified by the audio-thread while the GC is scanning it.
Dec 04 2011
next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
You can make the GC know about any thread you want. Look at thread_attachThi=
s and related routines in core.thread.=20

Sent from my iPhone

On Dec 4, 2011, at 5:08 AM, Norbert Nemec <Norbert Nemec-online.de> wrote:


 On 04.12.2011 09:10, Marco Leise wrote:

 Am 03.12.2011, 19:47 Uhr, schrieb Andrej Mitrovic
 <andrej.mitrovich gmail.com>:
=20

 On 12/3/11, Marco Leise <Marco.Leise gmx.de> wrote:

 Am 03.12.2011, 19:05 Uhr, schrieb Andrej Mitrovic
 <andrej.mitrovich gmail.com>:
=20

 +1 on interest on having this. Back when I was attempting to port VST
 to D I got asked by a Steinberg dev how I can guarantee that D plugins=











 will work. But I couldn't guarantee it, if a GC collection were to run=











 the plugin would freeze, the host would crash, and the host company
 would likely get the blame.


=20
 You'll get some angry comments on this one by others. I'll lean back an=








d

 enjoy the show :D
=20


=20
 I don't follow..?


=20
 Hmm, and no one commented either. I thought someone would jump in and
 say that you could disable the GC and that malloc isn't realtime either
 and for that reason the host application should not crash when a plugin
 doesn't deliver in time. The audio should stutter and that's it. So: no
 realtime guarantees whatsoever on consumer OSs, but a high probability
 that it will just work.


=20
 Indeed, malloc is not real-time safe. It is common wisdom that real-time a=


udio code should not handle any memory allocations.

=20
 To this point, D is just as safe as C++: don't do any operations that may a=


llocate memory in the audio thread, so the GC will never be invoked in real-=
time critical code.

=20
 Where it gets more involved, though, is that D garbage collection is not t=


hread safe. When it starts, all registered threads are stopped.

=20
 Note: the audio-thread is typically created externally with special settin=


gs, so D will not know about it and the GC will not stop it. As long as the a=
udio-thread never does any pointer assignments that are relevant to the GC, e=
verything should be safe.

=20
 Just make sure that any garbage collected objects that the audio thread ac=


cesses have live pointer from areas that the GC knows about. That way it wil=
l not matter whether the GC scans the audio-data or not or whether this data=
 is modified by the audio-thread while the GC is scanning it.
Dec 04 2011
parent Norbert Nemec <Norbert Nemec-online.de> writes:
Indeed, but here the issue is actually opposite: you *don't* want the GC 
to know about the audio-thread.



On 04.12.2011 18:05, Sean Kelly wrote:

 You can make the GC know about any thread you want. Look at thread_attachThis
and related routines in core.thread.

 Sent from my iPhone

 On Dec 4, 2011, at 5:08 AM, Norbert Nemec<Norbert Nemec-online.de>  wrote:


 On 04.12.2011 09:10, Marco Leise wrote:

 Am 03.12.2011, 19:47 Uhr, schrieb Andrej Mitrovic
 <andrej.mitrovich gmail.com>:


 On 12/3/11, Marco Leise<Marco.Leise gmx.de>  wrote:

 Am 03.12.2011, 19:05 Uhr, schrieb Andrej Mitrovic
 <andrej.mitrovich gmail.com>:


 +1 on interest on having this. Back when I was attempting to port VST
 to D I got asked by a Steinberg dev how I can guarantee that D plugins
 will work. But I couldn't guarantee it, if a GC collection were to run
 the plugin would freeze, the host would crash, and the host company
 would likely get the blame.


 You'll get some angry comments on this one by others. I'll lean back and
 enjoy the show :D


 I don't follow..?


 Hmm, and no one commented either. I thought someone would jump in and
 say that you could disable the GC and that malloc isn't realtime either
 and for that reason the host application should not crash when a plugin
 doesn't deliver in time. The audio should stutter and that's it. So: no
 realtime guarantees whatsoever on consumer OSs, but a high probability
 that it will just work.


 Indeed, malloc is not real-time safe. It is common wisdom that real-time audio
code should not handle any memory allocations.

 To this point, D is just as safe as C++: don't do any operations that may
allocate memory in the audio thread, so the GC will never be invoked in
real-time critical code.

 Where it gets more involved, though, is that D garbage collection is not
thread safe. When it starts, all registered threads are stopped.

 Note: the audio-thread is typically created externally with special settings,
so D will not know about it and the GC will not stop it. As long as the
audio-thread never does any pointer assignments that are relevant to the GC,
everything should be safe.

 Just make sure that any garbage collected objects that the audio thread
accesses have live pointer from areas that the GC knows about. That way it will
not matter whether the GC scans the audio-data or not or whether this data is
modified by the audio-thread while the GC is scanning it.
Dec 04 2011
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/4/2011 5:08 AM, Norbert Nemec wrote:

 Indeed, malloc is not real-time safe. It is common wisdom that real-time audio
 code should not handle any memory allocations.

 To this point, D is just as safe as C++: don't do any operations that may
 allocate memory in the audio thread, so the GC will never be invoked in
 real-time critical code.

 Where it gets more involved, though, is that D garbage collection is not thread
 safe. When it starts, all registered threads are stopped.

 Note: the audio-thread is typically created externally with special settings,
so
 D will not know about it and the GC will not stop it. As long as the
 audio-thread never does any pointer assignments that are relevant to the GC,
 everything should be safe.

 Just make sure that any garbage collected objects that the audio thread
accesses
 have live pointer from areas that the GC knows about. That way it will not
 matter whether the GC scans the audio-data or not or whether this data is
 modified by the audio-thread while the GC is scanning it.


You're right, except for one point. If we ever do a moving GC, then the GC has 
to either know about all the pointers (so it can modify them) or the pointers 
must be 'pinned' (the pointed to object cannot be moved).

The way to 'pin' an object in D is to have a pointer to the object (a root)
that 
the GC knows about that is not within a GC allocated object; i.e. that root is 
on the stack, in static data, or is in some other memory like malloc'd data.

Java's JNI interface requires explicit pinning, but this is incompatible with 
D's desire for easy interoperation with C.
Dec 04 2011
parent reply Norbert Nemec <Norbert Nemec-online.de> writes:
On 04.12.2011 21:04, Walter Bright wrote:

 You're right, except for one point. If we ever do a moving GC, then the
 GC has to either know about all the pointers (so it can modify them) or
 the pointers must be 'pinned' (the pointed to object cannot be moved).

 The way to 'pin' an object in D is to have a pointer to the object (a
 root) that the GC knows about that is not within a GC allocated object;
 i.e. that root is on the stack, in static data, or is in some other
 memory like malloc'd data.

 Java's JNI interface requires explicit pinning, but this is incompatible
 with D's desire for easy interoperation with C.


Right - thanks for the hint!

That would leave the following rules for real-time audio code in D:

a) Never allocate memory within the audio thread (be aware of library 
code or language intrinsics that may internally allocate memory)
-> that way, the GC will never be invoked from the audio thread

b) Make sure that the audio thread is not registered with the GC
-> that way, the GC running in any thread, will not stall the GC

c) Make sure that any heap allocated memory that is used by the audio 
thread is also referenced from outside the audio-thread by pointers that 
are not located inside GC controlled objects.
-> that way memory is guaranteed not to be moved or deallocated by the 
GC, even if the audio-thread itself modifies pointers while the GC is 
running

By following these rules, the D language should be as safe for of hard 
real-time requirements as C or any other language.

Beware that all this does not really cover the initial case of this 
thread: I am talking of a setup where the GC is running in a different 
non-real-time thread. This approach allows full use of the regular 
library, as long as you keep memory setup and object creation out of the 
RT thread.
Dec 05 2011
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/5/2011 11:52 AM, Norbert Nemec wrote:

 On 04.12.2011 21:04, Walter Bright wrote:

 You're right, except for one point. If we ever do a moving GC, then the
 GC has to either know about all the pointers (so it can modify them) or
 the pointers must be 'pinned' (the pointed to object cannot be moved).

 The way to 'pin' an object in D is to have a pointer to the object (a
 root) that the GC knows about that is not within a GC allocated object;
 i.e. that root is on the stack, in static data, or is in some other
 memory like malloc'd data.

 Java's JNI interface requires explicit pinning, but this is incompatible
 with D's desire for easy interoperation with C.


 Right - thanks for the hint!

 That would leave the following rules for real-time audio code in D:

 a) Never allocate memory within the audio thread (be aware of library code or
 language intrinsics that may internally allocate memory)
 -> that way, the GC will never be invoked from the audio thread

 b) Make sure that the audio thread is not registered with the GC
 -> that way, the GC running in any thread, will not stall the GC

 c) Make sure that any heap allocated memory that is used by the audio thread is
 also referenced from outside the audio-thread by pointers that are not located
 inside GC controlled objects.
 -> that way memory is guaranteed not to be moved or deallocated by the GC, even
 if the audio-thread itself modifies pointers while the GC is running

 By following these rules, the D language should be as safe for of hard
real-time
 requirements as C or any other language.

 Beware that all this does not really cover the initial case of this thread: I
am
 talking of a setup where the GC is running in a different non-real-time thread.
 This approach allows full use of the regular library, as long as you keep
memory
 setup and object creation out of the RT thread.


These are great rules. I need to pull this all together into a how-to article.
Dec 05 2011
prev sibling parent reply Tobias Pankrath <tobias pankrath.net> writes:
 Right - thanks for the hint!
 
 That would leave the following rules for real-time audio code in D:
 
 [snip]


What's about message passing? Is message passing hard real time ready?
Dec 05 2011
next sibling parent reply Norbert Nemec <Norbert Nemec-online.de> writes:
On 05.12.2011 21:40, Tobias Pankrath wrote:

 Right - thanks for the hint!

 That would leave the following rules for real-time audio code in D:

 [snip]


 What's about message passing? Is message passing hard real time ready?


The issue actually came up for me a few weeks ago.

In principle, all you need are non-blocking queues to send messages to, 
from or even between RT threads. With the atomic operations in D, such a 
queue is fairly straightforward to implement.

Unfortunately, I could not find any OS support for non-blocking message 
passing. More specifically, there does not seem to be any way to wake a 
sleeping thread by sending a message from a RT thread. The only option 
is periodically querying the message queue. Not optimal, but probably 
sufficient for most purposes.
Dec 05 2011
parent reply Sean Cavanaugh <WorksOnMyMachine gmail.com> writes:
On 12/5/2011 3:22 PM, Norbert Nemec wrote:

 On 05.12.2011 21:40, Tobias Pankrath wrote:

 Right - thanks for the hint!

 That would leave the following rules for real-time audio code in D:

 [snip]


 What's about message passing? Is message passing hard real time ready?


 The issue actually came up for me a few weeks ago.

 In principle, all you need are non-blocking queues to send messages to,
 from or even between RT threads. With the atomic operations in D, such a
 queue is fairly straightforward to implement.

 Unfortunately, I could not find any OS support for non-blocking message
 passing. More specifically, there does not seem to be any way to wake a
 sleeping thread by sending a message from a RT thread. The only option
 is periodically querying the message queue. Not optimal, but probably
 sufficient for most purposes.


In windows land you would use MsgWaitForMultipleObject on the sleepy 
thread and wake it up with a kernel object of some kind (an Event most 
likely)
Dec 07 2011
parent reply Norbert Nemec <Norbert Nemec-online.de> writes:
On 07.12.2011 18:14, Sean Cavanaugh wrote:

 On 12/5/2011 3:22 PM, Norbert Nemec wrote:

 On 05.12.2011 21:40, Tobias Pankrath wrote:

 Right - thanks for the hint!

 That would leave the following rules for real-time audio code in D:

 [snip]


 What's about message passing? Is message passing hard real time ready?


 The issue actually came up for me a few weeks ago.

 In principle, all you need are non-blocking queues to send messages to,
 from or even between RT threads. With the atomic operations in D, such a
 queue is fairly straightforward to implement.

 Unfortunately, I could not find any OS support for non-blocking message
 passing. More specifically, there does not seem to be any way to wake a
 sleeping thread by sending a message from a RT thread. The only option
 is periodically querying the message queue. Not optimal, but probably
 sufficient for most purposes.


 In windows land you would use MsgWaitForMultipleObject on the sleepy
 thread and wake it up with a kernel object of some kind (an Event most
 likely)


How do you generate the kernel object? Is this operation non-blocking?
Dec 09 2011
parent Sean Kelly <sean invisibleduck.org> writes:
On Dec 9, 2011, at 12:04 AM, Norbert Nemec wrote:


 On 07.12.2011 18:14, Sean Cavanaugh wrote:

 
 
 In windows land you would use MsgWaitForMultipleObject on the sleepy
 thread and wake it up with a kernel object of some kind (an Event most
 likely)


 
 How do you generate the kernel object? Is this operation non-blocking?


You might want to look at core.sync.condition.
Dec 09 2011
prev sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
No. Message-passing allocates a block of memory to transport the message. Ev=
en with a free-list there may be an occasional allocation. I could look into=
 using an array instead of a linked list, but that would mean memmoves. In r=
eceive.=20

Sent from my iPhone

On Dec 5, 2011, at 12:40 PM, Tobias Pankrath <tobias pankrath.net> wrote:


 Right - thanks for the hint!
=20
 That would leave the following rules for real-time audio code in D:
=20
 [snip]


=20
 What's about message passing? Is message passing hard real time ready?
Dec 06 2011
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/6/11 8:11 PM, Sean Kelly wrote:

 No. Message-passing allocates a block of memory to transport the message. Even
with a free-list there may be an occasional allocation. I could look into using
an array instead of a linked list, but that would mean memmoves. In receive.

 Sent from my iPhone

 On Dec 5, 2011, at 12:40 PM, Tobias Pankrath<tobias pankrath.net>  wrote:


 Right - thanks for the hint!

 That would leave the following rules for real-time audio code in D:

 [snip]


 What's about message passing? Is message passing hard real time ready?




A circular fixed-size list is possible with blocking on full list. There 
would be no allocation.

Andrei
Dec 06 2011
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/4/11, Marco Leise <Marco.Leise gmx.de> wrote:

 Hmm, and no one commented either. I thought someone would jump in and say
 that you could disable the GC.


Right, I've heard this story before but has anyone actually tried
using D without the GC? I'm genuinely curious. I'd also wonder how
usable D would be without the GC. I mean, if all you're left with is
writing C-style D code then you might as well use C, right? I don't
know, I've never used D in a GC-less environment, I'm not trying to
piss anyone off. :)
Dec 04 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 12/04/2011 09:56 PM, Andrej Mitrovic wrote:

 On 12/4/11, Marco Leise<Marco.Leise gmx.de>  wrote:

 Hmm, and no one commented either. I thought someone would jump in and say
 that you could disable the GC.


 Right, I've heard this story before but has anyone actually tried
 using D without the GC? I'm genuinely curious. I'd also wonder how
 usable D would be without the GC. I mean, if all you're left with is
 writing C-style D code then you might as well use C, right? I don't
 know, I've never used D in a GC-less environment, I'm not trying to
 piss anyone off. :)


Are you suggesting that the only thing that discriminates C and D is 
that D supports GC? =)
Dec 04 2011
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
There we go, into the bee's nest. I'll just keep quiet.
Dec 04 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-12-03 19:05, Andrej Mitrovic wrote:

 +1 on interest on having this. Back when I was attempting to port VST
 to D I got asked by a Steinberg dev how I can guarantee that D plugins
 will work. But I couldn't guarantee it, if a GC collection were to run
 the plugin would freeze, the host would crash, and the host company
 would likely get the blame. A GC-less environment would be great for
 realtime plugins, I'm not sure how arrays/slices would work, but I
 guess custom Array refcounted structs could be made or something.


Seems like a very badly designed plugin or plugin system if the host 
crashes when a plugin freezes.

-- 
/Jacob Carlborg
Dec 04 2011
parent Norbert Nemec <Norbert Nemec-online.de> writes:
On 04.12.2011 12:53, Jacob Carlborg wrote:

 On 2011-12-03 19:05, Andrej Mitrovic wrote:

 +1 on interest on having this. Back when I was attempting to port VST
 to D I got asked by a Steinberg dev how I can guarantee that D plugins
 will work. But I couldn't guarantee it, if a GC collection were to run
 the plugin would freeze, the host would crash, and the host company
 would likely get the blame. A GC-less environment would be great for
 realtime plugins, I'm not sure how arrays/slices would work, but I
 guess custom Array refcounted structs could be made or something.


 Seems like a very badly designed plugin or plugin system if the host
 crashes when a plugin freezes.


Generally, audio-plugin systems are extremely sensitive. The interface 
itself is real-time performance critical and any kind of protection 
against errors in the plugins comes with a run-time penalty.

A freeze in the plugin is relatively harmless and should be caught 
somehow. Still - recovering from it is typically not possible without 
restarting the audio thread.
Dec 04 2011
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 03 Dec 2011 12:43:08 -0500, Marco Leise <Marco.Leise gmx.de> wrote:


 Am 03.12.2011, 10:01 Uhr, schrieb Benjamin Thaut  
 <code benjamin-thaut.de>:


 Thanks, That doesn't sound to bad. If I manage to get a non leaking non  
 gc version of d-runtime working would there be any interest in that?

 Kind Regards
 Benjamin Thaut


 That would be most interesting! How do you imagine this could work  
 though? As an example D array slices work on GC memory and you can have  
 dozens of slices on the same memory block. Would these memory blocks be  
 reference counted then?


D array appending operations require a GC.  You'd have to avoid them.   
This should be possible with some changes to druntime (just find all the  
cases and remove them).

But what's difficult is finding leaks which don't have any anchor to  
search for.  For example, how do you search for code that allocates an  
array and *doesn't* deallocate it?

It's definitely an interesting and worthwhile project.

-Steve
Dec 05 2011
next sibling parent reply Hans Uhlig <hans.uhlig teamaol.com> writes:
Ok, this is probably a silly question but why are the array appending 
operations dependent on the GC. Why cant you allocate from a fixed pool? 
or can you it just requires writing your own malloc/free implementation?

On 12/5/2011 5:46 AM, Steven Schveighoffer wrote:

 On Sat, 03 Dec 2011 12:43:08 -0500, Marco Leise <Marco.Leise gmx.de> wrote:


 Am 03.12.2011, 10:01 Uhr, schrieb Benjamin Thaut
 <code benjamin-thaut.de>:


 Thanks, That doesn't sound to bad. If I manage to get a non leaking
 non gc version of d-runtime working would there be any interest in that?

 Kind Regards
 Benjamin Thaut


 That would be most interesting! How do you imagine this could work
 though? As an example D array slices work on GC memory and you can
 have dozens of slices on the same memory block. Would these memory
 blocks be reference counted then?


 D array appending operations require a GC. You'd have to avoid them.
 This should be possible with some changes to druntime (just find all the
 cases and remove them).

 But what's difficult is finding leaks which don't have any anchor to
 search for. For example, how do you search for code that allocates an
 array and *doesn't* deallocate it?

 It's definitely an interesting and worthwhile project.

 -Steve
Dec 05 2011
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 05 Dec 2011 11:34:18 -0500, Hans Uhlig <hans.uhlig teamaol.com>  
wrote:


 Ok, this is probably a silly question but why are the array appending  
 operations dependent on the GC. Why cant you allocate from a fixed pool?  
 or can you it just requires writing your own malloc/free implementation?


This might help:

http://www.dsource.org/projects/dcollections/wiki/ArrayArticle

To summarize:

arr ~= 1;

What does this do?  It appends 1 to the slice.  So what happens if that  
new element can't fit in the given memory block?  Given how d slices work,  
you have no idea whether there is another slice that refers to that same  
original slice.  Not invalidating other slices is a requirement of  
appending.  So you have to allocate new memory, and leave the old memory  
in tact.

But what if nobody else is referencing that memory?  It becomes leaked.   
This is why the GC is necessary for array appending *as defined by the D  
spec* to work.  Without the GC, you must either deallocate on appending,  
making all other slices to the old data invalid, or you must leak memory.

Now, there are other ways to do array implementations.  For instance, an  
array could be a full-blown reference type, and the actual heap data  
hidden by the array.  Then you can avoid leaking, but that still doesn't  
implement all the features of D slices.  It cannot be a drop-in  
replacement.

In reality, there is no way to provide the same features as D's arrays and  
slices without a GC.  So simply put, you need to redesign your code not to  
use those features.  Hence the project of making druntime avoid all  
GC-requiring features.

-Steve
Dec 05 2011
prev sibling next sibling parent David Nadlinger <see klickverbot.at> writes:
On 12/5/11 2:46 PM, Steven Schveighoffer wrote:

 But what's difficult is finding leaks which don't have any anchor to
 search for. For example, how do you search for code that allocates an
 array and *doesn't* deallocate it?


By using a decent memory debugger/profiler like Valgrind?

David
Dec 05 2011
prev sibling next sibling parent reply Johannes Pfau <spam example.com> writes:
Steven Schveighoffer wrote:

On Sat, 03 Dec 2011 12:43:08 -0500, Marco Leise <Marco.Leise gmx.de>
wrote:


 Am 03.12.2011, 10:01 Uhr, schrieb Benjamin Thaut  
 <code benjamin-thaut.de>:


 Thanks, That doesn't sound to bad. If I manage to get a non leaking
 non gc version of d-runtime working would there be any interest in
 that?

 Kind Regards
 Benjamin Thaut


 That would be most interesting! How do you imagine this could work  
 though? As an example D array slices work on GC memory and you can
 have dozens of slices on the same memory block. Would these memory
 blocks be reference counted then?


D array appending operations require a GC.  You'd have to avoid
them. This should be possible with some changes to druntime (just find
all the cases and remove them).

But what's difficult is finding leaks which don't have any anchor to  
search for.  For example, how do you search for code that allocates
an array and *doesn't* deallocate it?


I'd say don't replace the GC with a simple malloc/free stub, remove it
completely. Add a switch to dmd (-nogc) which disables all features
which need the gc (especially 'new', array appending, ...). This will
break a lot of code, but you won't have silent memory leaks this way.

Code which should be usable with and without GC could use the new
allocators API once that's finished, or a special "version(NOGC)"
code path.

-- 
Johannes Pfau
Dec 05 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-12-05 17:56, Johannes Pfau wrote:

 I'd say don't replace the GC with a simple malloc/free stub, remove it
 completely. Add a switch to dmd (-nogc) which disables all features
 which need the gc (especially 'new', array appending, ...). This will
 break a lot of code, but you won't have silent memory leaks this way.

 Code which should be usable with and without GC could use the new
 allocators API once that's finished, or a special "version(NOGC)"
 code path.


You could just remove the GC and you will get linker errors when it's 
used. A -nogc flag would be better, but this could be usable workaround. 
It may be possible to add a GC implementation that just adds static 
asserts for all functions, I don't know.

-- 
/Jacob Carlborg
Dec 05 2011
prev sibling parent Somedude <lovelydear mailmetrash.com> writes:
Le 05/12/2011 14:46, Steven Schveighoffer a écrit :

 On Sat, 03 Dec 2011 12:43:08 -0500, Marco Leise <Marco.Leise gmx.de> wrote:
 

 Am 03.12.2011, 10:01 Uhr, schrieb Benjamin Thaut
 <code benjamin-thaut.de>:


 Thanks, That doesn't sound to bad. If I manage to get a non leaking
 non gc version of d-runtime working would there be any interest in that?

 Kind Regards
 Benjamin Thaut


 That would be most interesting! How do you imagine this could work
 though? As an example D array slices work on GC memory and you can
 have dozens of slices on the same memory block. Would these memory
 blocks be reference counted then?


 
 D array appending operations require a GC.  You'd have to avoid them. 
 This should be possible with some changes to druntime (just find all the
 cases and remove them).
 
 But what's difficult is finding leaks which don't have any anchor to
 search for.  For example, how do you search for code that allocates an
 array and *doesn't* deallocate it?
 
 It's definitely an interesting and worthwhile project.
 
 -Steve


You can modify malloc and free so that :
1) malloc stores in a hash map {address => source file + line & size},
2) free deletes the entry with key address if it exists
At any time, you can check what has not been released by printing the
content of the hashmap .

I did that in C once, in a project where I couldn't use Valgrind.
Dec 05 2011
prev sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Sat, 03 Dec 2011 19:05:02 +0100, Andrej Mitrovic  
<andrej.mitrovich gmail.com> wrote:


 +1 on interest on having this. Back when I was attempting to port VST
 to D I got asked by a Steinberg dev how I can guarantee that D plugins
 will work. But I couldn't guarantee it, if a GC collection were to run
 the plugin would freeze, the host would crash, and the host company
 would likely get the blame. A GC-less environment would be great for
 realtime plugins, I'm not sure how arrays/slices would work, but I
 guess custom Array refcounted structs could be made or something.


That's not true.
In your audio routines you must not call unbounded routines,
i.e. you'd better avoid allocations in the first place or
use a constant time allocator.
Any serious VST host will use two threads to avoid drop-outs.
One high priority for driving audio and one low priority for GUI.
Halting the GUI thread during GC allocations will not affect the audio.
Dec 05 2011