www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - GC.calloc(), then what?

reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
1) After allocating memory by GC.calloc() to place objects on it, what 
else should one do? In what situations does one need to call addRoot() 
or addRange()?

2) Does the answer to the previous question differ for struct objects 
versus class objects?

3) Is there a difference between core.stdc.stdlib.calloc() and 
GC.calloc() in that regard? Which one to use in what situation?

4) Are the random bit patterns in a malloc()'ed memory always a concern 
for false pointers? Does that become a concern after calling addRoot() 
or addRange()? If so, why would anyone ever malloc() instead of always 
calloc()'ing?

Ali
Jun 27 2014
parent reply "safety0ff" <safety0ff.dev gmail.com> writes:
On Friday, 27 June 2014 at 07:03:28 UTC, Ali Çehreli wrote:

 1) After allocating memory by GC.calloc() to place objects on 
 it, what else should one do?


Use std.conv.emplace.


 In what situations does one need to call addRoot() or 
 addRange()?


Add root creates an internal reference within the GC to the 
memory pointed by the argument (void* p.)
This pins the memory so that it won't be collected by the GC. 
E.g. you're going to pass a string to an extern C function, and 
the function will store a pointer to the string within its own 
data structures. Since the GC won't have access to the data 
structures, you must addRoot it to avoid creating a dangling 
pointer in the C data structure.

Add range is usually for cases when you use 
stdc.stdlib.malloc/calloc and place pointers to GC managed memory 
within that memory. This allows the GC to scan that memory for 
pointers during collection, otherwise it may reclaim memory which 
is pointed to my malloc'd memory.


 2) Does the answer to the previous question differ for struct 
 objects versus class objects?


No.


 3) Is there a difference between core.stdc.stdlib.calloc() and 
 GC.calloc() in that regard? Which one to use in what situation?


One is GC managed, the other is not. calloc simply means the 
memory is pre-zero'd, it has nothing to do with "C allocation" / 
"allocation in the C language"


 4) Are the random bit patterns in a malloc()'ed memory always a 
 concern for false pointers? Does that become a concern after 
 calling addRoot() or addRange()?


If by malloc you're talking about stdc.stdlib.malloc then:
It only becomes a concern after you call addRange, and the false 
pointers potential is only present within the range you gave to 
addRange.
So if you over-allocate using malloc and give the entire memory 
range to addRange, then any false pointers in the un-intialized 
portion become a concern.

If you're talking about GC.malloc():
Currently the GC zeros the memory unless you allocate NO_SCAN 
memory, so it only differs in the NO_SCAN case.


 If so, why would anyone ever malloc() instead of always 
 calloc()'ing?


To save on redundant zero'ing.
Jun 27 2014
next sibling parent reply "safety0ff" <safety0ff.dev gmail.com> writes:
I realize that my answer isn't completely clear in some cases, if 
you still have questions, ask away.
Jun 27 2014
parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 06/27/2014 12:53 AM, safety0ff wrote:

 I realize that my answer isn't completely clear in some cases, if you
 still have questions, ask away.


Done! That's why we are here anyway. :p

Ali
Jun 27 2014
prev sibling next sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
Thank you for your responses. I am partly enlightened. :p

On 06/27/2014 12:34 AM, safety0ff wrote:


 On Friday, 27 June 2014 at 07:03:28 UTC, Ali Çehreli wrote:

 1) After allocating memory by GC.calloc() to place objects on it, what
 else should one do?


 Use std.conv.emplace.


That much I know. :) I have actually finished the first draft of 
translating my memory management chapter (the last one in the book!) and 
trying to make sure that the information is correct.


 In what situations does one need to call addRoot() or addRange()?


 Add root creates an internal reference within the GC to the memory
 pointed by the argument (void* p.)
 This pins the memory so that it won't be collected by the GC. E.g.
 you're going to pass a string to an extern C function, and the function
 will store a pointer to the string within its own data structures. Since
 the GC won't have access to the data structures, you must addRoot it to
 avoid creating a dangling pointer in the C data structure.


Additionally and according to the documentation, any other GC blocks 
will be considered live. So, addRoot makes a true roots where the GC 
starts its scanning from.


 Add range is usually for cases when you use stdc.stdlib.malloc/calloc
 and place pointers to GC managed memory within that memory. This allows
 the GC to scan that memory for pointers during collection, otherwise it
 may reclaim memory which is pointed to my malloc'd memory.


One part that I don't understand in the documentation is "if p points 
into a GC-managed memory block, addRange does not mark this block as live".



Does that mean that if I have objects in my addRange'd memory that in 
turn have references to objects in the GC-managed memory, my references 
in my memory may be stale?

If so, does that mean that if I manage objects in my memory, all their 
members should be managed by me as well?

This seems to bring two types of GC-managed memory:

1) addRoot'ed memory that gets scanned deep (references are followed)

2) addRange'd memory that gets scanned shallow (references are not followed)

See, that's confusing: What does that mean? I still hold the memory 
block anyway; what does the GC achieve by scanning my memory if it's not 
going to follow references anyway?


 2) Does the answer to the previous question differ for struct objects
 versus class objects?


 No.


 3) Is there a difference between core.stdc.stdlib.calloc() and
 GC.calloc() in that regard? Which one to use in what situation?


 One is GC managed, the other is not. calloc simply means the memory is
 pre-zero'd, it has nothing to do with "C allocation" / "allocation in
 the C language"


I know even that much. ;) I find people's malloc+memset code amusing.


 4) Are the random bit patterns in a malloc()'ed memory always a
 concern for false pointers? Does that become a concern after calling
 addRoot() or addRange()?


 If by malloc you're talking about stdc.stdlib.malloc then:
 It only becomes a concern after you call addRange,


But addRange doesn't seem to make sense for stdlib.malloc'ed memory, 
right? The reason is, that memory is not managed by the GC so there is 
no danger of losing that memory due to a collection anyway. It will go 
away only when I call stdlib.free.


 and the false
 pointers potential is only present within the range you gave to addRange.
 So if you over-allocate using malloc and give the entire memory range to
 addRange, then any false pointers in the un-intialized portion become a
 concern.


Repeating myself, that makes sense but I don't see when I would need 
addRange on a stdlib.malloc'ed memory.


 If you're talking about GC.malloc():
 Currently the GC zeros the memory unless you allocate NO_SCAN memory, so
 it only differs in the NO_SCAN case.


So, the GC's default behavior is to scan the memory, necessitating 
clearing the contents? That seems to make GC.malloc() behave the same as 
GC.calloc() by default, doesn't it?

So, is this guideline right?

   "GC.malloc() makes sense only with NO_SCAN."


 If so, why would anyone ever malloc() instead of always calloc()'ing?


 To save on redundant zero'ing.


And again, redundant zero'ing is saved only when used with NO_SCAN.

I think I finally understand the main difference between stdlib.malloc 
and GC.malloc: The latter gets collected by the GC.

Another question: Do GC.malloc'ed and GC.calloc'ed memory scanned deep?

Ali
Jun 27 2014
next sibling parent reply "safety0ff" <safety0ff.dev gmail.com> writes:
On Friday, 27 June 2014 at 08:17:07 UTC, Ali Çehreli wrote:

 Thank you for your responses. I am partly enlightened. :p


I know you're a knowledgeable person in the D community, I may 
have stated many things you already knew, but I tried to answer 
the questions as-is.



 On 06/27/2014 12:34 AM, safety0ff wrote:


 Add range is usually for cases when you use


 stdc.stdlib.malloc/calloc

 and place pointers to GC managed memory within that memory.


 This allows

 the GC to scan that memory for pointers during collection,


 otherwise it

 may reclaim memory which is pointed to my malloc'd memory.


 One part that I don't understand in the documentation is "if p 
 points into a GC-managed memory block, addRange does not mark 
 this block as live".

 [SNIP]

 See, that's confusing: What does that mean? I still hold the 
 memory block anyway; what does the GC achieve by scanning my 
 memory if it's not going to follow references anyway?


The GC _will_ follow references (i.e. scan deeply,) that's the 
whole point of addRange.
What that documentation is saying is that:

If you pass a range R through addRange, and R lies in the GC 
heap, then once there are no pointers (roots) to R, the GC will 
collect it anyway regardless that you called addRange on it.

In other words, prefer using addRoot for GC memory and addRange 
for non-GC memory.



 4) Are the random bit patterns in a malloc()'ed memory




 always a

 concern for false pointers? Does that become a concern after




 calling

 addRoot() or addRange()?


 If by malloc you're talking about stdc.stdlib.malloc then:
 It only becomes a concern after you call addRange,


 But addRange doesn't seem to make sense for stdlib.malloc'ed 
 memory, right? The reason is, that memory is not managed by the 
 GC so there is no danger of losing that memory due to a 
 collection anyway. It will go away only when I call stdlib.free.


addRange almost exclusively makes sense with stdlib.malloc'ed 
memory.
As you've stated: If you pass it GC memory it does not mark the 
block as live.

I believe the answer above clears things up: the GC does scan the 
range, and scanning is always "deep" (i.e. when it finds pointers 
to unmarked GC memory, it marks them.)

Conversely, addRoot exclusively makes sense with GC memory.


 If you're talking about GC.malloc():
 Currently the GC zeros the memory unless you allocate NO_SCAN


 memory, so

 it only differs in the NO_SCAN case.


 So, the GC's default behavior is to scan the memory, 
 necessitating clearing the contents? That seems to make 
 GC.malloc() behave the same as GC.calloc() by default, doesn't 
 it?



I don't believe it's necessary to clear it, it's just a measure 
against false pointers (AFAIK.)



 So, is this guideline right?

   "GC.malloc() makes sense only with NO_SCAN."



I wouldn't make a guideline like that, just say that: if you want 
the memory to be guaranteed to be zero'd use GC.calloc.

However, due to GC internals (for preventing false pointers,) 
GC.malloc'd memory  will often be zero'd anyway.


 If so, why would anyone ever malloc() instead of always




 calloc()'ing?

 To save on redundant zero'ing.


 And again, redundant zero'ing is saved only when used with 
 NO_SCAN.


Yup.


 I think I finally understand the main difference between 
 stdlib.malloc and GC.malloc: The latter gets collected by the 
 GC.


Yup.


 Another question: Do GC.malloc'ed and GC.calloc'ed memory 
 scanned deep?


Yes, only NO_SCAN memory doesn't get scanned, everything else 
does.
Jun 27 2014
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 06/27/2014 01:49 AM, safety0ff wrote:


 On Friday, 27 June 2014 at 08:17:07 UTC, Ali Çehreli wrote:

 Thank you for your responses. I am partly enlightened. :p


 I know you're a knowledgeable person in the D community, I may have
 stated many things you already knew, but I tried to answer the questions
 as-is.


I appreciated your answers, which were very helpful. What I meant was, I 
was "partially" enlightened but still had some questions. I am in much 
better shape now. :)

Ali
Jun 27 2014
parent "safety0ff" <safety0ff.dev gmail.com> writes:
On Friday, 27 June 2014 at 23:26:55 UTC, Ali Çehreli wrote:

 I appreciated your answers, which were very helpful. What I 
 meant was, I was "partially" enlightened but still had some 
 questions. I am in much better shape now. :)


Yea, I understood what you meant. :)
Jun 28 2014
prev sibling next sibling parent reply "safety0ff" <safety0ff.dev gmail.com> writes:
On Friday, 27 June 2014 at 08:17:07 UTC, Ali Çehreli wrote:

 So, the GC's default behavior is to scan the memory, 
 necessitating clearing the contents? That seems to make 
 GC.malloc() behave the same as GC.calloc() by default, doesn't 
 it?


Yes.
compare:
https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L543
to:
https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L419
Jun 27 2014
parent "safety0ff" <safety0ff.dev gmail.com> writes:
On Friday, 27 June 2014 at 09:20:53 UTC, safety0ff wrote:

 Yes.
 compare:
 https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L543
 to:
 https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L419


Actually, I just realized that I was wrong in saying "the memory 
likely be cleared by malloc" it's only the overallocation that 
gets cleared.
Jun 27 2014
prev sibling parent "eles" <eles eles.com> writes:
On Friday, 27 June 2014 at 08:17:07 UTC, Ali Çehreli wrote:

 Thank you for your responses. I am partly enlightened. :p

 On 06/27/2014 12:34 AM, safety0ff wrote:


 On Friday, 27 June 2014 at 07:03:28 UTC, Ali Çehreli wrote:





 But addRange doesn't seem to make sense for stdlib.malloc'ed 
 memory, right? The reason is, that memory is not managed by the 
 GC so there is no danger of losing that memory due to a 
 collection anyway. It will go away only when I call stdlib.free.


It is not about that, but about the fact that this unmanaged 
memory *might contain* references towards managed memory.

If you intend to place such references into this particular chunk 
of memory, then you need to tell GC to scan the memory chunk for 
references towards managed memory.

Otherwise, the GC might ignore this chunk of memory, find 
elsewhere no references towards a managed object, delete the 
managed object, then your pointer placed in the unmanaged memory 
becomes dangling.
Jun 27 2014
prev sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Friday, 27 June 2014 at 07:34:55 UTC, safety0ff wrote:

 On Friday, 27 June 2014 at 07:03:28 UTC, Ali Çehreli wrote:

 1) After allocating memory by GC.calloc() to place objects on 
 it, what else should one do?


 Use std.conv.emplace.


And possibly set BlkInfo flags to indicate whether the block has
pointers, and the finalize flag to indicate that it's an object.
I'd look at _d_newclass in Druntime/src/rt/lifetime.d for the
specifics.

To be honest, I think the GC interface is horribly outdated, but
my proposal for a redesign (first in 2010, then again in 2012 and
once again in 2013) never gained traction.  In short, what I'd
really like to have is a way to tell the GC to allocate an object
of type T.  Perhaps Andrei's allocators will sort this out and
the issue will be moot.  For reference:

http://lists.puremagic.com/pipermail/d-runtime/2010-August/000075.html
http://lists.puremagic.com/pipermail/d-runtime/2012-April/001095.html
http://lists.puremagic.com/pipermail/d-runtime/2013-July/001840.html
Jun 27 2014