www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - static void arrays under garbage control?

reply captaindet <2krnk gmx.net> writes:
if i understand correctly, static arrays are exempt from GC scanning for memory
pointers
  
http://dlang.org/garbage.html : "Pointers in D can be broadly divided into two
categories: Those that point to garbage collected memory, and those that do
not. Examples of the latter are pointers created by calls to C's malloc(),
pointers received from C library routines, pointers to static data."


but there is also a warning for void arrays

http://dlang.org/arrays.html : The garbage collector "will scan void[] arrays
for pointers, since such an array may have been implicitly converted from an
array of pointers or an array of elements that contain pointers."


does this warning only apply to dynamic void[] arrays but not to static
void[CTconstant] arrays?

(because sometimes the docs also mean static arrays even if just "type[]" is
written.)

thanks!

ps: this is for 32bit apps
Feb 25 2015
next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 26 February 2015 at 01:15:37 UTC, captaindet wrote:

 pointers to static data."


Keep in mind that this is pointers TO static data - that is, if 
the object itself is static, it is not to be collected.

static int[5] foo = [1,2,3,4,5]; // foo will never be collected, 
even if there are no pointers to it


But, the static array itself will be scanned:

static Object[2] objects;

The objects stored in there won't be garbage collected because 
this array is still scanned.


So


 does this warning only apply to dynamic void[] arrays but not 
 to static void[CTconstant] arrays?


Both of those will be scanned for pointers.
Feb 25 2015
parent reply captaindet <2krnk gmx.net> writes:
On 2015-02-25 19:24, Adam D. Ruppe wrote:

 does this warning only apply to dynamic void[] arrays but not to static
void[CTconstant] arrays?


 Both of those will be scanned for pointers.


thanks, adam,

so i should always use

struct Stuff2Do{
     static ubyte[1024*1024] buffer4speed = void; // even if untyped at this
point

     // more
}

to avoid that the GC is scanning my buffer for pointers?
Feb 25 2015
next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
You don't have to do all that, if it is typed ubyte[] instead of 
void[], it shouldn't be scanned at all anyway.
Feb 25 2015
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Wed, Feb 25, 2015 at 08:20:37PM -0600, captaindet via Digitalmars-d-learn
wrote:
[...]

 struct Stuff2Do{
     static ubyte[1024*1024] buffer4speed = void; // even if untyped at this
point
 
     // more
 }


[...]

Tangential note: be careful with putting a large static array inside a
struct. Structs are passed by value, which means that if you didn't
allocate that struct on the heap and you pass it around to various
functions, you will overflow your stack very quickly -- every function
call that passes the struct will consume 1MB of stack space. Many OSes
do not allocate that much space for the runtime stack.


T

-- 
Real Programmers use "cat > a.out".
Feb 25 2015
parent captaindet <2krnk gmx.net> writes:
On 2015-02-25 20:45, H. S. Teoh via Digitalmars-d-learn wrote:

 On Wed, Feb 25, 2015 at 08:20:37PM -0600, captaindet via Digitalmars-d-learn
wrote:
 [...]

 struct Stuff2Do{
      static ubyte[1024*1024] buffer4speed = void; // even if untyped at this
point

      // more
 }


 [...]

 Tangential note: be careful with putting a large static array inside a
 struct. Structs are passed by value, which means that if you didn't
 allocate that struct on the heap and you pass it around to various
 functions, you will overflow your stack very quickly -- every function
 call that passes the struct will consume 1MB of stack space. Many OSes
 do not allocate that much space for the runtime stack.


 T


  
this is why i tried to get away with a 'static' static array, which only exists
once for all Stuff2Do structs. but as it seems, i'll rather need on the order
of 512MB buffer, so i will probably go with c.stdlib.malloc

/det
Feb 25 2015
prev sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Wed, 25 Feb 2015 20:20:37 -0600, captaindet wrote:


 On 2015-02-25 19:24, Adam D. Ruppe wrote:

 does this warning only apply to dynamic void[] arrays but not to
 static void[CTconstant] arrays?


 Both of those will be scanned for pointers.


=20
 thanks, adam,
=20
 so i should always use
=20
 struct Stuff2Do{
      static ubyte[1024*1024] buffer4speed =3D void; // even if untyped at
      this point
=20
      // more
 }
=20
 to avoid that the GC is scanning my buffer for pointers?


compiler is clever enough to not mark `ubyte` arrays as GC ranges.=
Feb 25 2015
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/25/15 8:15 PM, captaindet wrote:

 if i understand correctly, static arrays are exempt from GC scanning for
 memory pointers

 http://dlang.org/garbage.html : "Pointers in D can be broadly divided
 into two categories: Those that point to garbage collected memory, and
 those that do not. Examples of the latter are pointers created by calls
 to C's malloc(), pointers received from C library routines, pointers to
 static data."


 but there is also a warning for void arrays

 http://dlang.org/arrays.html : The garbage collector "will scan void[]
 arrays for pointers, since such an array may have been implicitly
 converted from an array of pointers or an array of elements that contain
 pointers."


 does this warning only apply to dynamic void[] arrays but not to static
 void[CTconstant] arrays?

 (because sometimes the docs also mean static arrays even if just
 "type[]" is written.)

 thanks!

 ps: this is for 32bit apps


Somewhat missing in this disscusion is:

the GC does not know what an array's type currently is, it only knows 
what it was when it was allocated. So for instance:

ubyte[] arr = cast(ubyte[])(new void[100]); // scanned for pointers
void[] arr = new ubyte[100]; // not scanned for pointers.

In fact, the GC has no idea of type at all. It just knows about memory 
blocks, and whether those blocks are flagged as having pointers or not 
having pointers. The call to new is what tells the GC "hm.. this is a 
type that may contain pointers, set that flag!"

Static data I believe is always scanned conservatively because no type 
information is stored for it ever, even on allocation (i.e. program 
startup).

-Steve
Feb 26 2015
parent reply captaindet <2krnk gmx.net> writes:
On 2015-02-26 10:07, Steven Schveighoffer wrote:

 Static data I believe is always scanned conservatively because no
 type information is stored for it ever, even on allocation (i.e.
 program startup).  


ouh, the confusion goes on... are you saying that

{
     // will be all scanned by GC for
     // potential pointers into GC managed memory:
     void[16] buffer0 = void;
     ubyte[16] buffer1;
     uint[4] buffer3;
     
     // will not be scanned by GC for pointers:
     void[] buffer4 = cast(void[])(new ubyte[16]);
     uint[] buffer5 = cast(uint[])(new ubyte[16]);
}
Feb 26 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/26/15 11:57 AM, captaindet wrote:

 On 2015-02-26 10:07, Steven Schveighoffer wrote:

 Static data I believe is always scanned conservatively because no
 type information is stored for it ever, even on allocation (i.e.
 program startup).


 ouh, the confusion goes on... are you saying that

 {
      // will be all scanned by GC for
      // potential pointers into GC managed memory:
      void[16] buffer0 = void;
      ubyte[16] buffer1;
      uint[4] buffer3;


Yes, all 3 are stack based. This is not static data, which would be like 
uint[4] at module level. Those are also treated as scannable, as the 
entire stack of every thread is scanned.


      // will not be scanned by GC for pointers:
      void[] buffer4 = cast(void[])(new ubyte[16]);
      uint[] buffer5 = cast(uint[])(new ubyte[16]);


Correct, since they are allocated as ubyte[]. uint[] also would not be 
scanned.

-Steve
Feb 26 2015
next sibling parent captaindet <2krnk gmx.net> writes:
On 2015-02-26 12:07, Steven Schveighoffer wrote:

 On 2/26/15 11:57 AM, captaindet wrote:

 On 2015-02-26 10:07, Steven Schveighoffer wrote:

 Static data I believe is always scanned conservatively because no
 type information is stored for it ever, even on allocation (i.e.
 program startup).


 ouh, the confusion goes on... are you saying that

 {
 // will be all scanned by GC for
 // potential pointers into GC managed memory:
 void[16] buffer0 = void;
 ubyte[16] buffer1;
 uint[4] buffer3;


 Yes, all 3 are stack based. This is not static data, which would be like
uint[4] at module level. Those are also treated as scannable, as the entire
stack of every thread is scanned.


 // will not be scanned by GC for pointers:
 void[] buffer4 = cast(void[])(new ubyte[16]);
 uint[] buffer5 = cast(uint[])(new ubyte[16]);


 Correct, since they are allocated as ubyte[]. uint[] also would not be scanned.

 -Steve


got it, finally! thanks to everyone for their help.

/det
Feb 26 2015
prev sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 02/26/2015 10:07 AM, Steven Schveighoffer wrote:


      // will not be scanned by GC for pointers:
      void[] buffer4 = cast(void[])(new ubyte[16]);
      uint[] buffer5 = cast(uint[])(new ubyte[16]);


 Correct, since they are allocated as ubyte[]. uint[] also would not be
 scanned.


I've been under the impression that current GC being imprecise, even an 
unfortunate bit-pattern in an int would keep otherwise dead objects 
alive. Is that still true for a single int?

And is 'new int[N]' better in this regard because it never keeps objects 
alive?

Ali
Feb 26 2015
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/26/15 2:28 PM, Ali Çehreli wrote:

 On 02/26/2015 10:07 AM, Steven Schveighoffer wrote:

  >>      // will not be scanned by GC for pointers:
  >>      void[] buffer4 = cast(void[])(new ubyte[16]);
  >>      uint[] buffer5 = cast(uint[])(new ubyte[16]);
  >
  > Correct, since they are allocated as ubyte[]. uint[] also would not be
  > scanned.

 I've been under the impression that current GC being imprecise, even an
 unfortunate bit-pattern in an int would keep otherwise dead objects
 alive. Is that still true for a single int?


It depends on where the int is, not whether it's an int or not. An int 
on the stack will be scanned as if it were a pointer, because the GC 
does not know the type for those bits.

An int on the heap will be scanned if it is in a block marked as having 
pointers.


 And is 'new int[N]' better in this regard because it never keeps objects
 alive?


Correct, at the point you allocate new int[N], the type system is aware 
that the block will NOT contain pointers, so the GC is informed of this.

Now, if you said new int[][N], the block contains pointers, as each 
element has a pointer and length. But the lengths will ALSO be scanned 
even though they aren't pointers.

This is a heuristic for the scanner. It's very blunt, but it's better 
than completely imprecise :)

A truly precise scanner would be informed not only of the block bit 
patterns (which bits are alive, which are pointers, etc), but also of 
the stack frame and static data bit patterns. Such a scanner would be 
better at avoiding leaked memory, but possibly perform worse, since it 
may spend extra time parsing type data when the entire block is all 
pointers, etc. It also requires more metadata, and can affect cache 
coherency. I'll put forth right now, I am NOT the person to ask about 
this in any detail :)

-Steve
Feb 26 2015