www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - GC and void[N] in struct

reply vit <vit vit.vit> writes:
Hello,
I have this struct:

struct S{
     uint kind;
     void[N] data_;

}

Instances of struct S are allocated by standard GC new and 
S.data_ can contain pointers/ranges to GC allocated data.
If is GC disabled then  program run fine. But when is GC enabled 
then it fail randomly.

If the definition of S look like this:

struct S{
     void[N] data_;
     uint kind;
}

then program run fine with GC.enable.

Whats the problem? Something with alignment?
Aug 06 2018
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/6/18 2:22 PM, vit wrote:

 Hello,
 I have this struct:
 
 struct S{
      uint kind;
      void[N] data_;


define "N"


 
 }
 
 Instances of struct S are allocated by standard GC new and S.data_ can 
 contain pointers/ranges to GC allocated data.
 If is GC disabled then  program run fine. But when is GC enabled then it 
 fail randomly.


how does it fail?

-Steve
Aug 06 2018
parent reply vit <vit vit.vit> writes:
On Monday, 6 August 2018 at 18:28:11 UTC, Steven Schveighoffer 
wrote:

 On 8/6/18 2:22 PM, vit wrote:

 Hello,
 I have this struct:
 
 struct S{
      uint kind;
      void[N] data_;


 define "N"


 
 }
 
 Instances of struct S are allocated by standard GC new and 
 S.data_ can contain pointers/ranges to GC allocated data.
 If is GC disabled then  program run fine. But when is GC 
 enabled then it fail randomly.


 how does it fail?

 -Steve



private auto sizeOf(T)(){return T.sizeof;}

struct ExprImpl(Ts...){
     enum N = max(staticMap!(sizeOf, Ts));

     invariant(kind_ != 0);
     uint kind_ = 0;
     void[N] data_;

     this(T)(auto ref T x){/+emplace T to data_ and change kind_ 
to something != 0+/}
}

Ts == structs



data change without triggering invariant after allocation in 
other part of program.
Aug 06 2018
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/6/18 2:59 PM, vit wrote:

 On Monday, 6 August 2018 at 18:28:11 UTC, Steven Schveighoffer wrote:

 On 8/6/18 2:22 PM, vit wrote:

 Hello,
 I have this struct:

 struct S{
      uint kind;
      void[N] data_;


 define "N"


 }

 Instances of struct S are allocated by standard GC new and S.data_ 
 can contain pointers/ranges to GC allocated data.
 If is GC disabled then  program run fine. But when is GC enabled then 
 it fail randomly.


 how does it fail?


 
 
 private auto sizeOf(T)(){return T.sizeof;}


Hm... wouldn't enum sizeOf(T) = T.sizeof work better?


 
 struct ExprImpl(Ts...){
      enum N = max(staticMap!(sizeOf, Ts));


This is clever!


 
      invariant(kind_ != 0);
      uint kind_ = 0;
      void[N] data_;
 
      this(T)(auto ref T x){/+emplace T to data_ and change kind_ to 
 something != 0+/}
 }
 
 Ts == structs
 
 
 
 data change without triggering invariant after allocation in other part 
 of program.


Most definitely this is alignment problem.

Here is what I *think* is happening:

1. You are constructing one of these structs, and storing a pointer as 
the T type.
2. You are on a 64-bit CPU.
3. The pointer is misaligned on the CPU, so when the GC scans this 
struct to see if it's pointing at anything, it sees one half as the 
kind_ value, and the other half is half of the pointer.
4. It misses the object being pointed at by the T inside the struct, and 
collects it, leaving a dangling pointer.
5. Memory corruption.

when you put the void[N] member *first*, it can properly align the item 
(most cases where the compiler is placing data, it starts out aligned) 
but this does not guarantee you have proper alignment, as void[N] has no 
alignment constraints.

I'd recommend instead, changing the uint kind_ to a size_t. This not 
only aligns the void[N] to size_t size, which should put any pointers in 
the right place, but it also makes sure the entire struct is aligned.

-Steve
Aug 06 2018
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/6/18 3:43 PM, Steven Schveighoffer wrote:

 On 8/6/18 2:59 PM, vit wrote:

 On Monday, 6 August 2018 at 18:28:11 UTC, Steven Schveighoffer wrote:

 On 8/6/18 2:22 PM, vit wrote:

 Hello,
 I have this struct:

 struct S{
      uint kind;
      void[N] data_;


 define "N"


 }

 Instances of struct S are allocated by standard GC new and S.data_ 
 can contain pointers/ranges to GC allocated data.
 If is GC disabled then  program run fine. But when is GC enabled 
 then it fail randomly.


 how does it fail?



 private auto sizeOf(T)(){return T.sizeof;}


 
 Hm... wouldn't enum sizeOf(T) = T.sizeof work better?
 

 struct ExprImpl(Ts...){
      enum N = max(staticMap!(sizeOf, Ts));


 
 This is clever!
 

      invariant(kind_ != 0);
      uint kind_ = 0;
      void[N] data_;

      this(T)(auto ref T x){/+emplace T to data_ and change kind_ to 
 something != 0+/}
 }

 Ts == structs



 data change without triggering invariant after allocation in other 
 part of program.


 
 Most definitely this is alignment problem.
 
 Here is what I *think* is happening:
 
 1. You are constructing one of these structs, and storing a pointer as 
 the T type.
 2. You are on a 64-bit CPU.
 3. The pointer is misaligned on the CPU, so when the GC scans this 
 struct to see if it's pointing at anything, it sees one half as the 
 kind_ value, and the other half is half of the pointer.
 4. It misses the object being pointed at by the T inside the struct, and 
 collects it, leaving a dangling pointer.
 5. Memory corruption.
 
 when you put the void[N] member *first*, it can properly align the item 
 (most cases where the compiler is placing data, it starts out aligned) 
 but this does not guarantee you have proper alignment, as void[N] has no 
 alignment constraints.
 
 I'd recommend instead, changing the uint kind_ to a size_t. This not 
 only aligns the void[N] to size_t size, which should put any pointers in 
 the right place, but it also makes sure the entire struct is aligned.


BTW, is there a reason you aren't just using Algebraic?

https://dlang.org/phobos/std_variant.html#.Algebraic

-Steve
Aug 06 2018
parent reply vit <vit vit.vit> writes:
On Monday, 6 August 2018 at 19:56:03 UTC, Steven Schveighoffer 
wrote:

 BTW, is there a reason you aren't just using Algebraic?

 https://dlang.org/phobos/std_variant.html#.Algebraic

 -Steve


primarily visit for Algebraic isn't pure,  nogc,  safe, nothrow.
Aug 06 2018
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 6 August 2018 at 20:22:36 UTC, vit wrote:

 On Monday, 6 August 2018 at 19:56:03 UTC, Steven Schveighoffer 
 wrote:

 BTW, is there a reason you aren't just using Algebraic?

 https://dlang.org/phobos/std_variant.html#.Algebraic

 -Steve


 primarily visit for Algebraic isn't pure,  nogc,  safe, nothrow.


I wrote the 'sumtype' package to solve this exact problem:

https://code.dlang.org/packages/sumtype
Aug 06 2018
parent vit <vit vit.vit> writes:
On Monday, 6 August 2018 at 21:23:36 UTC, Paul Backus wrote:

 On Monday, 6 August 2018 at 20:22:36 UTC, vit wrote:

 On Monday, 6 August 2018 at 19:56:03 UTC, Steven Schveighoffer 
 wrote:

 BTW, is there a reason you aren't just using Algebraic?

 https://dlang.org/phobos/std_variant.html#.Algebraic

 -Steve


 primarily visit for Algebraic isn't pure,  nogc,  safe, 
 nothrow.


 I wrote the 'sumtype' package to solve this exact problem:

 https://code.dlang.org/packages/sumtype


I'm using simpler (and less powerful) version with different  
visit/visitor syntax:


struct Foo{
	string foo;
}
struct Bar1{
	string bar;
}
struct Bar2{
	string bar;
}

void main()pure nothrow  safe  nogc{
	Variant!(true, Foo, Bar1, Bar2) var; ///Nullable == true
	
	static visit(T)(auto ref const(T) x, string def = ""){
		import std.experimental.all;
		
		static if(is(T == Foo)){
			return x.foo;
		}
		else static if(false
			|| is(T == Bar1)
			|| is(T == Bar2)
		){
			return x.bar;
		}
		else static if(is(T == typeof(null))){
			return def;
		}
		else static assert(0, "no impl");
	}
	
	assert(var.isa!null);
	assert(var.visitor!visit("null") == "null");
	
	var = Foo("foo");
	assert(var.isa!Foo);
	assert(var.visitor!visit == "foo");
	
	var = Bar1("bar1");
	assert(var.isa!Bar1);
	assert(var.visitor!visit == "bar1");
	
	var = Bar2("bar2");
	assert(var.isa!Bar2);
	assert(var.visitor!visit == "bar2");
	
	var = null;
	assert(var.isa!null);
	
	auto var2 = Variant!(false, Foo, Bar1, 
Bar2)(Foo("foo"));///Nullable == false
	
	assert(var2.visitor!visit == "foo");
	assert(var2.as!Foo.foo == "foo");
	
	///var2 = null;		//error, variant is not null
	

}

full code: https://dpaste.dzfl.pl/d83ecca23694
Aug 06 2018
prev sibling parent Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:
On Monday, 6 August 2018 at 19:43:17 UTC, Steven Schveighoffer 
wrote:

 On 8/6/18 2:59 PM, vit wrote:

 struct ExprImpl(Ts...){
      enum N = max(staticMap!(sizeOf, Ts));


 This is clever!


No need to be clever though - we've got std.traits.Largest for 
exactly this kind of purpose.

--
   Simen
Aug 06 2018
prev sibling parent reply nkm1 <t4nk074 openmailbox.org> writes:
On Monday, 6 August 2018 at 18:22:24 UTC, vit wrote:

 Hello,
 I have this struct:

 struct S{
     uint kind;
     void[N] data_;

 }

 Instances of struct S are allocated by standard GC new and 
 S.data_ can contain pointers/ranges to GC allocated data.
 If is GC disabled then  program run fine. But when is GC 
 enabled then it fail randomly.

 If the definition of S look like this:

 struct S{
     void[N] data_;
     uint kind;
 }

 then program run fine with GC.enable.

 Whats the problem? Something with alignment?


Probably. Try something like:

struct S
{
     uint kind;
     align((void *).alignof) void[N] data_;
}

And see if it solves the problem.
Aug 06 2018
parent vit <vit vit.vit> writes:
On Monday, 6 August 2018 at 19:17:58 UTC, nkm1 wrote:

 On Monday, 6 August 2018 at 18:22:24 UTC, vit wrote:

 Hello,
 I have this struct:

 struct S{
     uint kind;
     void[N] data_;

 }

 Instances of struct S are allocated by standard GC new and 
 S.data_ can contain pointers/ranges to GC allocated data.
 If is GC disabled then  program run fine. But when is GC 
 enabled then it fail randomly.

 If the definition of S look like this:

 struct S{
     void[N] data_;
     uint kind;
 }

 then program run fine with GC.enable.

 Whats the problem? Something with alignment?


 Probably. Try something like:

 struct S
 {
     uint kind;
     align((void *).alignof) void[N] data_;
 }

 And see if it solves the problem.


align((void *).alignof) work, thanks.
Aug 06 2018