www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Workaround for typeid access violation

reply "Etienne" <etcimon gmail.com> writes:
There is a bug regarding unordered object collection in the GC. 
My finalizer accesses another GC-allocated object and the 
application *sometimes* crashes here:

void _d_invariant(Object o)
{   ClassInfo c;

     //printf("__d_invariant(%p)\n", o);

     // BUG: needs to be filename/line of caller, not library 
routine
     assert(o !is null); // just do null check, not invariant check

     c = typeid(o);

         ^--------- this is the crash location

The culprit seems to be these operations:

00007ff6`881f324b 488b4510        mov     rax,qword ptr [rbp+10h]
00007ff6`881f324f 488b10          mov     rdx,qword ptr [rax]
00007ff6`881f3252 488b1a          mov     rbx,qword ptr [rdx] 
ds:00000000`00000000=????????????????

The vtable lookup wants to dereference a null entry. Not sure how 
I can fix this, but in the meantime I think typeid could actually 
add a small check on RDX and return null if that's what it is. 
Any input?
Jun 16 2015
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/16/15 10:00 AM, Etienne wrote:

 There is a bug regarding unordered object collection in the GC. My
 finalizer accesses another GC-allocated object


Don't do that. Nothing is guaranteed in order of destruction. Note that 
after calling the destructor of an object, the vtable pointer is nulled.

-Steve
Jun 16 2015
parent "Etienne" <etcimon gmail.com> writes:
On Tuesday, 16 June 2015 at 14:32:45 UTC, Steven Schveighoffer 
wrote:

 On 6/16/15 10:00 AM, Etienne wrote:

 There is a bug regarding unordered object collection in the 
 GC. My
 finalizer accesses another GC-allocated object


 Don't do that. Nothing is guaranteed in order of destruction. 
 Note that after calling the destructor of an object, the vtable 
 pointer is nulled.

 -Steve


Obviously the debugger shows that the object's destructor wasn't 
called. I have the class set as final and all the locals are 
there. Somehow its vtable is destroyed before though.

When I comment out the _d_invariant part at this point until the 
end, I can do a crazy amount of requests (this is a webserver) 
and there will be no crash at all.

So with the proper flags, its obviously possible to access a 
"final class" object locals during destruction, isn't this why 
the GC isn't re-entrant in the first place?
Jun 16 2015
prev sibling next sibling parent reply "rsw0x" <anonymous anonymous.com> writes:
On Tuesday, 16 June 2015 at 14:00:55 UTC, Etienne wrote:

 There is a bug regarding unordered object collection in the GC. 
 My finalizer accesses another GC-allocated object and the 
 application *sometimes* crashes here:

 void _d_invariant(Object o)
 {   ClassInfo c;

     //printf("__d_invariant(%p)\n", o);

     // BUG: needs to be filename/line of caller, not library 
 routine
     assert(o !is null); // just do null check, not invariant 
 check

     c = typeid(o);

         ^--------- this is the crash location

 The culprit seems to be these operations:

 00007ff6`881f324b 488b4510        mov     rax,qword ptr 
 [rbp+10h]
 00007ff6`881f324f 488b10          mov     rdx,qword ptr [rax]
 00007ff6`881f3252 488b1a          mov     rbx,qword ptr [rdx] 
 ds:00000000`00000000=????????????????

 The vtable lookup wants to dereference a null entry. Not sure 
 how I can fix this, but in the meantime I think typeid could 
 actually add a small check on RDX and return null if that's 
 what it is. Any input?


This is undefined behavior, the only solution is "don't do it"
see my thread
http://forum.dlang.org/post/vcpcjujvkbuoswyzycat forum.dlang.org

destructors as they are shouldn't exist at all, they are 
incredibly bug prone.

Bye.
Jun 16 2015
parent reply "Etienne" <etcimon gmail.com> writes:
On Tuesday, 16 June 2015 at 15:39:06 UTC, rsw0x wrote:

 destructors as they are shouldn't exist at all, they are 
 incredibly bug prone.

 Bye.


To be fair, everything is bug prone until you understand them. GC 
finalization is done in a single lock, none of the memory is 
re-used, so objects can have their own "destroyed" flags and 
destroy eachother fine if the typeinfo issue isn't there.

On the other hand, by keeping the GC destructors, we must agree 
that all the destructors are declared this way:

shared  nogc nothrow ~this()

Only then can we count it in as having predictable behavior.

I can't really agree on removing finalizers, I'd really love to 
master GC destructors in D so that I can return plain class 
objects from my functions with data allocated elsewhere, like in 
any managed language out there. It's really the only way. Don't 
get me wrong, I allocate and free on a freelist or pool 
everywhere I can, but the GC has its advantages and I'd really 
like to put it to work (safely) to build more convenient APIs.
Jun 16 2015
next sibling parent reply "rsw0x" <anonymous anonymous.com> writes:
On Tuesday, 16 June 2015 at 16:28:54 UTC, Etienne wrote:

 On Tuesday, 16 June 2015 at 15:39:06 UTC, rsw0x wrote:

 destructors as they are shouldn't exist at all, they are 
 incredibly bug prone.

 Bye.


 To be fair, everything is bug prone until you understand them.


No, they are just bug prone.


 GC finalization is done in a single lock, none of the memory is 
 re-used, so objects can have their own "destroyed" flags and 
 destroy eachother fine if the typeinfo issue isn't there.


Implementation-defined behavior.


 On the other hand, by keeping the GC destructors, we must agree 
 that all the destructors are declared this way:

 shared  nogc nothrow ~this()

 Only then can we count it in as having predictable behavior.


Still not predictable, as they can be ran in any thread, in any 
order, and be ran parallel.


 I can't really agree on removing finalizers, I'd really love to 
 master GC destructors in D so that I can return plain class 
 objects from my functions with data allocated elsewhere, like 
 in any managed language out there. It's really the only way. 
 Don't get me wrong, I allocate and free on a freelist or pool 
 everywhere I can, but the GC has its advantages and I'd really 
 like to put it to work (safely) to build more convenient APIs.


You're attempting to use GC for a problem that they don't solve 
because you don't have other tools to fix it. When all you have 
is a hammer, everything looks like a nail.
Jun 16 2015
parent reply "Etienne" <etcimon gmail.com> writes:
On Tuesday, 16 June 2015 at 20:08:36 UTC, rsw0x wrote:

 You're attempting to use GC for a problem that they don't solve 
 because you don't have other tools to fix it. When all you have 
 is a hammer, everything looks like a nail.


Well no, I have plenty of tools. I use a memory library that does 
a wide range of everything I need 
https://github.com/etcimon/memutils .. It's not like I'm reliant 
only on it, like it's my hammer and I only see nails..

It's simply easier to manage the lifetime of objects through the 
GC because sometimes I have them referred to as delegates at a 
lower level. I like having finalizers because those objects may 
hold thread-local containers.

Isn't that what we'd want? To use the GC only when it's important 
and manual memory management at other times? What happens when 
you need the two and your manually allocated objects are tracked 
by a GC-allocated object? Well I'm experimenting a solution here, 
or am I wrong to do so? I know the behavior is undefined in the 
docs, but I didn't post this in D.learn did I?

I think you're the one who's actually looking at everything like 
a nail wrt improving performance in benchmarks.
Jun 16 2015
next sibling parent "rsw0x" <anonymous anonymous.com> writes:
On Tuesday, 16 June 2015 at 20:30:50 UTC, Etienne wrote:

 On Tuesday, 16 June 2015 at 20:08:36 UTC, rsw0x wrote:

 You're attempting to use GC for a problem that they don't 
 solve because you don't have other tools to fix it. When all 
 you have is a hammer, everything looks like a nail.


 Well no, I have plenty of tools. I use a memory library that 
 does a wide range of everything I need 
 https://github.com/etcimon/memutils .. It's not like I'm 
 reliant only on it, like it's my hammer and I only see nails..

 It's simply easier to manage the lifetime of objects through 
 the GC because sometimes I have them referred to as delegates 
 at a lower level. I like having finalizers because those 
 objects may hold thread-local containers.


A GC is not for managing an object's lifetime. There are no 
guarantees from the GC.


 Isn't that what we'd want? To use the GC only when it's 
 important and manual memory management at other times? What 
 happens when you need the two and your manually allocated 
 objects are tracked by a GC-allocated object?


The correct answer would be for the GC object to be upgraded to 
an RCO because the resource's lifetime is now bound to the GC 
object.


 I think you're the one who's actually looking at everything 
 like a nail wrt improving performance in benchmarks.


No, I'm looking at correctness and not piling hack upon hack that 
will fall apart at the first sneeze to the GC codebase. 
Performance just happens to be a side effect of correctness.
Jun 16 2015
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/16/15 4:30 PM, Etienne wrote:

 On Tuesday, 16 June 2015 at 20:08:36 UTC, rsw0x wrote:

 You're attempting to use GC for a problem that they don't solve
 because you don't have other tools to fix it. When all you have is a
 hammer, everything looks like a nail.


 Well no, I have plenty of tools. I use a memory library that does a wide
 range of everything I need https://github.com/etcimon/memutils .. It's
 not like I'm reliant only on it, like it's my hammer and I only see nails..


Finalizers cannot access external GC memory. period. It's just not 
supported.

http://dlang.org/class.html#destructors

"This means that when the garbage collector calls a destructor for an 
object of a class that has members that are references to garbage 
collected objects, those references may no longer be valid. This means 
that destructors cannot reference sub objects."

If you want to manage memory from a GC'd object, you can use C malloc 
and C free.

Or, you can write your own GC that solves this problem that Sun/Oracle 
couldn't :)

In all seriousness, you could potentially have exceptions to this rule, 
but it would take a lot of cajoling of druntime and some funky  UDA magic.

-Steve
Jun 16 2015
parent reply "Etienne Cimon" <etcimon gmail.com> writes:
On Tuesday, 16 June 2015 at 22:21:28 UTC, Steven Schveighoffer 
wrote:

 If you want to manage memory from a GC'd object, you can use C 
 malloc and C free.

 Or, you can write your own GC that solves this problem that 
 Sun/Oracle couldn't :)

 In all seriousness, you could potentially have exceptions to 
 this rule, but it would take a lot of cajoling of druntime and 
 some funky  UDA magic.

 -Steve


Yeah, I'm going to make a personal branch and develop a GC 
thread-local, and ensure memory/vtbl is left intact during 
collection/finalization, it's a good experiment and I believe it 
will be a way to solve the issue in the near future. Of course, 
no support for moving shared objects in a foreign thread 
anymore.. like that was any useful in the first place :P
Jun 16 2015
parent reply "Etienne Cimon" <etcimon gmail.com> writes:
On Tuesday, 16 June 2015 at 22:31:38 UTC, Etienne Cimon wrote:

 Yeah, I'm going to make a personal branch and develop a GC 
 thread-local, and ensure memory/vtbl is left intact during 
 collection/finalization, it's a good experiment and I believe 
 it will be a way to solve the issue in the near future. Of 
 course, no support for moving shared objects in a foreign 
 thread anymore.. like that was any useful in the first place :P


My libraries run fine on a TLS GC so far:

https://github.com/etcimon/druntime/commit/7da3939637bd1642400dbed83e3b0ff2844386ac

Only error was with a signal handler trying to allocate on the 
GC. I think it'll be worth it for me to use this from now on. 
There's no locking and possibly no stop the world.
Jun 16 2015
next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Wed, 17 Jun 2015 02:35:39 +0000, Etienne Cimon wrote:


 Only error was with a signal handler trying to allocate on the GC.


signal handlers shouldn't allocate at all, being that from GC, or from=20
libc, or from some other allocator. the only thing signal handler can do=20
safely is setting some flag. anything other is working by pure luck.=
Jun 17 2015
prev sibling parent reply "Laeeth Isharc" <nospamlaeeth laeethnospam.laeeth.com> writes:
On Wednesday, 17 June 2015 at 02:35:40 UTC, Etienne Cimon wrote:

 On Tuesday, 16 June 2015 at 22:31:38 UTC, Etienne Cimon wrote:

 Yeah, I'm going to make a personal branch and develop a GC 
 thread-local, and ensure memory/vtbl is left intact during 
 collection/finalization, it's a good experiment and I believe 
 it will be a way to solve the issue in the near future. Of 
 course, no support for moving shared objects in a foreign 
 thread anymore.. like that was any useful in the first place :P


 My libraries run fine on a TLS GC so far:

 https://github.com/etcimon/druntime/commit/7da3939637bd1642400dbed83e3b0ff2844386ac

 Only error was with a signal handler trying to allocate on the 
 GC. I think it'll be worth it for me to use this from now on. 
 There's no locking and possibly no stop the world.


why 'possibly' ?
Jun 17 2015
parent reply "Etienne" <etcimon gmail.com> writes:
On Wednesday, 17 June 2015 at 18:41:13 UTC, Laeeth Isharc wrote:

 On Wednesday, 17 June 2015 at 02:35:40 UTC, Etienne Cimon wrote:

 On Tuesday, 16 June 2015 at 22:31:38 UTC, Etienne Cimon wrote:

 Yeah, I'm going to make a personal branch and develop a GC 
 thread-local, and ensure memory/vtbl is left intact during 
 collection/finalization, it's a good experiment and I believe 
 it will be a way to solve the issue in the near future. Of 
 course, no support for moving shared objects in a foreign 
 thread anymore.. like that was any useful in the first place 
 :P


 My libraries run fine on a TLS GC so far:

 https://github.com/etcimon/druntime/commit/7da3939637bd1642400dbed83e3b0ff2844386ac

 Only error was with a signal handler trying to allocate on the 
 GC. I think it'll be worth it for me to use this from now on. 
 There's no locking and possibly no stop the world.


 why 'possibly' ?


I hadn't tested it when I wrote that message. I committed the 
changes today and my tests were successful in a thread-local GC, 
however I'm still scanning the other thread contexts. The only 
limitation is that you can't move objects between threads, a 
pointer must remain in the thread that created it if the object 
is used in another thread.

https://github.com/etcimon/druntime/commit/78a2bca1356e0514c516e5261649ca0bf686b4cb
Jun 17 2015
parent reply "Laeeth Isharc" <nospamlaeeth laeethnospam.laeeth.com> writes:
On Wednesday, 17 June 2015 at 19:52:41 UTC, Etienne wrote:
https://github.com/etcimon/druntime/commit/7da3939637bd1642400dbed83e3b0ff2844386ac

 Only error was with a signal handler trying to allocate on 
 the GC. I think it'll be worth it for me to use this from now 
 on. There's no locking and possibly no stop the world.


 why 'possibly' ?


 I hadn't tested it when I wrote that message. I committed the 
 changes today and my tests were successful in a thread-local 
 GC, however I'm still scanning the other thread contexts. The 
 only limitation is that you can't move objects between threads, 
 a pointer must remain in the thread that created it if the 
 object is used in another thread.


 https://github.com/etcimon/druntime/commit/78a2bca1356e0514c516e5261649ca0bf686b4cb




I am no compiler/runtime guru, but it sounds like if it were 
possible to make this a swappable option for D this might be very 
important.  Is this true, and how much work is involved in making 
this industrial strength?
Jun 17 2015
next sibling parent reply "rsw0x" <anonymous anonymous.com> writes:
On Wednesday, 17 June 2015 at 20:19:45 UTC, Laeeth Isharc wrote:

 On Wednesday, 17 June 2015 at 19:52:41 UTC, Etienne wrote:
 https://github.com/etcimon/druntime/commit/7da3939637bd1642400dbed83e3b0ff2844386ac

 [...]


 why 'possibly' ?


 I hadn't tested it when I wrote that message. I committed the 
 changes today and my tests were successful in a thread-local 
 GC, however I'm still scanning the other thread contexts. The 
 only limitation is that you can't move objects between 
 threads, a pointer must remain in the thread that created it 
 if the object is used in another thread.


 https://github.com/etcimon/druntime/commit/78a2bca1356e0514c516e5261649ca0bf686b4cb




 I am no compiler/runtime guru, but it sounds like if it were 
 possible to make this a swappable option for D this might be 
 very important.  Is this true, and how much work is involved in 
 making this industrial strength?


that it falls apart as soon as you introduce any form of 
non-message passing threading and/or global variables?
if it was this easy, it would be done already.
Jun 17 2015
parent "Laeeth Isharc" <nospamlaeeth laeethnospam.laeeth.com> writes:
On Wednesday, 17 June 2015 at 21:28:46 UTC, rsw0x wrote:

 On Wednesday, 17 June 2015 at 20:19:45 UTC, Laeeth Isharc wrote:



 I am no compiler/runtime guru, but it sounds like if it were 
 possible to make this a swappable option for D this might be 
 very important.  Is this true, and how much work is involved 
 in making this industrial strength?


 that it falls apart as soon as you introduce any form of 
 non-message passing threading and/or global variables?


you can't make a special section for shared memory and global 
variables without incurring very high performance costs under 
this apporach?


 if it was this easy, it would be done already.


I personally have found this to be a poor heuristic in life.
Jun 17 2015
prev sibling parent reply "Etienne" <etcimon gmail.com> writes:
On Wednesday, 17 June 2015 at 20:19:45 UTC, Laeeth Isharc wrote:

 On Wednesday, 17 June 2015 at 19:52:41 UTC, Etienne wrote:
 https://github.com/etcimon/druntime/commit/7da3939637bd1642400dbed83e3b0ff2844386ac

 Only error was with a signal handler trying to allocate on 
 the GC. I think it'll be worth it for me to use this from 
 now on. There's no locking and possibly no stop the world.


 why 'possibly' ?


 I hadn't tested it when I wrote that message. I committed the 
 changes today and my tests were successful in a thread-local 
 GC, however I'm still scanning the other thread contexts. The 
 only limitation is that you can't move objects between 
 threads, a pointer must remain in the thread that created it 
 if the object is used in another thread.


 https://github.com/etcimon/druntime/commit/78a2bca1356e0514c516e5261649ca0bf686b4cb




 I am no compiler/runtime guru, but it sounds like if it were 
 possible to make this a swappable option for D this might be 
 very important.  Is this true, and how much work is involved in 
 making this industrial strength?


To me this is 100 times more stable simply because destructors 
can no longer be called from any thread. I'm also going to keep 
this implementation for its guarantee that destructors will be 
called. The absence of locking and stop the world makes it a very 
high performance GC.

Of course, the tradeoff is the absence of support in moving 
objects between threads. That's really an insignificant 
limitation compared to the above points. I don't expect it to be 
merged simply because nobody is familiar enough with this type of 
GC to have a say, except for what a GC shouldn't guarantee in 
general. And I'm really tired of people telling me what I can't 
or shouldn't do, when I'm brainstorming on this forum.
Jun 17 2015
parent reply "Laeeth Isharc" <nospamlaeeth laeethnospam.laeeth.com> writes:
On Wednesday, 17 June 2015 at 21:35:34 UTC, Etienne wrote:


 I am no compiler/runtime guru, but it sounds like if it were 
 possible to make this a swappable option for D this might be 
 very important.  Is this true, and how much work is involved 
 in making this industrial strength?


 To me this is 100 times more stable simply because destructors 
 can no longer be called from any thread. I'm also going to keep 
 this implementation for its guarantee that destructors will be 
 called. The absence of locking and stop the world makes it a 
 very high performance GC.

 Of course, the tradeoff is the absence of support in moving 
 objects between threads. That's really an insignificant 
 limitation compared to the above points. I don't expect it to 
 be merged simply because nobody is familiar enough with this 
 type of GC to have a say, except for what a GC shouldn't 
 guarantee in general. And I'm really tired of people telling me 
 what I can't or shouldn't do, when I'm brainstorming on this 
 forum.


Do you have any links to reading material on this type of GC?

I appreciate that it may in the general case be more stable, but 
in the particular implementation usually there is some distance 
to go from something good enough for personal use to something 
that others can depend on without having to understand the nitty 
gritty of the implementation.  So that is what I was asking...
Jun 17 2015
next sibling parent reply "Etienne Cimon" <etcimon gmail.com> writes:
On Wednesday, 17 June 2015 at 22:21:21 UTC, Laeeth Isharc wrote:

 On Wednesday, 17 June 2015 at 21:35:34 UTC, Etienne wrote:


 I am no compiler/runtime guru, but it sounds like if it were 
 possible to make this a swappable option for D this might be 
 very important.  Is this true, and how much work is involved 
 in making this industrial strength?


 To me this is 100 times more stable simply because destructors 
 can no longer be called from any thread. I'm also going to 
 keep this implementation for its guarantee that destructors 
 will be called. The absence of locking and stop the world 
 makes it a very high performance GC.

 Of course, the tradeoff is the absence of support in moving 
 objects between threads. That's really an insignificant 
 limitation compared to the above points. I don't expect it to 
 be merged simply because nobody is familiar enough with this 
 type of GC to have a say, except for what a GC shouldn't 
 guarantee in general. And I'm really tired of people telling 
 me what I can't or shouldn't do, when I'm brainstorming on 
 this forum.


 Do you have any links to reading material on this type of GC?

 I appreciate that it may in the general case be more stable, 
 but in the particular implementation usually there is some 
 distance to go from something good enough for personal use to 
 something that others can depend on without having to 
 understand the nitty gritty of the implementation.  So that is 
 what I was asking...


No reading material. It's straightforward from my train of 
though, your object will be collected if it's not either in the 
immediate global namespace, in the current thread's stack, in the 
current thread's local namespace or in objects that were 
allocated in the current thread's GC.

People saying that this will fail in basic message-passing or 
multi-threading are right. If you move your object in another 
thread's object even though the other thread's object is in the 
global namespace, you can't expect it to be tracked by the 
thread-local GC.

e.g. __gshared MyObj g_obj1;

Thread 1: g_obj1 = new MyObj;
Thread 2: g_obj1.obj2 = new MyObj;
Thread 3: write(g_obj1.obj2); <-- access violation (probably)

That's the "moving objects between threads" limitation I'm 
talking about. You need to use manual memory management and 
containers if you're going to use it for this purpose.
Jun 17 2015
next sibling parent reply "Etienne" <etcimon gmail.com> writes:
On Wednesday, 17 June 2015 at 22:35:13 UTC, Etienne Cimon wrote:

 On Wednesday, 17 June 2015 at 22:21:21 UTC, Laeeth Isharc wrote:

 [...]


 No reading material. It's straightforward from my train of 
 though, your object will be collected if it's not either in the 
 immediate global namespace, in the current thread's stack, in 
 the current thread's local namespace or in objects that were 
 allocated in the current thread's GC.

 [...]


I think a shared GC should also be available, but only for 
objects created with `new shared`. We have an entire type system 
to support that.
Jun 17 2015
parent "Laeeth Isharc" <nospamlaeeth laeethnospam.laeeth.com> writes:
On Thursday, 18 June 2015 at 00:54:35 UTC, Etienne wrote:

 On Wednesday, 17 June 2015 at 22:35:13 UTC, Etienne Cimon wrote:

 On Wednesday, 17 June 2015 at 22:21:21 UTC, Laeeth Isharc 
 wrote:

 [...]


 No reading material. It's straightforward from my train of 
 though, your object will be collected if it's not either in 
 the immediate global namespace, in the current thread's stack, 
 in the current thread's local namespace or in objects that 
 were allocated in the current thread's GC.

 [...]


 I think a shared GC should also be available, but only for 
 objects created with `new shared`. We have an entire type 
 system to support that.


Thank you for the colour.  Laeeth.
Jun 17 2015
prev sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Wed, 17 Jun 2015 22:35:12 +0000, Etienne Cimon wrote:


 e.g. __gshared MyObj g_obj1;
=20
 Thread 1: g_obj1 =3D new MyObj;
 Thread 2: g_obj1.obj2 =3D new MyObj;
 Thread 3: write(g_obj1.obj2); <-- access violation (probably)


so no way to anchor the whole object tree by assigning it to __gshared=20
root? sure, this will never make it into mainline.=
Jun 18 2015
parent reply "Etienne Cimon" <etcimon gmail.com> writes:
On Thursday, 18 June 2015 at 11:43:18 UTC, ketmar wrote:

 On Wed, 17 Jun 2015 22:35:12 +0000, Etienne Cimon wrote:


 e.g. __gshared MyObj g_obj1;
 
 Thread 1: g_obj1 = new MyObj;
 Thread 2: g_obj1.obj2 = new MyObj;
 Thread 3: write(g_obj1.obj2); <-- access violation (probably)


 so no way to anchor the whole object tree by assigning it to 
 __gshared root? sure, this will never make it into mainline.


__gshared is a little sketchy. We can have TLS GC by default by 
piping `new shared` to a shared GC. It's even safer, because then 
we have the type system helping out.

e.g. shared MyObj g_obj1;
Thread 1: g_obj1 = new shared MyObj;
Thread 2: g_obj1.obj2 = new shared MyObj;
Thread 3: write(g_obj1.obj2); <-- success!
Jun 18 2015
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Thu, 18 Jun 2015 12:08:10 +0000, Etienne Cimon wrote:


 On Thursday, 18 June 2015 at 11:43:18 UTC, ketmar wrote:

 On Wed, 17 Jun 2015 22:35:12 +0000, Etienne Cimon wrote:


 e.g. __gshared MyObj g_obj1;
=20
 Thread 1: g_obj1 =3D new MyObj;
 Thread 2: g_obj1.obj2 =3D new MyObj;
 Thread 3: write(g_obj1.obj2); <-- access violation (probably)


 so no way to anchor the whole object tree by assigning it to __gshared
 root? sure, this will never make it into mainline.


=20
 __gshared is a little sketchy. We can have TLS GC by default by piping
 `new shared` to a shared GC. It's even safer, because then we have the
 type system helping out.
=20
 e.g. shared MyObj g_obj1;
 Thread 1: g_obj1 =3D new shared MyObj;
 Thread 2: g_obj1.obj2 =3D new shared MyObj;
 Thread 3: write(g_obj1.obj2); <-- success!


besides, wouldn't it break `std.concurrency`? (ketmar have to dig into=20
code instead of asking...)=
Jun 18 2015
parent "Etienne" <etcimon gmail.com> writes:
On Thursday, 18 June 2015 at 14:11:42 UTC, ketmar wrote:

 On Thu, 18 Jun 2015 12:08:10 +0000, Etienne Cimon wrote:


 On Thursday, 18 June 2015 at 11:43:18 UTC, ketmar wrote:

 On Wed, 17 Jun 2015 22:35:12 +0000, Etienne Cimon wrote:


 e.g. __gshared MyObj g_obj1;
 
 Thread 1: g_obj1 = new MyObj;
 Thread 2: g_obj1.obj2 = new MyObj;
 Thread 3: write(g_obj1.obj2); <-- access violation (probably)


 so no way to anchor the whole object tree by assigning it to 
 __gshared root? sure, this will never make it into mainline.


 
 __gshared is a little sketchy. We can have TLS GC by default 
 by piping `new shared` to a shared GC. It's even safer, 
 because then we have the type system helping out.
 
 e.g. shared MyObj g_obj1;
 Thread 1: g_obj1 = new shared MyObj;
 Thread 2: g_obj1.obj2 = new shared MyObj;
 Thread 3: write(g_obj1.obj2); <-- success!


 besides, wouldn't it break `std.concurrency`? (ketmar have to 
 dig into code instead of asking...)


If it does, we fix it. I've never seen druntime/phobos mismatched 
versions (2.067 & 2.066) compiled together successfully, we 
shouldn't allow a necessary change to be blocked because of this 
alone.

As a workaround, you can also do __gshared MyObj g_obj1 = cast() 
new shared MyObj;
Jun 18 2015
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Wednesday, 17 June 2015 at 22:21:21 UTC, Laeeth Isharc wrote:

 Do you have any links to reading material on this type of GC?


Nim probably uses this sort of GC. Last I heard the plan was to 
introduce thread groups.
Jun 18 2015
prev sibling parent reply "Wyatt" <wyatt.epp gmail.com> writes:
On Wednesday, 17 June 2015 at 22:21:21 UTC, Laeeth Isharc wrote:

 Do you have any links to reading material on this type of GC?


This comes to mind, along with the citations:
http://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/local-gc.pdf

-Wyatt
Jun 18 2015
next sibling parent reply "Etienne" <etcimon gmail.com> writes:
On Thursday, 18 June 2015 at 15:09:46 UTC, Wyatt wrote:

 On Wednesday, 17 June 2015 at 22:21:21 UTC, Laeeth Isharc wrote:

 Do you have any links to reading material on this type of GC?


 This comes to mind, along with the citations:
 http://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/local-gc.pdf

 -Wyatt


That's exactly what we're talking about, but we do the work that 
was done by the globalisation policy by using the `new shared` 
keyword. We end up with a language that support the very 
foundation of what I'd say is the most solid Garbage Collection 
model.

"To  summarise  these  results,  it  seems  that
although  the  local  heap  collector  performs  many  fewer  
young-
generation collections and fewer synchronisations, the benefits 
are
offset  to  some  extent  by  the  extra  work  being  done  by  
the  col-
lector to maintain the global-heap invariant. "

"Globalising  the  entire  transitive  closure  was  on  average  
93%
slower, and results ranged from 7% faster to 577% slower."

"A garbage collector with local per-processor heaps can outperform
a stop-the-world parallel garbage collector in raw parallel 
through-
put, and exhibits more robust performance through having fewer
all-core synchronisations. "
Jun 18 2015
parent reply "Wyatt" <wyatt.epp gmail.com> writes:
On Thursday, 18 June 2015 at 15:19:19 UTC, Etienne wrote:

 On Thursday, 18 June 2015 at 15:09:46 UTC, Wyatt wrote:

 This comes to mind, along with the citations:
 http://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/local-gc.pdf

 -Wyatt


 That's exactly what we're talking about, but we do the work 
 that was done by the globalisation policy by using the `new 
 shared` keyword.


Maybe I misunderstood.  My interpretation was you were 
disallowing references from shared heap to local heap and 
obviating the whole globalisation problem.  No migrations; just 
explicit marking of global objects with "new shared". (Though I'm 
curious how you handle liveness of the global heap; it seems like 
that depends on information from each thread's allocator?)

-Wyatt
Jun 18 2015
parent "Etienne Cimon" <etcimon gmail.com> writes:
On Thursday, 18 June 2015 at 15:43:10 UTC, Wyatt wrote:

 On Thursday, 18 June 2015 at 15:19:19 UTC, Etienne wrote:

 On Thursday, 18 June 2015 at 15:09:46 UTC, Wyatt wrote:

 This comes to mind, along with the citations:
 http://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/local-gc.pdf

 -Wyatt


 That's exactly what we're talking about, but we do the work 
 that was done by the globalisation policy by using the `new 
 shared` keyword.


 Maybe I misunderstood.  My interpretation was you were 
 disallowing references from shared heap to local heap and


Exactly, unless you cast(shared) before you copy into the global 
heap - the local object must be kept trackable in the local heap 
or space. In any case, the shared is recursive and disallows 
this, so it's perfectly suitable to keep a shared GC consistent.


 obviating the whole globalisation problem.  No migrations; just 
 explicit marking of global objects with "new shared". (Though 
 I'm curious how you handle liveness of the global heap; it 
 seems like that depends on information from each thread's 
 allocator?)


The global heap and every thread will be scanned in stop the 
world setting just like the current GC. However, the idea is to 
use it less by making it exclusively for objects that were 
created with `new shared` `new immutable` `.idup` or `.sdup` (so 
that we can duplicate local objects into the shared GC 
conveniently)

In all of my security library (Botan) or TCP library (libasync) 
or even the vibe.d implementation at 
https://github.com/etcimon/vibe.d I haven't encountered a single 
instance of an object that was moved to another thread. The 
shared GC will be known to be very rarely useful.
Jun 18 2015
prev sibling parent "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
On Thursday, 18 June 2015 at 15:09:46 UTC, Wyatt wrote:

 On Wednesday, 17 June 2015 at 22:21:21 UTC, Laeeth Isharc wrote:

 Do you have any links to reading material on this type of GC?


 This comes to mind, along with the citations:
 http://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/local-gc.pdf

 -Wyatt


Thank you.

Laeeth
Jun 18 2015
prev sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 16 Jun 2015 16:28:53 +0000, Etienne wrote:


 To be fair, everything is bug prone until you understand them. GC
 finalization is done in a single lock, none of the memory is re-used, so
 objects can have their own "destroyed" flags and destroy eachother fine
 if the typeinfo issue isn't there.


this is an implementation detail. nothing guarantees that is will stay=20
like that even for one more commit. relying on this means that your code=20
is bugged and can break at any time, without warning. more than that, if=20
user pulled in another GC implementation, completely adhering to specs,=20
your code simply goes nuts, forcing the poor user to guess what's wrong.

so as other people already wrote, simply "don't do it". there will be no=20
way to do what you want here until GC requirements in specs will be=20
changed (and this is unlikely to happen).=
Jun 17 2015
parent reply "Etienne Cimon" <etcimon gmail.com> writes:
On Wednesday, 17 June 2015 at 14:19:33 UTC, ketmar wrote:

 this is an implementation detail. nothing guarantees that is 
 will stay like that even for one more commit. relying on this 
 means that your code is bugged and can break at any time, 
 without warning. more than that, if user pulled in another GC 
 implementation, completely adhering to specs, your code simply 
 goes nuts, forcing the poor user to guess what's wrong.

 so as other people already wrote, simply "don't do it". there 
 will be no way to do what you want here until GC requirements 
 in specs will be changed (and this is unlikely to happen).


If we can freeze the druntime interface anytime soon (while 
adding said support to the interface), the library can be subject 
to micro-optimizations by deferring the linking process and 
allowing it to be compiled and linked through dub.

This allows applications to use whatever GC is best for them. 
Parallel, precise, thread-local, you name it.

This being said, I know my use of the core.gc implementation 
carries no forward guarantees, so, I might end up dragging it 
along for a while and merging only certain parts of druntime in 
the future.
Jun 17 2015
parent ketmar <ketmar ketmar.no-ip.org> writes:
On Wed, 17 Jun 2015 14:47:12 +0000, Etienne Cimon wrote:


 This being said, I know my use of the core.gc implementation carries no
 forward guarantees, so, I might end up dragging it along for a while and
 merging only certain parts of druntime in the future.


sure, you can do that; i'm doing a similar thing with my Aliced. but the=20
price is that people with "vanilla" implementation can't use your code=20
anymore. ah, and sometimes you're forgetting about some prerequisites,=20
and your code stops working too. ;-)=
Jun 17 2015
prev sibling parent "Kagamin" <spam here.lot> writes:
On Tuesday, 16 June 2015 at 14:00:55 UTC, Etienne wrote:

 There is a bug regarding unordered object collection in the GC. 
 My finalizer accesses another GC-allocated object


Well, theoretically you can replace GC, which will allow you do 
that. But that's hardly practical, the behavior you're asking for 
is unsound anyway.
Jun 16 2015