www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Garbage collector collects live objects

reply "Ruslan Mullakhmetov" <public mullakh.com> writes:
Hi,

I experience very strange problem: GC somehow collects live 
objects.

I found it because i got segfaults. After debugging and tracing i 
found this is because of accessing not allocated memory.

I did the following checks:

- added to some class invariant check for access to suspicious 
members with assertion

assert(GC.addrOf(cast(void*)x) !is null);


where it fails DETERMINISTICALLY at some point

- printing address of allocated classes where i observe the 
following pattern

-> ctor
      check
      check
      check
<- dtor
      check (fails)

could anybody advice me with something? I got really frustrated 
by this strange behaviour which i can not fix right now.

key observations:
- it is deterministically behaviour (what gets me even more 
confused cause GC collections as far as i know runs from time to 
time)
- i do not play with pointers optimisation like hiding its in 
ints or floats.
- i operate with large uniformly distributed (video) data in 
memory where pointer like patterns may occur. but this is not the 
case cause (1) it brings at worst long living objects (2) input 
sequence constant but allocated pointers each run different.
Dec 09 2014
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/9/14 8:54 AM, Ruslan Mullakhmetov wrote:

 Hi,

 I experience very strange problem: GC somehow collects live objects.

 I found it because i got segfaults. After debugging and tracing i found
 this is because of accessing not allocated memory.

 I did the following checks:

 - added to some class invariant check for access to suspicious members
 with assertion

 assert(GC.addrOf(cast(void*)x) !is null);


 where it fails DETERMINISTICALLY at some point

 - printing address of allocated classes where i observe the following
 pattern

 -> ctor
       check
       check
       check
 <- dtor
       check (fails)

 could anybody advice me with something? I got really frustrated by this
 strange behaviour which i can not fix right now.

 key observations:
 - it is deterministically behaviour (what gets me even more confused
 cause GC collections as far as i know runs from time to time)
 - i do not play with pointers optimisation like hiding its in ints or
 floats.
 - i operate with large uniformly distributed (video) data in memory
 where pointer like patterns may occur. but this is not the case cause
 (1) it brings at worst long living objects (2) input sequence constant
 but allocated pointers each run different.


A random guess, since you haven't posted any code, are you accessing GC 
resources inside a destructor? If so, that is not guaranteed to work. A 
class destructor, or a destructor of a struct that is contained inside a 
class, can only be used to destroy NON-GC resources.

If you want more help, you need to post some code. Something that 
minimally causes the issue would be good.

-Steve
Dec 09 2014
parent reply "Ruslan Mullakhmetov" <nobody example.com> writes:
On Tuesday, 9 December 2014 at 14:23:06 UTC, Steven Schveighoffer 
wrote:

 On 12/9/14 8:54 AM, Ruslan Mullakhmetov wrote:

 Hi,

 I experience very strange problem: GC somehow collects live 
 objects.

 I found it because i got segfaults. After debugging and 
 tracing i found
 this is because of accessing not allocated memory.

 I did the following checks:

 - added to some class invariant check for access to suspicious 
 members
 with assertion

 assert(GC.addrOf(cast(void*)x) !is null);


 where it fails DETERMINISTICALLY at some point

 - printing address of allocated classes where i observe the 
 following
 pattern

 -> ctor
      check
      check
      check
 <- dtor
      check (fails)

 could anybody advice me with something? I got really 
 frustrated by this
 strange behaviour which i can not fix right now.

 key observations:
 - it is deterministically behaviour (what gets me even more 
 confused
 cause GC collections as far as i know runs from time to time)
 - i do not play with pointers optimisation like hiding its in 
 ints or
 floats.
 - i operate with large uniformly distributed (video) data in 
 memory
 where pointer like patterns may occur. but this is not the 
 case cause
 (1) it brings at worst long living objects (2) input sequence 
 constant
 but allocated pointers each run different.


 A random guess, since you haven't posted any code, are you 
 accessing GC resources inside a destructor? If so, that is not 
 guaranteed to work. A class destructor, or a destructor of a 
 struct that is contained inside a class, can only be used to 
 destroy NON-GC resources.

 If you want more help, you need to post some code. Something 
 that minimally causes the issue would be good.

 -Steve


No, there is no accessing GC resources in dtors.

the only usage of dtor in one class is

	~this()
	{
		_file.close();
	}

where _file is of type std.file.File

i'll try to extract problem to any observable source code but all 
my previous attempts lead to problem being diminish.
Dec 09 2014
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/9/14 9:52 AM, Ruslan Mullakhmetov wrote:

 No, there is no accessing GC resources in dtors.

 the only usage of dtor in one class is

      ~this()
      {
          _file.close();
      }

 where _file is of type std.file.File


That should work I think.


 i'll try to extract problem to any observable source code but all my
 previous attempts lead to problem being diminish.


Have you tried dustmite? https://github.com/CyberShadow/DustMite

-Steve
Dec 09 2014
prev sibling parent reply ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Tue, 09 Dec 2014 14:52:53 +0000
Ruslan Mullakhmetov via Digitalmars-d-learn
<digitalmars-d-learn puremagic.com> wrote:


 On Tuesday, 9 December 2014 at 14:23:06 UTC, Steven Schveighoffer=20
 wrote:

 On 12/9/14 8:54 AM, Ruslan Mullakhmetov wrote:

 Hi,

 I experience very strange problem: GC somehow collects live=20
 objects.

 I found it because i got segfaults. After debugging and=20
 tracing i found
 this is because of accessing not allocated memory.

 I did the following checks:

 - added to some class invariant check for access to suspicious=20
 members
 with assertion

 assert(GC.addrOf(cast(void*)x) !is null);


 where it fails DETERMINISTICALLY at some point

 - printing address of allocated classes where i observe the=20
 following
 pattern

 -> ctor
      check
      check
      check
 <- dtor
      check (fails)

 could anybody advice me with something? I got really=20
 frustrated by this
 strange behaviour which i can not fix right now.

 key observations:
 - it is deterministically behaviour (what gets me even more=20
 confused
 cause GC collections as far as i know runs from time to time)
 - i do not play with pointers optimisation like hiding its in=20
 ints or
 floats.
 - i operate with large uniformly distributed (video) data in=20
 memory
 where pointer like patterns may occur. but this is not the=20
 case cause
 (1) it brings at worst long living objects (2) input sequence=20
 constant
 but allocated pointers each run different.


 A random guess, since you haven't posted any code, are you=20
 accessing GC resources inside a destructor? If so, that is not=20
 guaranteed to work. A class destructor, or a destructor of a=20
 struct that is contained inside a class, can only be used to=20
 destroy NON-GC resources.

 If you want more help, you need to post some code. Something=20
 that minimally causes the issue would be good.

 -Steve


=20
 No, there is no accessing GC resources in dtors.
=20
 the only usage of dtor in one class is
=20
 	~this()
 	{
 		_file.close();
 	}
=20
 where _file is of type std.file.File
=20
 i'll try to extract problem to any observable source code but all=20
 my previous attempts lead to problem being diminish.


that file can be already finalized. please remember that `~this()` is
more a finalizer than destructor, and it's called on *dead* object.
here this means that any other object in your object (including
structs) can be already finalized at the time GC decides to call your
finalizer.

just avoid "destructors" unless you *really* need that. in your case
simply let GC finalize your File, don't try to help GC. this is not C++
(or any other language without GC) and "destructors" aren't destructing
anything at all. "destructors" must clean up the things that GC cannot
(malloc()'ed memory, for example), and nothing else.
Dec 09 2014
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/9/14 11:17 AM, ketmar via Digitalmars-d-learn wrote:

 On Tue, 09 Dec 2014 14:52:53 +0000
 Ruslan Mullakhmetov via Digitalmars-d-learn
 <digitalmars-d-learn puremagic.com> wrote:


 On Tuesday, 9 December 2014 at 14:23:06 UTC, Steven Schveighoffer
 wrote:

 On 12/9/14 8:54 AM, Ruslan Mullakhmetov wrote:

 Hi,

 I experience very strange problem: GC somehow collects live
 objects.

 I found it because i got segfaults. After debugging and
 tracing i found
 this is because of accessing not allocated memory.

 I did the following checks:

 - added to some class invariant check for access to suspicious
 members
 with assertion

 assert(GC.addrOf(cast(void*)x) !is null);


 where it fails DETERMINISTICALLY at some point

 - printing address of allocated classes where i observe the
 following
 pattern

 -> ctor
       check
       check
       check
 <- dtor
       check (fails)

 could anybody advice me with something? I got really
 frustrated by this
 strange behaviour which i can not fix right now.

 key observations:
 - it is deterministically behaviour (what gets me even more
 confused
 cause GC collections as far as i know runs from time to time)
 - i do not play with pointers optimisation like hiding its in
 ints or
 floats.
 - i operate with large uniformly distributed (video) data in
 memory
 where pointer like patterns may occur. but this is not the
 case cause
 (1) it brings at worst long living objects (2) input sequence
 constant
 but allocated pointers each run different.


 A random guess, since you haven't posted any code, are you
 accessing GC resources inside a destructor? If so, that is not
 guaranteed to work. A class destructor, or a destructor of a
 struct that is contained inside a class, can only be used to
 destroy NON-GC resources.

 If you want more help, you need to post some code. Something
 that minimally causes the issue would be good.

 -Steve


 No, there is no accessing GC resources in dtors.

 the only usage of dtor in one class is

 	~this()
 	{
 		_file.close();
 	}

 where _file is of type std.file.File

 i'll try to extract problem to any observable source code but all
 my previous attempts lead to problem being diminish.


 that file can be already finalized. please remember that `~this()` is
 more a finalizer than destructor, and it's called on *dead* object.
 here this means that any other object in your object (including
 structs) can be already finalized at the time GC decides to call your
 finalizer.


File is specially designed (although it's not perfect) to be able to 
close in the GC. Its ref-counted payload is placed on the C heap to 
allow access during finalization.

That being said, you actually don't need to write the above in the class 
finalizer, _file's destructor will automatically be called.


 just avoid "destructors" unless you *really* need that. in your case
 simply let GC finalize your File, don't try to help GC. this is not C++
 (or any other language without GC) and "destructors" aren't destructing
 anything at all. "destructors" must clean up the things that GC cannot
 (malloc()'ed memory, for example), and nothing else.


Good advice ;)

I would say other than library writers, nobody should ever write a class 
dtor.

-Steve
Dec 09 2014
next sibling parent reply "Ruslan Mullakhmetov" <public mullakh.com> writes:
On Tuesday, 9 December 2014 at 16:53:02 UTC, Steven Schveighoffer
wrote:

 On 12/9/14 11:17 AM, ketmar via Digitalmars-d-learn wrote:

 On Tue, 09 Dec 2014 14:52:53 +0000
 Ruslan Mullakhmetov via Digitalmars-d-learn
 <digitalmars-d-learn puremagic.com> wrote:


 On Tuesday, 9 December 2014 at 14:23:06 UTC, Steven 
 Schveighoffer
 wrote:

 On 12/9/14 8:54 AM, Ruslan Mullakhmetov wrote:

 Hi,

 I experience very strange problem: GC somehow collects live
 objects.

 I found it because i got segfaults. After debugging and
 tracing i found
 this is because of accessing not allocated memory.

 I did the following checks:

 - added to some class invariant check for access to 
 suspicious
 members
 with assertion

 assert(GC.addrOf(cast(void*)x) !is null);


 where it fails DETERMINISTICALLY at some point

 - printing address of allocated classes where i observe the
 following
 pattern

 -> ctor
      check
      check
      check
 <- dtor
      check (fails)

 could anybody advice me with something? I got really
 frustrated by this
 strange behaviour which i can not fix right now.

 key observations:
 - it is deterministically behaviour (what gets me even more
 confused
 cause GC collections as far as i know runs from time to 
 time)
 - i do not play with pointers optimisation like hiding its 
 in
 ints or
 floats.
 - i operate with large uniformly distributed (video) data in
 memory
 where pointer like patterns may occur. but this is not the
 case cause
 (1) it brings at worst long living objects (2) input 
 sequence
 constant
 but allocated pointers each run different.


 A random guess, since you haven't posted any code, are you
 accessing GC resources inside a destructor? If so, that is 
 not
 guaranteed to work. A class destructor, or a destructor of a
 struct that is contained inside a class, can only be used to
 destroy NON-GC resources.

 If you want more help, you need to post some code. Something
 that minimally causes the issue would be good.

 -Steve


 No, there is no accessing GC resources in dtors.

 the only usage of dtor in one class is

 	~this()
 	{
 		_file.close();
 	}

 where _file is of type std.file.File

 i'll try to extract problem to any observable source code but 
 all
 my previous attempts lead to problem being diminish.


 that file can be already finalized. please remember that 
 `~this()` is
 more a finalizer than destructor, and it's called on *dead* 
 object.
 here this means that any other object in your object (including
 structs) can be already finalized at the time GC decides to 
 call your
 finalizer.


 File is specially designed (although it's not perfect) to be 
 able to close in the GC. Its ref-counted payload is placed on 
 the C heap to allow access during finalization.

 That being said, you actually don't need to write the above in 
 the class finalizer, _file's destructor will automatically be 
 called.


 just avoid "destructors" unless you *really* need that. in 
 your case
 simply let GC finalize your File, don't try to help GC. this 
 is not C++
 (or any other language without GC) and "destructors" aren't 
 destructing
 anything at all. "destructors" must clean up the things that 
 GC cannot
 (malloc()'ed memory, for example), and nothing else.


 Good advice ;)

 I would say other than library writers, nobody should ever 
 write a class dtor.

 -Steve



thanks, I got it: either C++ or D dtors are minefield =)

but i still have no clue how to overcome GC =(
Dec 09 2014
parent reply ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Tue, 09 Dec 2014 17:18:44 +0000
Ruslan Mullakhmetov via Digitalmars-d-learn
<digitalmars-d-learn puremagic.com> wrote:


 thanks, I got it: either C++ or D dtors are minefield =3D)


and in D this is filed *made* of mines. ;-)


 but i still have no clue how to overcome GC =3D(


why do you want to fight with GC? most of the time GC is your friend.

are you trying to have predictable finalization? you don't have to
fight with GC in this case too, there are alot of other methods.
wrapper structs, `scoped!`, `RefCounted` and so on. you can have weak
references too (this is a hack, but it should work until we got
compacting (or precise?) GC ;-).
Dec 09 2014
parent reply "Ruslan Mullakhmetov" <public mullakh.com> writes:
On Wednesday, 10 December 2014 at 02:43:19 UTC, ketmar via 
Digitalmars-d-learn wrote:

 On Tue, 09 Dec 2014 17:18:44 +0000
 Ruslan Mullakhmetov via Digitalmars-d-learn
 <digitalmars-d-learn puremagic.com> wrote:

 but i still have no clue how to overcome GC =(


 why do you want to fight with GC? most of the time GC is your 
 friend.


see the topic: i got corruption when dereferencing object.
Dec 10 2014
parent ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 10 Dec 2014 08:32:12 +0000
Ruslan Mullakhmetov via Digitalmars-d-learn
<digitalmars-d-learn puremagic.com> wrote:


 On Wednesday, 10 December 2014 at 02:43:19 UTC, ketmar via=20
 Digitalmars-d-learn wrote:

 On Tue, 09 Dec 2014 17:18:44 +0000
 Ruslan Mullakhmetov via Digitalmars-d-learn
 <digitalmars-d-learn puremagic.com> wrote:

 but i still have no clue how to overcome GC =3D(


 why do you want to fight with GC? most of the time GC is your=20
 friend.


=20
 see the topic: i got corruption when dereferencing object.


that is easily fixable: just stop dereferencing it! ;-)
Dec 10 2014
prev sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 12/09/2014 08:53 AM, Steven Schveighoffer wrote:


 On 12/9/14 11:17 AM, ketmar via Digitalmars-d-learn wrote:



 that file can be already finalized. please remember that `~this()` is
 more a finalizer than destructor, and it's called on *dead* object.




Agreed: D has a terminology issue here, what we call a "class 
destructor" is a "class finalizer". I have added D to the Wikipedia article:

   http://en.wikipedia.org/wiki/Finalizer

Now I want to improve some of my chapters.


 here this means that any other object in your object (including
 structs) can be already finalized at the time GC decides to call your
 finalizer.




Although I know the fact above, I think ketmar makes a distinction (that 
is new to me) that a finalizer is a function that does not reference 
class members but a destructor does (or safely can).


 File is specially designed (although it's not perfect) to be able to
 close in the GC. Its ref-counted payload is placed on the C heap to
 allow access during finalization.

 That being said, you actually don't need to write the above in the class
 finalizer, _file's destructor will automatically be called.


 just avoid "destructors" unless you *really* need that. in your case
 simply let GC finalize your File, don't try to help GC. this is not C++
 (or any other language without GC) and "destructors" aren't destructing
 anything at all. "destructors" must clean up the things that GC cannot
 (malloc()'ed memory, for example), and nothing else.


 Good advice ;)

 I would say other than library writers, nobody should ever write a class
 dtor.

 -Steve


Can somebody elaborate on that guideline please. Given a case where 
runtime polymorphism is needed, so that we have to use classes, does the 
guideline simply mean that "arrange for manual cleanup" or is there more 
in that guideline? I am confused about the "library writers" part. :)

Thank you,
Ali
Mar 25 2015
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/25/15 9:51 AM, Ali Çehreli wrote:

 On 12/09/2014 08:53 AM, Steven Schveighoffer wrote:

  > On 12/9/14 11:17 AM, ketmar via Digitalmars-d-learn wrote:

  >> that file can be already finalized. please remember that `~this()` is
  >> more a finalizer than destructor, and it's called on *dead* object.

 Agreed: D has a terminology issue here, what we call a "class
 destructor" is a "class finalizer". I have added D to the Wikipedia
 article:

    http://en.wikipedia.org/wiki/Finalizer

 Now I want to improve some of my chapters.

  >> here this means that any other object in your object (including
  >> structs) can be already finalized at the time GC decides to call your
  >> finalizer.

 Although I know the fact above, I think ketmar makes a distinction (that
 is new to me) that a finalizer is a function that does not reference
 class members but a destructor does (or safely can).


Think of it this way -- a D destructor can access data from it's own 
object. This means it CAN access its own members that are structs or 
value types. It should NOT access referenced data, unless that data is 
allocated outside the GC.

In Tango, there was a finalizer and a destructor. The finalizer was 
called from the GC. The destructor was called when using the 'delete' 
operator explicitly (and I think it also called the finalizer 
afterward). Having this informational structure helps to know where to 
put what cleanup code. I would love it if druntime added this feature, 
or something similar.


  > File is specially designed (although it's not perfect) to be able to
  > close in the GC. Its ref-counted payload is placed on the C heap to
  > allow access during finalization.
  >
  > That being said, you actually don't need to write the above in the class
  > finalizer, _file's destructor will automatically be called.
  >
  >> just avoid "destructors" unless you *really* need that. in your case
  >> simply let GC finalize your File, don't try to help GC. this is not C++
  >> (or any other language without GC) and "destructors" aren't destructing
  >> anything at all. "destructors" must clean up the things that GC cannot
  >> (malloc()'ed memory, for example), and nothing else.
  >>
  >
  > Good advice ;)
  >
  > I would say other than library writers, nobody should ever write a class
  > dtor.
  >
  > -Steve

 Can somebody elaborate on that guideline please. Given a case where
 runtime polymorphism is needed, so that we have to use classes, does the
 guideline simply mean that "arrange for manual cleanup" or is there more
 in that guideline? I am confused about the "library writers" part. :)


Destructors should only be used to clean non-GC resources (and in terms 
of completeness, you should implement such a destructor). It is 
important to remember that the GC may NEVER get around to calling your 
destructor. This means you cannot depend on it releasing non-GC 
resources in a timely manner.

A good example is a file descriptor. If you have an object that contains 
a file descriptor you SHOULD have a destructor which closes the file 
descriptor if not already closed. However, you should NOT technically 
rely on it being called in a timely manner. For cases where you can do 
it synchronously, use a specific method to close the file descriptor.

There is a school of thought that says it is an error to rely on the GC 
to destroy the FD, so why even have the destructor. But I see no reason 
to leak the FD out of spite.

This is why I say it should be only library writers -- they are the ones 
creating wrapper objects and using low level operations.

Of course, this isn't a hard and fast rule.

-Steve
Mar 25 2015
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
It may happen if only reference to an object is stored in memory 
block marked as data-only (using ubyte[] for a buffer is probably 
most common reason I have encountered)
Dec 09 2014
parent reply "Ruslan Mullakhmetov" <public mullakh.com> writes:
On Tuesday, 9 December 2014 at 16:13:25 UTC, Dicebot wrote:

 It may happen if only reference to an object is stored in 
 memory block marked as data-only (using ubyte[] for a buffer is 
 probably most common reason I have encountered)


Thanks for interesting hypothesis, but that's not the issue.

innocent though collected objects are living in D array MyClass[] 
which are living in assoc array as value.

i checked attributes for GC block holding this array:

```
FINALIZE NO_SCAN NO_MOVE APPENDABLE NO_INTERIOR
```

I really doubting about NO_INTERIOR. can anybody confirm me that 
is's working with array slicing which i heavily use?


also i found that block size is quite small

<pre>
                 array: [100A2FD00, 100A2F700, 100A33B80, 
100A33500, 100A3FE80, 100A3F980, 100A3F400, 100A72600, 100A7DF80, 
100A7DA80, 100A7D500]
		array ptr: 100A72580 root: 100A72580:128 attr: FINALIZE NO_SCAN 
NO_MOVE APPENDABLE NO_INTERIOR
		[100985A00] keys: [1] as: 1 au: 100A2FD00
		[100985A00] keys: [1] as: 1 au: 100A2F700
		[100985A00] keys: [1] as: 1 au: 100A33B80
</pre>

array holds 11 64bit pointers but it's block size is only 128 
bytes < 11 * 64 = 704 bytes. what's wrong with this arithmetics?
Dec 09 2014
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/9/14 12:40 PM, Ruslan Mullakhmetov wrote:

 On Tuesday, 9 December 2014 at 16:13:25 UTC, Dicebot wrote:

 It may happen if only reference to an object is stored in memory block
 marked as data-only (using ubyte[] for a buffer is probably most
 common reason I have encountered)


 Thanks for interesting hypothesis, but that's not the issue.

 innocent though collected objects are living in D array MyClass[] which
 are living in assoc array as value.

 i checked attributes for GC block holding this array:

 ```
 FINALIZE NO_SCAN NO_MOVE APPENDABLE NO_INTERIOR
 ```


That does not sound right at all. No block should ever have both 
FINALIZE (reserved for objects only) and APPENDABLE (reserved for arrays 
only).


 I really doubting about NO_INTERIOR. can anybody confirm me that is's
 working with array slicing which i heavily use?


 also i found that block size is quite small

 <pre>
                  array: [100A2FD00, 100A2F700, 100A33B80, 100A33500,
 100A3FE80, 100A3F980, 100A3F400, 100A72600, 100A7DF80, 100A7DA80,
 100A7D500]
          array ptr: 100A72580 root: 100A72580:128 attr: FINALIZE NO_SCAN
 NO_MOVE APPENDABLE NO_INTERIOR
          [100985A00] keys: [1] as: 1 au: 100A2FD00
          [100985A00] keys: [1] as: 1 au: 100A2F700
          [100985A00] keys: [1] as: 1 au: 100A33B80
 </pre>

 array holds 11 64bit pointers but it's block size is only 128 bytes < 11
 * 64 = 704 bytes. what's wrong with this arithmetics?


I think there is something you are missing, or something is very 
corrupt. Can you show the code that prints this?

-Steve
Dec 09 2014
next sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 12/09/2014 11:56 AM, Steven Schveighoffer wrote:


 i checked attributes for GC block holding this array:

 ```
 FINALIZE NO_SCAN NO_MOVE APPENDABLE NO_INTERIOR
 ```


 That does not sound right at all. No block should ever have both
 FINALIZE (reserved for objects only) and APPENDABLE (reserved for arrays
 only).


FINALIZE and APPENDABLE together sounds like "an array that holds class 
objects."

I think I get it as I write this: Do we mean that the array should 
always hold class references and the class objects should live on other 
blocks? If so, the memory block for the objects can be marked as 
FINALIZE? What block should be APPENDABLE?

Of course, this may be all in the documentation but I can't understand 
it. ;) Here is what is says for FINALIZE: "Finalize the data in this 
block on collect." (I will study that part a little more. :p)



Ali
Dec 09 2014
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/9/14 3:18 PM, Ali Çehreli wrote:

 On 12/09/2014 11:56 AM, Steven Schveighoffer wrote:

  >> i checked attributes for GC block holding this array:
  >>
  >> ```
  >> FINALIZE NO_SCAN NO_MOVE APPENDABLE NO_INTERIOR
  >> ```
  >>
  >
  > That does not sound right at all. No block should ever have both
  > FINALIZE (reserved for objects only) and APPENDABLE (reserved for arrays
  > only).

 FINALIZE and APPENDABLE together sounds like "an array that holds class
 objects."

 I think I get it as I write this: Do we mean that the array should
 always hold class references and the class objects should live on other
 blocks? If so, the memory block for the objects can be marked as
 FINALIZE?


Yes, that's exactly right. A class is never allocated "inline" inside 
another object or an array.


 What block should be APPENDABLE?


The array of class references can be APPENDABLE.


 Of course, this may be all in the documentation but I can't understand
 it. ;) Here is what is says for FINALIZE: "Finalize the data in this
 block on collect." (I will study that part a little more. :p)




In truth, the code expects the block then to have a ClassInfo pointer at 
the beginning of the block.

See here:

https://github.com/D-Programming-Language/druntime/blob/master/src/rt/lifetime.d#L1225

-Steve
Dec 09 2014
prev sibling next sibling parent reply "Ruslan Mullakhmetov" <public mullakh.com> writes:
On Tuesday, 9 December 2014 at 19:56:30 UTC, Steven Schveighoffer 
wrote:

 On 12/9/14 12:40 PM, Ruslan Mullakhmetov wrote:

 On Tuesday, 9 December 2014 at 16:13:25 UTC, Dicebot wrote:
 i checked attributes for GC block holding this array:

 FINALIZE NO_SCAN NO_MOVE APPENDABLE NO_INTERIOR


 That does not sound right at all. No block should ever have 
 both FINALIZE (reserved for objects only) and APPENDABLE 
 (reserved for arrays only).


 also i found that block size is quite small

 <pre>
                 array: [100A2FD00, 100A2F700, 100A33B80, 
 100A33500,
 100A3FE80, 100A3F980, 100A3F400, 100A72600, 100A7DF80, 
 100A7DA80,
 100A7D500]
         array ptr: 100A72580 root: 100A72580:128 attr: 
 FINALIZE NO_SCAN
 NO_MOVE APPENDABLE NO_INTERIOR
         [100985A00] keys: [1] as: 1 au: 100A2FD00
         [100985A00] keys: [1] as: 1 au: 100A2F700
         [100985A00] keys: [1] as: 1 au: 100A33B80
 </pre>

 array holds 11 64bit pointers but it's block size is only 128 
 bytes < 11
 * 64 = 704 bytes. what's wrong with this arithmetics?


 I think there is something you are missing, or something is 
 very corrupt. Can you show the code that prints this?

 -Steve


here the piece of code i used to output this value

http://pastebin.com/cQf9Nghp

StreamIndex is ubyte
AccessUnit is some class
Dec 09 2014
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/9/14 3:24 PM, Ruslan Mullakhmetov wrote:

 On Tuesday, 9 December 2014 at 19:56:30 UTC, Steven Schveighoffer wrote:

 On 12/9/14 12:40 PM, Ruslan Mullakhmetov wrote:

 On Tuesday, 9 December 2014 at 16:13:25 UTC, Dicebot wrote:
 i checked attributes for GC block holding this array:

 FINALIZE NO_SCAN NO_MOVE APPENDABLE NO_INTERIOR


 That does not sound right at all. No block should ever have both
 FINALIZE (reserved for objects only) and APPENDABLE (reserved for
 arrays only).


 here the piece of code i used to output this value

 http://pastebin.com/cQf9Nghp


I literally had to compile this for myself before I saw the error:

if(bi && k) => if(bi & k)

Though that doesn't explain all the issues you reported. I'm curious 
what the output is after that though...

-Steve
Dec 09 2014
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/9/14 2:56 PM, Steven Schveighoffer wrote:

 On 12/9/14 12:40 PM, Ruslan Mullakhmetov wrote:



 array holds 11 64bit pointers but it's block size is only 128 bytes < 11
 * 64 = 704 bytes. what's wrong with this arithmetics?




Hah, just realized what's wrong. It's not 64 *bytes* per pointer, it's 
64 *bits*. So 8 bytes.

11 * 8 == 88.

Starting to sound more and more normal...

-Steve
Dec 09 2014
parent reply "Ruslan Mullakhmetov" <public mullakh.com> writes:
On Tuesday, 9 December 2014 at 21:38:57 UTC, Steven Schveighoffer 
wrote:

 On 12/9/14 2:56 PM, Steven Schveighoffer wrote:

 On 12/9/14 12:40 PM, Ruslan Mullakhmetov wrote:



 array holds 11 64bit pointers but it's block size is only 128 
 bytes < 11
 * 64 = 704 bytes. what's wrong with this arithmetics?




 Hah, just realized what's wrong. It's not 64 *bytes* per 
 pointer, it's 64 *bits*. So 8 bytes.

 11 * 8 == 88.

 Starting to sound more and more normal...

 -Steve



yes. that was the mistake. also after fixing bug in Blk 
Attributes printing i got more reasonable attrs

for object blk: FINALIZE
for array of objects blk: NO_SCAN APPENDABLE

this is sound good except for NO_SCAN.


I did simple test file in which allocate array of Foo objects 
(http://dpaste.dzfl.pl/89ab00a897f6)

there i see blk attrs only APPENDABLE without NO_SCAN.

as far as i understand GC will not scan this array for references 
and those if the only reference to object is stored in this array 
will not see it, those assume this object as **not** referenced 
and collects it, am i right?

the other question why this happens... try to debug more.
Dec 10 2014
parent reply "Ruslan Mullakhmetov" <public mullakh.com> writes:
On Wednesday, 10 December 2014 at 08:46:12 UTC, Ruslan 
Mullakhmetov wrote:

 yes. that was the mistake. also after fixing bug in Blk 
 Attributes printing i got more reasonable attrs

 for object blk: FINALIZE
 for array of objects blk: NO_SCAN APPENDABLE

 this is sound good except for NO_SCAN.

 ...
 the other question why this happens... try to debug more.


I've done more dubugging.

what i've found:

initially array blk has only attrs APPENDABLE, but after some 
time this blk is shrinked and reallocated (moved) and then 
NO_SCAN attr appears.


here the output of my extended logs:

--------
before tag: 1 len: 2 ptr: 103DD9058 root: 103DD8000:8192 attr: 
APPENDABLE
after tag: 1 len: 3 ptr: 103A21DD0 root: 103A21DC0:64 attr: 
NO_SCAN APPENDABLE
--------

this is produced by the following code

http://dpaste.dzfl.pl/0c6dc16270a1

so in a nutshell after appending to array via ~= operator blk 
attrs changed from APPENDABLE to NO_SCAN APPENDABLE which cause 
the problem.

why and how this happens? can anybody explain it to me?
Dec 10 2014
next sibling parent reply ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 10 Dec 2014 12:52:22 +0000
Ruslan Mullakhmetov via Digitalmars-d-learn
<digitalmars-d-learn puremagic.com> wrote:


 On Wednesday, 10 December 2014 at 08:46:12 UTC, Ruslan=20
 Mullakhmetov wrote:

 yes. that was the mistake. also after fixing bug in Blk=20
 Attributes printing i got more reasonable attrs

 for object blk: FINALIZE
 for array of objects blk: NO_SCAN APPENDABLE

 this is sound good except for NO_SCAN.

 ...
 the other question why this happens... try to debug more.


=20
 I've done more dubugging.
=20
 what i've found:
=20
 initially array blk has only attrs APPENDABLE, but after some=20
 time this blk is shrinked and reallocated (moved) and then=20
 NO_SCAN attr appears.
=20
=20
 here the output of my extended logs:
=20
 --------
 before tag: 1 len: 2 ptr: 103DD9058 root: 103DD8000:8192 attr:=20
 APPENDABLE
 after tag: 1 len: 3 ptr: 103A21DD0 root: 103A21DC0:64 attr:=20
 NO_SCAN APPENDABLE
 --------
=20
 this is produced by the following code
=20
 http://dpaste.dzfl.pl/0c6dc16270a1
=20
 so in a nutshell after appending to array via ~=3D operator blk=20
 attrs changed from APPENDABLE to NO_SCAN APPENDABLE which cause=20
 the problem.
=20
 why and how this happens? can anybody explain it to me?
=20


can you give us a minified code that causes this behavior?
Dec 10 2014
parent "Ruslan Mullakhmetov" <public mullakh.com> writes:
On Wednesday, 10 December 2014 at 13:00:45 UTC, ketmar via 
Digitalmars-d-learn wrote:

 can you give us a minified code that causes this behavior?


see previous post. the problem vanish if i try to extract it.
Dec 10 2014
prev sibling next sibling parent reply "Ruslan Mullakhmetov" <public mullakh.com> writes:
On Wednesday, 10 December 2014 at 12:52:24 UTC, Ruslan 
Mullakhmetov wrote:

 why and how this happens? can anybody explain it to me?


I tried to extract this and saw NO NO_SCAN attrs after moving blk:


the following piece of output produced by 
http://dpaste.dzfl.pl/6f773e17de92

len: 99996 ptr: 109DF0010 root: 109DF0000:1048576 attr: APPENDABLE
len: 99997 ptr: 109DF0010 root: 109DF0000:1048576 attr: APPENDABLE
len: 99998 ptr: 109DF0010 root: 109DF0000:1048576 attr: APPENDABLE
len: 99999 ptr: 109DF0010 root: 109DF0000:1048576 attr: APPENDABLE
len: 100000 ptr: 109DF0010 root: 109DF0000:1048576 attr: 
APPENDABLE
--- shrinked --
len: 1 ptr: 109EB3508 root: 109DF0000:1048576 attr: APPENDABLE
len: 2 ptr: 109EB3508 root: 109DF0000:1048576 attr: APPENDABLE
len: 3 ptr: 109EB3508 root: 109DF0000:1048576 attr: APPENDABLE
len: 4 ptr: 109EB3508 root: 109DF0000:1048576 attr: APPENDABLE
len: 5 ptr: 109EB3508 root: 109DF0000:1048576 attr: APPENDABLE
len: 6 ptr: 109F60640 root: 109F60640:64 attr: APPENDABLE
len: 7 ptr: 109F60640 root: 109F60640:64 attr: APPENDABLE
Dec 10 2014
parent ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 10 Dec 2014 13:03:21 +0000
Ruslan Mullakhmetov via Digitalmars-d-learn
<digitalmars-d-learn puremagic.com> wrote:


 On Wednesday, 10 December 2014 at 12:52:24 UTC, Ruslan=20
 Mullakhmetov wrote:

 why and how this happens? can anybody explain it to me?


=20
 I tried to extract this and saw NO NO_SCAN attrs after moving blk:


i strongly believe that you have some strange casts buried somewhere in
the depth of the complex code, or something similar.

maybe trackallocs.d from https://bitbucket.org/infognition/dstuff can
help to track (re)allocations. you can modify source to dump the flags.
Dec 10 2014
prev sibling next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/10/14 7:52 AM, Ruslan Mullakhmetov wrote:

 On Wednesday, 10 December 2014 at 08:46:12 UTC, Ruslan Mullakhmetov wrote:

 yes. that was the mistake. also after fixing bug in Blk Attributes
 printing i got more reasonable attrs

 for object blk: FINALIZE
 for array of objects blk: NO_SCAN APPENDABLE

 this is sound good except for NO_SCAN.

 ...
 the other question why this happens... try to debug more.


 I've done more dubugging.

 what i've found:

 initially array blk has only attrs APPENDABLE, but after some time this
 blk is shrinked and reallocated (moved) and then NO_SCAN attr appears.


 here the output of my extended logs:

 --------
 before tag: 1 len: 2 ptr: 103DD9058 root: 103DD8000:8192 attr: APPENDABLE
 after tag: 1 len: 3 ptr: 103A21DD0 root: 103A21DC0:64 attr: NO_SCAN
 APPENDABLE
 --------

 this is produced by the following code

 http://dpaste.dzfl.pl/0c6dc16270a1

 so in a nutshell after appending to array via ~= operator blk attrs
 changed from APPENDABLE to NO_SCAN APPENDABLE which cause the problem.

 why and how this happens? can anybody explain it to me?


I have an idea of what is happening, I will do some testing. Thanks for 
debugging this so far, this is useful info.

This is *definitely* a bug, if the code you gave is what caused that 
output. Appending should not add the NO_SCAN tag.

-Steve
Dec 11 2014
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/10/14 7:52 AM, Ruslan Mullakhmetov wrote:

 On Wednesday, 10 December 2014 at 08:46:12 UTC, Ruslan Mullakhmetov wrote:

 yes. that was the mistake. also after fixing bug in Blk Attributes
 printing i got more reasonable attrs

 for object blk: FINALIZE
 for array of objects blk: NO_SCAN APPENDABLE

 this is sound good except for NO_SCAN.

 ...
 the other question why this happens... try to debug more.


 I've done more dubugging.

 what i've found:

 initially array blk has only attrs APPENDABLE, but after some time this
 blk is shrinked and reallocated (moved) and then NO_SCAN attr appears.


 here the output of my extended logs:

 --------
 before tag: 1 len: 2 ptr: 103DD9058 root: 103DD8000:8192 attr: APPENDABLE
 after tag: 1 len: 3 ptr: 103A21DD0 root: 103A21DC0:64 attr: NO_SCAN
 APPENDABLE
 --------


My analysis so far:

1. The before/after makes sense except for the attribute and the offset. 
A realloc into a 64-byte block should NOT cause an offset of 16 bytes. I 
have found why it's happening, which is a bug, but not one that should 
cause the problem of setting the noscan bit (will file an issue on that).
2. In the array append code, the block attributes are obtained via 
GC.query, which has this code for getting the attributes:

https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L1792

Quoting from that function:

// reset the offset to the base pointer, otherwise the bits
// are the bits for the pointer, which may be garbage
offset = cast(size_t)(info.base - pool.baseAddr);
info.attr = getBits(pool, cast(size_t)(offset >> pool.shiftBy));

Which should get the correct bits. I suspected there was an issue with 
getting the wrong bits, but this code looks correct.

3. The runtime caches the block info for thread local data for append 
speed. A potential issue is that the attributes are cached from a 
previous use for that block, but the GC (and the runtime itself) SHOULD 
clear that cache entry when that block is freed, avoiding this issue. A 
potential way to check this is to assert in a debug build of druntime 
that the cached block info always equals the actual block info. Are you 
able to build a debug version of druntime to test this? I can give you 
the changes you should make. This would explain the great difficulty in 
reproducing the issue.

4. If your code is multi-threaded, but using __gshared, it can make the 
cache incorrect. Are you doing this?

But the cache is really the only possible place I can see where the bits 
are set incorrectly, given that you just verified the bits are correct 
before the append.

Can you just list the version of the compiler you are using? I want to 
make sure this isn't an issue that has already been fixed.

-Steve
Dec 11 2014
parent reply "Ruslan Mullakhmetov" <public mullakh.com> writes:
On Thursday, 11 December 2014 at 18:36:59 UTC, Steven 
Schveighoffer wrote:

 My analysis so far:

 2. In the array append code, the block attributes are obtained 
 via GC.query, which has this code for getting the attributes:

 https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L1792

 Quoting from that function:

 // reset the offset to the base pointer, otherwise the bits
 // are the bits for the pointer, which may be garbage
 offset = cast(size_t)(info.base - pool.baseAddr);
 info.attr = getBits(pool, cast(size_t)(offset >> pool.shiftBy));

 Which should get the correct bits. I suspected there was an 
 issue with getting the wrong bits, but this code looks correct.

 3. The runtime caches the block info for thread local data for 
 append speed. A potential issue is that the attributes are 
 cached from a previous use for that block, but the GC (and the 
 runtime itself) SHOULD clear that cache entry when that block 
 is freed, avoiding this issue. A potential way to check this is 
 to assert in a debug build of druntime that the cached block 
 info always equals the actual block info. Are you able to build 
 a debug version of druntime to test this? I can give you the 
 changes you should make. This would explain the great 
 difficulty in reproducing the issue.


I will try to build debug version of dmd compiler and check the 
issue.


 4. If your code is multi-threaded, but using __gshared, it can 
 make the cache incorrect. Are you doing this?


the app is multi-threaded via std.concurrency.

there is only one known to me place where __gshared is used: 
logging library (checked by searching through whole source tree). 
make stub for this lib and try, so identify whether cache 
invalidated by _gshared or not.


 But the cache is really the only possible place I can see where 
 the bits are set incorrectly, given that you just verified the 
 bits are correct before the append.

 Can you just list the version of the compiler you are using? I 
 want to make sure this isn't an issue that has already been 
 fixed.


the last. first of all i updated whole toolchain (dmd, dub).

$ dmd
DMD64 D Compiler v2.066.1


 -Steve


I started looking druntime and dmd source code myself before i 
checked the thread (thsnks for your help and feedback) and i have 
some questions. could you explain to me something?

i_m looking here 
https://github.com/D-Programming-Language/druntime/blob/v2.066.1/src/rt/lifetime.d#L591

-------

auto size = ti.next.tsize;

why `next`? it can be even null if this is last TypeInfo in the 
linked list.

---------

btw, i used suggested trackallocs.d and GC defenetely receives 
NO_SCAN

before tag: 1 len: 2 ptr: 103A78058 root: 103A77000:8192 attr: 
APPENDABLE
gc_qalloc(41, NO_SCAN APPENDABLE ) cc: 29106 asz: 10152603, ti: 
null ret: BlkInfo_(104423800, 64, 10)
after tag: 1 len: 3 ptr: 104423810 root: 104423800:64 attr: 
NO_SCAN APPENDABLE
Dec 12 2014
next sibling parent "Ruslan Mullakhmetov" <public mullakh.com> writes:
On Friday, 12 December 2014 at 12:53:00 UTC, Ruslan Mullakhmetov 
wrote:

 On Thursday, 11 December 2014 at 18:36:59 UTC, Steven 
 Schveighoffer wrote:

 My analysis so far:

 4. If your code is multi-threaded, but using __gshared, it can 
 make the cache incorrect. Are you doing this?


 the app is multi-threaded via std.concurrency.

 there is only one known to me place where __gshared is used: 
 logging library (checked by searching through whole source 
 tree). make stub for this lib and try, so identify whether 
 cache invalidated by _gshared or not.



removing __gshared seems does not helped.
Dec 12 2014
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/12/14 7:52 AM, Ruslan Mullakhmetov wrote:

 On Thursday, 11 December 2014 at 18:36:59 UTC, Steven Schveighoffer wrote:

 My analysis so far:

 2. In the array append code, the block attributes are obtained via
 GC.query, which has this code for getting the attributes:

 https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L1792


 Quoting from that function:

 // reset the offset to the base pointer, otherwise the bits
 // are the bits for the pointer, which may be garbage
 offset = cast(size_t)(info.base - pool.baseAddr);
 info.attr = getBits(pool, cast(size_t)(offset >> pool.shiftBy));

 Which should get the correct bits. I suspected there was an issue with
 getting the wrong bits, but this code looks correct.

 3. The runtime caches the block info for thread local data for append
 speed. A potential issue is that the attributes are cached from a
 previous use for that block, but the GC (and the runtime itself)
 SHOULD clear that cache entry when that block is freed, avoiding this
 issue. A potential way to check this is to assert in a debug build of
 druntime that the cached block info always equals the actual block
 info. Are you able to build a debug version of druntime to test this?
 I can give you the changes you should make. This would explain the
 great difficulty in reproducing the issue.


 I will try to build debug version of dmd compiler and check the issue.


A debug version of compiler is not necessary, not even a debug version 
of phobos, just druntime. But it's not going to matter yet, because I 
need to give you the asserts to put in there. I just wanted to know if 
you needed help doing it.


 4. If your code is multi-threaded, but using __gshared, it can make
 the cache incorrect. Are you doing this?


 the app is multi-threaded via std.concurrency.


This should be OK, you should not be able to share data that is not 
marked as shared.


 there is only one known to me place where __gshared is used: logging
 library (checked by searching through whole source tree). make stub for
 this lib and try, so identify whether cache invalidated by _gshared or not.


Here is where it might occur:

1. Due to shared data having typeinfo attached to it that it is actually 
shared, the runtime takes advantage of that. We can use a lock-free 
cache that is thread-local for anything not marked as shared, because 
nothing outside the thread can access that data.
2. __gshared gets around this because it is not marked as shared by the 
compiler. This means, if you, for instance, appended to a __gshared 
array, the runtime would treat it like a thread-local array. If you did 
this from multiple threads, the cache may be invalid in one or more of them.
3. Actual 'shared' arrays are not permitted to use the cache, so they 
should not have this issue.

I see that you removed the only instance of __gshared and it did not 
help. That at least rules that out.


 But the cache is really the only possible place I can see where the
 bits are set incorrectly, given that you just verified the bits are
 correct before the append.

 Can you just list the version of the compiler you are using? I want to
 make sure this isn't an issue that has already been fixed.


 the last. first of all i updated whole toolchain (dmd, dub).

 $ dmd
 DMD64 D Compiler v2.066.1


Thanks, this at least gives me a baseline to know what to test and debug 
with. I do not believe the code has had any significant fixes that would 
help with this issue since then.


 I started looking druntime and dmd source code myself before i checked
 the thread (thsnks for your help and feedback) and i have some
 questions. could you explain to me something?

 i_m looking here
 https://github.com/D-Programming-Language/druntime/blob/v2.066.1/src/rt/lifetime.d#L591


 -------

 auto size = ti.next.tsize;

 why `next`? it can be even null if this is last TypeInfo in the linked
 list.


This is the way the compiler constructs the type info. The first 
TypeInfo is always TypeInfo_Array (or TypeInfo_Shared, or const or 
whatever), and the .next is the typeinfo for the element type. all this 
does is get the size of an element. Since we know we are dealing with an 
array, we know next is always valid.



 btw, i used suggested trackallocs.d and GC defenetely receives NO_SCAN

 before tag: 1 len: 2 ptr: 103A78058 root: 103A77000:8192 attr: APPENDABLE
 gc_qalloc(41, NO_SCAN APPENDABLE ) cc: 29106 asz: 10152603, ti: null
 ret: BlkInfo_(104423800, 64, 10)
 after tag: 1 len: 3 ptr: 104423810 root: 104423800:64 attr: NO_SCAN
 APPENDABLE


This is good information, thanks. I will get back to you with a druntime 
branch to try. Can I email you at this address? If not, email me at the 
address from my post to let me know your contact, no reason to work 
through building issues on the public forum :)

-Steve
Dec 12 2014
next sibling parent "Ruslan Mullakhmetov" <public mullakh.com> writes:
On Friday, 12 December 2014 at 15:50:26 UTC, Steven Schveighoffer 
wrote:


 Can I email you at this address? If not, email me at the 
 address from my post to let me know your contact, no reason to 
 work through building issues on the public forum :)

 -Steve


reach me at theambient [] me__com
Dec 12 2014
prev sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/12/14 10:50 AM, Steven Schveighoffer wrote:

 On 12/12/14 7:52 AM, Ruslan Mullakhmetov wrote:



 btw, i used suggested trackallocs.d and GC defenetely receives NO_SCAN

 before tag: 1 len: 2 ptr: 103A78058 root: 103A77000:8192 attr: APPENDABLE
 gc_qalloc(41, NO_SCAN APPENDABLE ) cc: 29106 asz: 10152603, ti: null
 ret: BlkInfo_(104423800, 64, 10)
 after tag: 1 len: 3 ptr: 104423810 root: 104423800:64 attr: NO_SCAN
 APPENDABLE


 This is good information, thanks. I will get back to you with a druntime
 branch to try. Can I email you at this address? If not, email me at the
 address from my post to let me know your contact, no reason to work
 through building issues on the public forum :)


For those who were interested, we were not able to solve this problem, 
and unfortunately Ruslan's company cannot keep trying to debug, they 
have moved on to another language :(

If anyone else has this kind of issue (array appending causing flags to 
change), please let me know, I would like to make sure this gets solved.

I am creating 2 fixes, one to fix the issue with the offset (in 
progress) and after that I will attempt to ensure any changes to a 
block's flags stick when the array is appended (I found at least one 
place in Phobos where this can cause a bug, but it's not affecting his 
code).

Ruslan, you may want to try the second fix when it has been added just 
to see if it helps.

The two bug reports are:

https://issues.dlang.org/show_bug.cgi?id=13854
https://issues.dlang.org/show_bug.cgi?id=13878

-Steve
Dec 19 2014