• WebFreak001 (17/17) Jun 18 2021 we have an issue in D-Scanner / libdparse that with very large
• Andrei Alexandrescu (2/26) Jun 21 2021 32 or 64 bits?
• WebFreak001 (3/6) Jun 22 2021 64 bits on linux
• WebFreak001 (8/9) Jun 25 2021 so it seems the data that later gets corrupted was returned by

WebFreak001 <d.forum webfreak.org> writes:

we have an issue in D-Scanner / libdparse that with very large 
files memory corruption occurs. After further investigation it 
seems that the contents of a string might have been moved as they 
are being reallocated.

The string is constructed 
[here](https://github.com/dlang-community/libdparse/pull/387/files#diff-073a681f37b7ac58a75cb15be50882a5075a2ebcaa53efc4b
97cffa037ab6ddR155) which calls [this
function](https://github.com/dlang-community/libdparse/pull/387/files#diff-a10d3a7875381b5020edcd6e55a5555e4c43c8ab1947b30b4
b434ce8fa59117R512) which allocates the string using an appender!string.

When first constructed the string is fine but after lots of 
further processing of other stuff the data the string points at 
suddenly is no longer considered allocated by the GC (maybe moved 
without updating the struct members?) and is reused from other 
stuff (in my case it was std.regex putting ir code in there, as 
shown in a GDB data watchpoint that first write to it)

See 
https://github.com/dlang-community/libdparse/pull/387#issuecomment-863442801

Does this seem like it should be possible if all code was safe? 
(`immutable(char)[]` being moved by GC) or do you see any obvious 
issues in this construct?

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/18/21 1:45 PM, WebFreak001 wrote:
 we have an issue in D-Scanner / libdparse that with very large files 
 memory corruption occurs. After further investigation it seems that the 
 contents of a string might have been moved as they are being reallocated.
 
 The string is constructed 
 [here](https://github.com/dlang-community/libdparse/pull/387/files#diff-073a681f37b7ac58a75cb15be50882a5075a2ebcaa53efc4b
97cffa037ab6ddR155) 
 which calls [this 
 function](https://github.com/dlang-community/libdparse/pull/387/files#diff-a10d3a7875381b5020edcd6e55a5555e4c43c8ab1947b30b4
b434ce8fa59117R512) 
 which allocates the string using an appender!string.
 
 When first constructed the string is fine but after lots of further 
 processing of other stuff the data the string points at suddenly is no 
 longer considered allocated by the GC (maybe moved without updating the 
 struct members?) and is reused from other stuff (in my case it was 
 std.regex putting ir code in there, as shown in a GDB data watchpoint 
 that first write to it)
 
 See 
 https://github.com/dlang-community/libdparse/pull/387#issuecomment-863442801 
 
 
 Does this seem like it should be possible if all code was safe? 
 (`immutable(char)[]` being moved by GC) or do you see any obvious issues 
 in this construct?

32 or 64 bits?

WebFreak001 <d.forum webfreak.org> writes:

On Tuesday, 22 June 2021 at 03:21:26 UTC, Andrei Alexandrescu 
wrote:
 On 6/18/21 1:45 PM, WebFreak001 wrote:
 [...]

 32 or 64 bits?

64 bits on linux

WebFreak001 <d.forum webfreak.org> writes:

On Friday, 18 June 2021 at 17:45:25 UTC, WebFreak001 wrote:
 [...]

so it seems the data that later gets corrupted was returned by 
appender!string

Calling GC.addRoot right when creating the data causes the data 
not to go corrupt, so it seems that the GC thinks the data is no 
longer accessible.

This might seem like a more precise question: is it possible to 
debug when the GC thinks data is no longer accessible?