www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Either I'm confused or the gc is

reply donallen <donaldcallen gmail.com> writes:
I'm new to D, but not to programming (I wrote my first line of 
code 60 years ago and am retired from a long career as a 
developer and project manager).

I have a suite of personal finance applications that I wrote in 
C. I'd like to port them to D. The financial data is stored in a 
sqlite database. I spent yesterday re-writing the verifier 
application in D and have run into a perplexing issue.

The verifier does a recursive descent of the tree of accounts, 
checking a bunch of things at each node. After the checks are 
done, a query is done to discover how many children the current 
account has, at which point it allocates, with 'new', a dynamic 
array of Account structs. It then goes into a loop, collecting 
information from the db about each of the children, which gets 
stored in one of the Account structs in the array. Finally, it 
does a
foreach through the array, doing a recursive call on the tree 
walker.

Here's the relevant code:
````
         // Now do the children of this account
         // First determine how many there are
         bind_text(count_children, 1, account.guid);
         int n_children = one_row(count_children, 
&get_int).integer_value;
         if (n_children > 0)
         {
             Account[] children = new Account[n_children];
             bind_text(find_children, 1, account.guid);
             int i = 0;
             while (next_row_available_p(find_children, 
&reset_stmt))
             {
                 if (i > n_children-1)
                     panic("walk_account_tree: child index exceeds 
n_children");
                 children[i].name = 
fromStringz(sqlite3_column_text(find_children, 0)).dup;
                 children[i].path = format("%s:%s", account.path, 
children[i].name);
                 children[i].guid = 
fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid
                 children[i].commodity_guid = 
fromStringz(sqlite3_column_text(find_children, 2)).dup; // 
commodity_guid
                 children[i].flags = 
sqlite3_column_int(find_children,
                         3) | account.flags & 
(account_flag_descendents_are_assets
                         | 
account_flag_descendents_are_liabilities | 
account_flag_descendents_are_income
                         | account_flag_descendents_are_expenses | 
account_flag_descendents_are_marketable
                         | 
account_flag_self_and_descendents_are_tax_related
                         | 
account_flag_descendents_need_commodity_link); // flags
                 i = i + 1;
             }
             foreach (k, ref child; children)
                 walk_account_tree(&child, ancestor_flags | 
account.flags);
         }
````

The problem is that at some point, the verifier starts spewing 
bogus error messages about what it is seeing in the tree. Oddly, 
putting in debugging writelns results in the error messages not 
occurring -- a Heisenbug.  But working with gdb, I found that the 
account structs after the error messages start are zeroed. 
Turning on gc profiling tells me that 3 gcs have occurred. 
Disabling the gc results in the program running correctly -- no 
error messages (and I know the database has no errors because the 
C version, which has been around for awhile, confirms that the db 
is well formed).

I could post a PR, but I'm not sure that this is a bug. It could 
easily be a misunderstanding by me of D and its memory 
management. So I thought I'd try this post first, hoping that one 
of you who knows the language better than I do could point out a 
problem with my code. I do need to resolve this or abandon this 
project.

Thanks in advance for any help.
Oct 21 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Oct 21, 2020 at 04:24:41PM +0000, donallen via Digitalmars-d wrote:

 I'm new to D, but not to programming (I wrote my first line of code 60
 years ago and am retired from a long career as a developer and project
 manager).


Welcome aboard! ;-)


[...]

 ````
         // Now do the children of this account
         // First determine how many there are
         bind_text(count_children, 1, account.guid);
         int n_children = one_row(count_children, &get_int).integer_value;
         if (n_children > 0)
         {
             Account[] children = new Account[n_children];
             bind_text(find_children, 1, account.guid);
             int i = 0;
             while (next_row_available_p(find_children, &reset_stmt))
             {
                 if (i > n_children-1)
                     panic("walk_account_tree: child index exceeds
 n_children");
                 children[i].name =
 fromStringz(sqlite3_column_text(find_children, 0)).dup;
                 children[i].path = format("%s:%s", account.path,
 children[i].name);
                 children[i].guid =
 fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid
                 children[i].commodity_guid =
 fromStringz(sqlite3_column_text(find_children, 2)).dup; // commodity_guid
                 children[i].flags = sqlite3_column_int(find_children,
                         3) | account.flags &
 (account_flag_descendents_are_assets
                         | account_flag_descendents_are_liabilities |
 account_flag_descendents_are_income
                         | account_flag_descendents_are_expenses |
 account_flag_descendents_are_marketable
                         | account_flag_self_and_descendents_are_tax_related
                         | account_flag_descendents_need_commodity_link); //
 flags
                 i = i + 1;
             }
             foreach (k, ref child; children)
                 walk_account_tree(&child, ancestor_flags | account.flags);
         }
 ````
 
 The problem is that at some point, the verifier starts spewing bogus
 error messages about what it is seeing in the tree. Oddly, putting in
 debugging writelns results in the error messages not occurring -- a
 Heisenbug.  But working with gdb, I found that the account structs
 after the error messages start are zeroed. Turning on gc profiling
 tells me that 3 gcs have occurred.  Disabling the gc results in the
 program running correctly -- no error messages (and I know the
 database has no errors because the C version, which has been around
 for awhile, confirms that the db is well formed).


Without seeing the rest of your code, or a minimal (runnable) failing
test case, it's hard to say for certain what the problem is.  But if I
were to guess, it looks like the GC is prematurely collecting your
arrays, though I can't imagine why it would if you still retain
references to it.

One potential gotcha, since you mention that this is being ported from
C, is that if you pass pointers to GC-allocated objects to C code which
stores it somewhere, you need to make sure you retain a reference
somewhere on the D side of things, or else inform the GC of the
reference using core.memory.GC.addRoot.[1]  Otherwise, since the GC may
not be aware of where the C code stores the pointer, it may fail to find
it and wrongly believe that the object is dead, and thereby collect it
prematurely.

[1] See: https://dlang.org/library/core/memory/gc.add_root.html

Another potential problem is if your C code (or C-style code ported to
D) obscures pointers, e.g., using the XOR trick to implement a
doubly-linked list with only a single pointer field, the GC will not be
able to discover the reference, and thus may wrongly mark the referenced
object as dead.

Another thing that stuck out to me while glancing over your code
snippet, is that the `children` array doesn't seem to be stored
anywhere; this means it will go out of scope at the end of the scope and
possibly be collected, unless you retain at least one reference to an
array element somewhere.  I can't tell if this is important without
knowing the rest of your code, but it's something to look into.  (Though
it puzzles me why this would be a problem, since obviously your code
still has a reference to *something* in that array in order for the
verifier code to check it. But it's something to look into if there are
no other clues.)



 I could post a PR, but I'm not sure that this is a bug. It could
 easily be a misunderstanding by me of D and its memory management. So
 I thought I'd try this post first, hoping that one of you who knows
 the language better than I do could point out a problem with my code.
 I do need to resolve this or abandon this project.


[...]

Have you identified what's causing the problem? I.e., what would you put
in your PR?


T

-- 
He who does not appreciate the beauty of language is not worthy to bemoan its
flaws.
Oct 21 2020
next sibling parent donallen <donaldcallen gmail.com> writes:
On Wednesday, 21 October 2020 at 18:55:50 UTC, H. S. Teoh wrote:

 On Wed, Oct 21, 2020 at 04:24:41PM +0000, donallen via 
 Digitalmars-d wrote:

 [...]


 Welcome aboard! ;-)

 [...]


Essentially what I wrote in my post, if I became convinced by you 
folks that the problem was not mine.


 [...]
Oct 21 2020
prev sibling next sibling parent reply donallen <donaldcallen gmail.com> writes:
On Wednesday, 21 October 2020 at 18:55:50 UTC, H. S. Teoh wrote:

 On Wed, Oct 21, 2020 at 04:24:41PM +0000, donallen via 
 Digitalmars-d wrote:

 [...]


 Welcome aboard! ;-)


 [...]

 [...]


 Without seeing the rest of your code, or a minimal (runnable) 
 failing test case, it's hard to say for certain what the 
 problem is.  But if I were to guess, it looks like the GC is 
 prematurely collecting your arrays, though I can't imagine why 
 it would if you still retain references to it.

 One potential gotcha, since you mention that this is being 
 ported from C, is that if you pass pointers to GC-allocated 
 objects to C code which stores it somewhere, you need to make 
 sure you retain a reference somewhere on the D side of things, 
 or else inform the GC of the reference using 
 core.memory.GC.addRoot.[1]  Otherwise, since the GC may not be 
 aware of where the C code stores the pointer, it may fail to 
 find it and wrongly believe that the object is dead, and 
 thereby collect it prematurely.


I am not passing pointers to GC-allocated objects to C code. The 
only C code involved here is libsqlite3 and there the pointers I 
pass to it are those I obtained from it.


 [1] See: https://dlang.org/library/core/memory/gc.add_root.html

 Another potential problem is if your C code (or C-style code 
 ported to D) obscures pointers, e.g., using the XOR trick to 
 implement a doubly-linked list with only a single pointer 
 field, the GC will not be able to discover the reference, and 
 thus may wrongly mark the referenced object as dead.


Nope.


 Another thing that stuck out to me while glancing over your 
 code snippet, is that the `children` array doesn't seem to be 
 stored anywhere; this means it will go out of scope at the end 
 of the scope and possibly be collected, unless you retain at 
 least one reference to an array element somewhere.  I can't 
 tell if this is important without knowing the rest of your 
 code, but it's something to look into.  (Though it puzzles me 
 why this would be a problem, since obviously your code still 
 has a reference to *something* in that array in order for the 
 verifier code to check it. But it's something to look into if 
 there are no other clues.)


As I understand it from the documentation, dynamic arrays are 
represented as fat pointers that contain the array size and a 
pointer to the actual data in the heap. I assume, though the 
documentation doesn't say, that the size/pointer structure is on 
the stack. If so, every one of the children arrays has a pointer 
to it on the stack as the descent through the tree proceeds and 
therefore should not be garbage collectible until a particular 
call to the tree walker (of which I've provided the last bit of 
code, which deals with descending into the child accounts) 
returns to its caller.

It does look to me like those arrays are being collected 
prematurely and incorrectly.
Oct 21 2020
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Oct 21, 2020 at 09:35:43PM +0000, donallen via Digitalmars-d wrote:
[...]

 As I understand it from the documentation, dynamic arrays are
 represented as fat pointers that contain the array size and a pointer
 to the actual data in the heap. I assume, though the documentation
 doesn't say, that the size/pointer structure is on the stack.


That is correct, since you declared the `children` array as a local
variable.



 If so, every one of the children arrays has a pointer to it on the
 stack as the descent through the tree proceeds and therefore should
 not be garbage collectible until a particular call to the tree walker
 (of which I've provided the last bit of code, which deals with
 descending into the child accounts) returns to its caller.


Also correct. Barring a bug in the GC.



 It does look to me like those arrays are being collected prematurely
 and incorrectly.


It would appear to be so, but it's really hard to say without being able
to reproduce the problem locally.  Another possibility is memory
corruption caused by incorrect translation of C code to D. Generally,
this should be pretty rare: Walter designed D syntax to be similar to C
syntax such that if it compiles, it should work exactly as it does in
C, otherwise there should be a compile error.  But there *are* corner
cases where this may inadvertently happen.

There are also some gotchas with interfacing D code with C libraries
(sqlite, as you mentioned), which may appear to work at first but may
introduce subtle memory problems if not used correctly.

Of course, there *is* the possibility that there's a bug in the GC that
causes it to collect prematurely, but IMO the chances of that are pretty
low, because D arrays are one of its most-used core features, and if
something so basic doesn't work correctly, we would have seen (and
fixed) the problem a long time ago.  At least, it would have surfaced in
some of the large D projects out there.  But I haven't heard of any such
problem recently, except in connection with code that interfaces with C,
where sometimes mistakes can happen and memory corruption results.

But on the off-chance there *is* a bug in the GC: we'd greatly
appreciate it if you could somehow reduce the problem to a minimal test
case that we can reproduce locally so that we can investigate.


T

-- 
Не дорог подарок, дорога любовь.
Oct 21 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Wednesday, 21 October 2020 at 22:28:48 UTC, H. S. Teoh wrote:

 On Wed, Oct 21, 2020 at 09:35:43PM +0000, donallen via 
 Digitalmars-d wrote: [...]

 [...]


 That is correct, since you declared the `children` array as a 
 local variable.



 [...]


 Also correct. Barring a bug in the GC.



 [...]


 It would appear to be so, but it's really hard to say without 
 being able to reproduce the problem locally.  Another 
 possibility is memory corruption caused by incorrect 
 translation of C code to D. Generally, this should be pretty 
 rare: Walter designed D syntax to be similar to C syntax such 
 that if it compiles, it should work exactly as it does in C, 
 otherwise there should be a compile error.  But there *are* 
 corner cases where this may inadvertently happen.

 There are also some gotchas with interfacing D code with C 
 libraries (sqlite, as you mentioned), which may appear to work 
 at first but may introduce subtle memory problems if not used 
 correctly.

 Of course, there *is* the possibility that there's a bug in the 
 GC that causes it to collect prematurely, but IMO the chances 
 of that are pretty low, because D arrays are one of its 
 most-used core features, and if something so basic doesn't work 
 correctly, we would have seen (and fixed) the problem a long 
 time ago.  At least, it would have surfaced in some of the 
 large D projects out there.  But I haven't heard of any such 
 problem recently, except in connection with code that 
 interfaces with C, where sometimes mistakes can happen and 
 memory corruption results.

 But on the off-chance there *is* a bug in the GC: we'd greatly 
 appreciate it if you could somehow reduce the problem to a 
 minimal test case that we can reproduce locally so that we can 
 investigate.


I will try.

My suspicion that there's a gc problem largely rests on the fact 
that when I run the program with the gc disabled, the problem 
goes away -- it behaves as it should.

Do you know if there's any way to configure the gc so it tells 
you what it's doing? It would be great if it could be told to 
announce when it's running and what it's collecting. I did a 
little looking in the documentation yesterday for that and did 
not find anything. If you can't help in that area, I'll have a 
look at the gc source code.


 T
Oct 21 2020
parent Max Haughton <maxhaton gmail.com> writes:
On Wednesday, 21 October 2020 at 23:02:07 UTC, donallen wrote:

 On Wednesday, 21 October 2020 at 22:28:48 UTC, H. S. Teoh wrote:

 [...]


 I will try.

 My suspicion that there's a gc problem largely rests on the 
 fact that when I run the program with the gc disabled, the 
 problem goes away -- it behaves as it should.

 Do you know if there's any way to configure the gc so it tells 
 you what it's doing? It would be great if it could be told to 
 announce when it's running and what it's collecting. I did a 
 little looking in the documentation yesterday for that and did 
 not find anything. If you can't help in that area, I'll have a 
 look at the gc source code.


 [...]




There are some flags and options, but if you can't find one that 
fixes your bug I would use something like bpf or gdb to watch it 
manually (if possible). GDB supports d demangling.
Oct 21 2020
prev sibling parent Max Haughton <maxhaton gmail.com> writes:
On Wednesday, 21 October 2020 at 18:55:50 UTC, H. S. Teoh wrote:

 On Wed, Oct 21, 2020 at 04:24:41PM +0000, donallen via 
 Digitalmars-d wrote:

 [...]


 Welcome aboard! ;-)

 [...]


I have some D bolted into unreal engine, to avoid upsetting the 
beast I malloc and copy every GC'd (not performance code so the 
GC is fine) buffer into the games allocator.
Oct 21 2020
prev sibling next sibling parent reply Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Wednesday, 21 October 2020 at 16:24:41 UTC, donallen wrote:

 I'm new to D, but not to programming (I wrote my first line of 
 code 60 years ago and am retired from a long career as a 
 developer and project manager).

 [...]


What's your i and n_children on panic?
Oct 21 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Wednesday, 21 October 2020 at 19:30:40 UTC, Imperatorn wrote:

 On Wednesday, 21 October 2020 at 16:24:41 UTC, donallen wrote:

 I'm new to D, but not to programming (I wrote my first line of 
 code 60 years ago and am retired from a long career as a 
 developer and project manager).

 [...]


 What's your i and n_children on panic?


There is no panic. The program starts complaining about the 
account data that it is checking, spewing bogus error messages. 
This is because the Account structures have been zeroed at some 
point in the recursive descent.

But I did set a breakpoint in gdb at the place where the error 
messages are issued, and i was something like 230 of an 
n_children of something like 350. I don't think either number is 
particularly significant, other than that this was an account 
that had a lot of children and so was likely to trigger a gc.
Oct 21 2020
parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Wednesday, 21 October 2020 at 21:26:36 UTC, donallen wrote:

 On Wednesday, 21 October 2020 at 19:30:40 UTC, Imperatorn wrote:

 On Wednesday, 21 October 2020 at 16:24:41 UTC, donallen wrote:

 I'm new to D, but not to programming (I wrote my first line 
 of code 60 years ago and am retired from a long career as a 
 developer and project manager).

 [...]


 What's your i and n_children on panic?


 There is no panic. The program starts complaining about the 
 account data that it is checking, spewing bogus error messages. 
 This is because the Account structures have been zeroed at some 
 point in the recursive descent.

 But I did set a breakpoint in gdb at the place where the error 
 messages are issued, and i was something like 230 of an 
 n_children of something like 350. I don't think either number 
 is particularly significant, other than that this was an 
 account that had a lot of children and so was likely to trigger 
 a gc.


Oh, sorry, I mis-read something 🐣

A divide and conquer approach would otherwise be to enable() and 
disable() GC at various places to narrow it down. Or even do a 
collect() and look at addrOf as stated earlier.

Would be good to know if there really is a bug in the GC or not. 
I suspect not, but you can't know for sure until you try.
Oct 22 2020
prev sibling next sibling parent ikod <igor.khasilev gmail.com> writes:
On Wednesday, 21 October 2020 at 16:24:41 UTC, donallen wrote:

 I'm new to D, but not to programming (I wrote my first line of 
 code 60 years ago and am retired from a long career as a 
 developer and project manager).


Very cool!


 Thanks in advance for any help.


Most likely GC clears prematurely some of your data (as H.S.Teoh 
pointed out).
You can try to disable GC for a single program run with 
--DRT-gcopt="disable:1" option. Just add it to your command line 
args and see what happens.

Another investigation can be done using memory checker like 
'valgrind'.

Regards,
Oct 21 2020
prev sibling next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
Try adding:

import core.memory : GC;
Account[] children = new Account[n_children];

GC.addRoot(children.ptr);
scope(exit)
	GC.removeRoot(children.ptr);

With the GC turned on, does it still cause issues?
Also are you compiling for 64bit or 32bit?
Oct 21 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 00:53:55 UTC, rikki cattermole 
wrote:

 Try adding:

 import core.memory : GC;
 Account[] children = new Account[n_children];

 GC.addRoot(children.ptr);
 scope(exit)
 	GC.removeRoot(children.ptr);

 With the GC turned on, does it still cause issues?
 Also are you compiling for 64bit or 32bit?


I tried this. With the gc on, the problem still occurs.

I am working on 64-bit Arch Linux systems, up-to-date, DMD64 D 
Compiler v2.094.0
Oct 21 2020
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 22/10/2020 5:02 PM, donallen wrote:

 I tried this. With the gc on, the problem still occurs.


It won't be the GC then.

I would look at stack corruption, so maybe look into valgrind.
Oct 21 2020
prev sibling parent reply tsbockman <thomas.bockman gmail.com> writes:
On Thursday, 22 October 2020 at 04:02:10 UTC, donallen wrote:

 I tried this. With the gc on, the problem still occurs.

 I am working on 64-bit Arch Linux systems, up-to-date, DMD64 D 
 Compiler v2.094.0


Have you tried using a different compiler? I know with the big 
languages like C or Java the meme is "it's never a compiler bug", 
but D is developed with *far* fewer resources, and compiler bugs 
are, sadly, not rare.

For example, DMD 2.094+ have this still-unfixed back-end 
regression that may possibly be causing your problem:

https://issues.dlang.org/show_bug.cgi?id=21325

The LDC and GDC back-ends are much more reliable (and better in 
other ways) than DMD's because they are tested and maintained by 
much larger communities.

I recommend giving the latest LDC a try, as well as a 
significantly older compiler of your choice, before assuming that 
the problem is in your code, or in the garbage collector.
Oct 22 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 18:54:24 UTC, tsbockman wrote:

 On Thursday, 22 October 2020 at 04:02:10 UTC, donallen wrote:

 I tried this. With the gc on, the problem still occurs.

 I am working on 64-bit Arch Linux systems, up-to-date, DMD64 D 
 Compiler v2.094.0


 Have you tried using a different compiler?


Yes -- to no avail:
dca franz:~/Software/newcash_d/verifier$ make
cd ../library ; make
make[1]: Entering directory '/home/dca/Software/newcash_d/library'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/dca/Software/newcash_d/library'
#dmd -g -profile=gc -I=../library -L='-lsqlite3' verifier.d 
../library/lib.o
ldc -g -I=../library -L='-lsqlite3' verifier.d ../library/lib.o
../library/lib.o:lib.d:function 
_D3std9exception__T7bailOutHTC9ExceptionZQwFNaNfAyamMAxaZv: 
error: undefined reference to '_d_throwdwarf'
../library/lib.o:lib.d:function 
_D3std3utf__T10decodeImplVbi1VEQBd8typecons__T4FlagVAyaa19_7573655265706c6163656d656e744463686172ZQCa
0TAxwZQDrFNaQkKmZw: error: undefined reference to '_d_throwdwarf'
../library/lib.o:lib.d:function 
_D3std6format__T14formattedWriteTSQBg5array__T8AppenderTAyaZQoTaTAaZQ
aFNaNfKQBsMxAaQtZk: error: undefined reference to '_d_throwdwarf'
../library/lib.o:lib.d:function 
_D3std6format__T6getNthVAyaa13_696e7465676572207769647468SQCe6traits10isIntegral
iTAaZQCsFNaNfkQmZi: error: undefined reference to '_d_throwdwarf'
../library/lib.o:lib.d:function 
_D3std6format__T13formatElementTSQBf5stdio4File17LockingTextWriterTAyaTaZQCfFNfKQBwQqMKxSQDjQDi__T
0FormatSpecTaZQpZv: error: undefined reference to '__dmd_begin_catch'
../library/lib.o:lib.d:function 
_D3std6format__T15formatValueImplTSQBh5stdio4File17LockingTextWriterTwTaZQCfFNfKQBuwMKxSQDiQDh__T
0FormatSpecTaZQpZv: error: undefined reference to '_memset32'
../library/lib.o:lib.d:DW.ref.__dmd_personality_v0: error: 
undefined reference to '__dmd_personality_v0'
../library/lib.o:lib.d:function 
_D3std9algorithm9iteration__T8splitterVAyaa6_61203d3d2062TQtTQwZQBjFQBdQBgZ6Result11__xopEqualsFKxSQDtQDsQDl__TQDeVQCya6_61203d3d2062TQDrTQDvZQE
FQEdQEgZQDaKxQCiZb: error: undefined reference to
'_D4core8internal5array8equality__T8__equalsTyaTyaZQqFNaNbNiNfMAyaMQeZb'
../library/lib.o:lib.d:function 
_D3std9algorithm9iteration__T8splitterVAyaa6_61203d3d2062TQtTQwZQBjFQBdQBgZ6Result11__xopEqualsFKxSQDtQDsQDl__TQDeVQCya6_61203d3d2062TQDrTQDvZQE
FQEdQEgZQDaKxQCiZb: error: undefined reference to
'_D4core8internal5array8equality__T8__equalsTyaTyaZQqFNaNbNiNfMAyaMQeZb'
../library/lib.o:lib.d:function 
_D3std4conv__T10emplaceRefTAyaTQeTQhZQxFKQoKQrZ1S11__xopEqualsFKxSQCmQCl__TQCjTQCaTQCeTQCiZQCzF
QCrKQCvZQCfKxQBsZb: error: undefined reference to
'_D4core8internal5array8equality__T8__equalsTyaTyaZQqFNaNbNiNfMAyaMQeZb'
../library/lib.o:lib.d:function 
_D4core8internal5array8equality__T8__equalsTxAyaTxQfZQtFNaNbNiNfMAxQwMQfZb:
error: undefined reference to
'_D4core8internal5array8equality__T8__equalsTyaTyaZQqFNaNbNiNfMAyaMQeZb'
collect2: error: ld returned 1 exit status
Error: /usr/bin/cc failed with status: 1
make: *** [makefile:11: verifier] Error 1
dca franz:~/Software/newcash_d/verifier$ make
cd ../library ; make
make[1]: Entering directory '/home/dca/Software/newcash_d/library'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/dca/Software/newcash_d/library'
#dmd -g -profile=gc -I=../library -L='-lsqlite3' verifier.d 
../library/lib.o
gdc -g -I=../library -L='-lsqlite3' verifier.d ../library/lib.o
verifier.d:4:8: error: module lib is in file 'lib.d' which cannot 
be read
     4 | import lib;
       |        ^
import path[0] = /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/d
make: *** [makefile:11: verifier] Error 1
dca franz:~/Software/newcash_d/verifier$ ls -l ../library/lib.d
-rw-r--r-- 1 dca allen 12244 Oct 20 11:19 ../library/lib.d
dca franz:~/Software/newcash_d/verifier$

I hate to say this, because so much of this project makes sense 
to me -- the design of the language, the effort that has been put 
into documentation, the helpfulness of the community -- but I 
think I have to throw in the towel. I can't spend any more time 
on this and it appears that D is just not solid enough for the 
work I'm trying to do with it.
Oct 22 2020
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 22 October 2020 at 19:38:42 UTC, donallen wrote:

 [snip]

 I hate to say this, because so much of this project makes sense 
 to me -- the design of the language, the effort that has been 
 put into documentation, the helpfulness of the community -- but 
 I think I have to throw in the towel. I can't spend any more 
 time on this and it appears that D is just not solid enough for 
 the work I'm trying to do with it.


 From the community's perspective, if there is a compiler bug that 
is causing your problem, then it will neither be found nor fixed, 
which is disappointing.

In general, you will always get better help when you are able to 
provide either the complete code or a reduced example and what 
commands you used to run it, as was mentioned above.
Oct 22 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 21:00:41 UTC, jmh530 wrote:

 On Thursday, 22 October 2020 at 19:38:42 UTC, donallen wrote:

 [snip]

 I hate to say this, because so much of this project makes 
 sense to me -- the design of the language, the effort that has 
 been put into documentation, the helpfulness of the community 
 -- but I think I have to throw in the towel. I can't spend any 
 more time on this and it appears that D is just not solid 
 enough for the work I'm trying to do with it.


 From the community's perspective, if there is a compiler bug 
 that is causing your problem, then it will neither be found nor 
 fixed, which is disappointing.

 In general, you will always get better help when you are able 
 to provide either the complete code or a reduced example and 
 what commands you used to run it, as was mentioned above.


I've already provided valgrind output that shows the gc 
referencing an uninitialized variable.

Please understand -- I don't have infinite time to spend on this 
and I do have alternatives for this work, e.g., nim or go or chez 
scheme. Given the valgrind output indicating what looks like a 
serious error, I'm not going to put the effort into trying to 
reduce this error to an example that I can share fully unless and 
until someone convinces me that what valgrind is saying doesn't 
indicate a bug in the gc.
Oct 22 2020
parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 10/22/20 2:28 PM, donallen wrote:


 I've already provided valgrind output that shows the gc referencing an
 uninitialized variable.


I had similar problems. Luckily, all my problems were issues with my code.

* In one case, the program was linked with dmd but I was loading a 
shared library (written again in D) but I was calling dlopen(). Instead, 
I had to call Runtime.loadLibrary.

* In another case, my D library was being loaded by Python. I had 
forgotten to initialize the D runtime. I had to call Runtime.initialize.

* In yet another case, I was calling myObject.close() in destructors, 
forgetting that myObject was already finalized by the GC.


 Please understand -- I don't have infinite time to spend on this


It's some more work but are you aware of Dustmite, which may be able to 
magically reduce your issue to a minimal case?

 
https://dlang.org/blog/2020/04/13/dustmite-the-general-purpose-data-reduction-tool/

Ali
Oct 22 2020
prev sibling parent reply user1234 <user1234 12.de> writes:
On Thursday, 22 October 2020 at 19:38:42 UTC, donallen wrote:

 On Thursday, 22 October 2020 at 18:54:24 UTC, tsbockman wrote:

 On Thursday, 22 October 2020 at 04:02:10 UTC, donallen wrote:

 I tried this. With the gc on, the problem still occurs.

 I am working on 64-bit Arch Linux systems, up-to-date, DMD64 
 D Compiler v2.094.0


 Have you tried using a different compiler?


 Yes -- to no avail:
 dca franz:~/Software/newcash_d/verifier$ make
 cd ../library ; make
 make[1]: Entering directory 
 '/home/dca/Software/newcash_d/library'
 make[1]: Nothing to be done for 'all'.
 make[1]: Leaving directory 
 '/home/dca/Software/newcash_d/library'
 #dmd -g -profile=gc -I=../library -L='-lsqlite3' verifier.d 
 ../library/lib.o
 ldc -g -I=../library -L='-lsqlite3' verifier.d ../library/lib.o
 ../library/lib.o:lib.d:function 
 _D3std9exception__T7bailOutHTC9ExceptionZQwFNaNfAyamMAxaZv: 
 error: undefined reference to '_d_throwdwarf'
 ../library/lib.o:lib.d:function [...]


The errors you get are because you must compile *everything* with 
ldc (program name should be ldc2 BTW), but the lib.o file is 
still produced with dmd as it contains DMD specific symbols.


 I hate to say this, because so much of this project makes sense 
 to me -- the design of the language, the effort that has been 
 put into documentation, the helpfulness of the community -- but 
 I think I have to throw in the towel. I can't spend any more 
 time on this and it appears that D is just not solid enough for 
 the work I'm trying to do with it.


Yeah, that's perfectly understandable.

As you've been said in one of the first answer this must be 
because D GC sees a pointer allocated with malloc, and this 
pointer looks orphan to the GC as it has no root.

Would you put something that's mallocated in a D struct or in a D 
array ?
 From the memory point of view the code you have shown is correct 
(the `fromStringz().dup` is exactly what's must be done to 
prevent problems when mixin manual and D managed memory) so the 
origin of the problem really *has* to be somewhere else.
Oct 22 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Oct 22, 2020 at 10:05:42PM +0000, user1234 via Digitalmars-d wrote:
[...]

 As you've been said in one of the first answer this must be because D
 GC sees a pointer allocated with malloc, and this pointer looks orphan
 to the GC as it has no root.


The GC does not collect objects allocated by malloc. (That would not
make any sense, since if you allocated it with malloc, obviously you
intend to deallocate it with free!)


T

-- 
I see that you JS got Bach.
Oct 22 2020
parent user1234 <user1234 12.de> writes:
On Thursday, 22 October 2020 at 23:07:28 UTC, H. S. Teoh wrote:

 On Thu, Oct 22, 2020 at 10:05:42PM +0000, user1234 via 
 Digitalmars-d wrote: [...]

 As you've been said in one of the first answer this must be 
 because D GC sees a pointer allocated with malloc, and this 
 pointer looks orphan to the GC as it has no root.


 The GC does not collect objects allocated by malloc. (That 
 would not make any sense, since if you allocated it with 
 malloc, obviously you intend to deallocate it with free!)


 T


Yeah you're right. I was actually thinking to the opposite case:
the parent is mallocated but the members are not, eg

---
import core.memory;
import std.experimental.allocator;
import std.experimental.allocator.mallocator;
import std.experimental.allocator.common;

enum fill = "azertyuiopqsdfghjklm";

struct Node
{
     this(string c){content = c;}
     string content;
     Node*[] nodes;
}

void main()
{
     Node* root = make!Node(Mallocator.instance, fill);
     foreach(immutable i; 0 .. 10000)
     {
         root.nodes ~= make!Node(Mallocator.instance, fill);
         foreach(immutable j; 0 .. 100)
             root.nodes[i].nodes ~= make!Node(Mallocator.instance, 
fill);
     }
     assert(root.content == fill);
     foreach(immutable i; 0 .. root.nodes.length)
     {
         assert(root.nodes[i].content == fill);
         foreach(immutable j; 0 .. root.nodes[i].nodes.length)
             assert(root.nodes[i].nodes[j].content == fill);
     }
}
---

which crashes because of that (it "should" crash but I dont have 
much memory installed so maybe this is not the case for everyone)
Oct 22 2020
prev sibling parent reply donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 22:05:42 UTC, user1234 wrote:

 On Thursday, 22 October 2020 at 19:38:42 UTC, donallen wrote:

 On Thursday, 22 October 2020 at 18:54:24 UTC, tsbockman wrote:

 On Thursday, 22 October 2020 at 04:02:10 UTC, donallen wrote:

 I tried this. With the gc on, the problem still occurs.

 I am working on 64-bit Arch Linux systems, up-to-date, DMD64 
 D Compiler v2.094.0


 Have you tried using a different compiler?


 Yes -- to no avail:
 dca franz:~/Software/newcash_d/verifier$ make
 cd ../library ; make
 make[1]: Entering directory 
 '/home/dca/Software/newcash_d/library'
 make[1]: Nothing to be done for 'all'.
 make[1]: Leaving directory 
 '/home/dca/Software/newcash_d/library'
 #dmd -g -profile=gc -I=../library -L='-lsqlite3' verifier.d 
 ../library/lib.o
 ldc -g -I=../library -L='-lsqlite3' verifier.d ../library/lib.o
 ../library/lib.o:lib.d:function 
 _D3std9exception__T7bailOutHTC9ExceptionZQwFNaNfAyamMAxaZv: 
 error: undefined reference to '_d_throwdwarf'
 ../library/lib.o:lib.d:function [...]


 The errors you get are because you must compile *everything* 
 with ldc (program name should be ldc2 BTW), but the lib.o file 
 is still produced with dmd as it contains DMD specific symbols.


Yes, you are right. I deleted the .o files and rebuilt everything 
with ldc and it linked properly. But it not only produces the 
same error as dmd, but a new one as well (see the last line:

A commodity with the same name as the account does exist. The 
account will be linked to it.
  requires a link to a commodity but doesn't have one.
A commodity with the same name as the account does exist. The 
account will be linked to it.
  requires a link to a commodity but doesn't have one.
A commodity with the same name as the account does exist. The 
account will be linked to it.
  requires a link to a commodity but doesn't have one.
A commodity with the same name as the account does exist. The 
account will be linked to it.
  requires a link to a commodity but doesn't have one.
A commodity with the same name as the account does exist. The 
account will be linked to it.
  requires a link to a commodity but doesn't have one.
A commodity with the same name as the account does exist. The 
account will be linked to it.
  requires a link to a commodity but doesn't have one.
A commodity with the same name as the account does exist. The 
account will be linked to it.
walk_account_tree: child index exceeds n_children
dca igor:~/Software/newcash_d/verifier$



 I hate to say this, because so much of this project makes 
 sense to me -- the design of the language, the effort that has 
 been put into documentation, the helpfulness of the community 
 -- but I think I have to throw in the towel. I can't spend any 
 more time on this and it appears that D is just not solid 
 enough for the work I'm trying to do with it.


 Yeah, that's perfectly understandable.

 As you've been said in one of the first answer this must be 
 because D GC sees a pointer allocated with malloc, and this 
 pointer looks orphan to the GC as it has no root.

 Would you put something that's mallocated in a D struct or in a 
 D array ?
 From the memory point of view the code you have shown is 
 correct (the `fromStringz().dup` is exactly what's must be done 
 to prevent problems when mixin manual and D managed memory) so 
 the origin of the problem really *has* to be somewhere else.
Oct 22 2020
parent user1234 <user1234 12.de> writes:
On Thursday, 22 October 2020 at 23:48:22 UTC, donallen wrote:

 I hate to say this, because so much of this project makes 
 sense to me -- the design of the language, the effort that 
 has been put into documentation, the helpfulness of the 
 community -- but I think I have to throw in the towel. I 
 can't spend any more time on this and it appears that D is 
 just not solid enough for the work I'm trying to do with it.






BTW I forgot to mention, even if it's too late I suppose, that to 
convert a C program to D a way to go is:

1. convert minimally and use -betterC
2. leave the -betterC mode
3. incrementally add more idiomatic D feature

see 
https://dlang.org/blog/2018/06/11/dasbetterc-converting-make-c-to-d/ for more
detail on the methodolgy.
Oct 23 2020
prev sibling next sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
donallen wrote:


 But working with gdb, I found that the account structs after 
 the error messages start are zeroed.


GC should not zero any memory by itself (it doesn't "clear" freed chunks). 
it looks like the GC is reusing the allocated memory.

you can use `core.memory.GC.addrOf()` to check if your pointers are 
still "alive". you can add calls to `addrOf()` before processing a data, it 
may help to catch the moment data is incorrectly freed (if that happens).
Oct 21 2020
parent donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 01:30:41 UTC, ketmar wrote:

 donallen wrote:


 But working with gdb, I found that the account structs after 
 the error messages start are zeroed.


 GC should not zero any memory by itself (it doesn't "clear" 
 freed chunks). it looks like the GC is reusing the allocated 
 memory.


The thought did occur to me as well that the zeroing probably 
occurs on allocation, but I don't see how you get from that to 
the behavior I'm observing, since the children array gets  
populated with data right after allocation. It appears that the 
Account structs being passed in the recursive calls to 
walk_account_tree, which are elements of the children array, are 
zeroed.


 you can use `core.memory.GC.addrOf()` to check if your pointers 
 are still "alive". you can add calls to `addrOf()` before 
 processing a data, it may help to catch the moment data is 
 incorrectly freed (if that happens).


I'll give this a try tomorrow. Thanks.
Oct 21 2020
prev sibling next sibling parent reply donallen <donaldcallen gmail.com> writes:
Some new things:

1. I tried building with the gc-profile option, like so:

dmd -g -profile=gc -I=../library -L='-lsqlite3' verifier.d 
../library/lib.o

When I run the program built this way, the problem does not 
occur. There is also no gc profiling output. Makes me wonder if 
the compile-time option somehow altered things (increased heap 
size?) that resulted in no gcs? Pure speculation on my part, but 
it is consistent with the behavior (program works correctly when 
the gc doesn't run, no profiling output suggests no gc).

But if I enable gc profiling at the command line:
verifier "--DRT-gcopt=profile:1" /tmp/Finances.newcash

the problem does occur and I get profiling output:
         Number of collections:  3
         Total GC prep time:  0 milliseconds
         Total mark time:  0 milliseconds
         Total sweep time:  0 milliseconds
         Max Pause Time:  0 milliseconds
         Grand total GC time:  0 milliseconds
GC summary:    5 MB,    3 GC    0 ms, Pauses    0 ms <    0 ms

2. I tried adding a call to addrOf(children.ptr), panicking on a 
null return, just before the recursive calls to the tree walker. 
No panic.

I will look into a way to reproduce this that I can give to the 
project. The problem is not the code -- it's the data. The 
database that the D version of the verifier is failing on 
contains literally decades of our financial transactions. So 
while there is nothing illegal, immoral or fattening in it, it's 
nobody's business but ours. So I need to create a new database 
with no personal info in it that provokes the problem.

I will also look into running the verifier under Valgrind.

I don't know when I will be able to devote more time to this. 
Like everyone, I've got other things on my plate.

Thanks to everyone who has tried to help.
Oct 22 2020
parent reply donallen <donaldcallen gmail.com> writes:
Realizing that running this thing under valgrind was easy, I did 
so:

valgrind --leak-check=yes --track-origins=yes ./verifier 
/tmp/Finances.newcash > /tmp/vg.output 2>&1

and then

ddemangle /tmp/vg.output > /tmp/vg-demangled.output

Here is the first half of the output (posting in two parts 
because it's big), which suggests a problem in the gc (the 
messages beginning "requires a link to a commodity ..." are the 
bogus errors produced by the verifier due to looking at zeroed 
Account structs; I've removed the duplicates):

==3854== Memcheck, a memory error detector
==3854== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward 
et al.
==3854== Using Valgrind-3.16.1 and LibVEX; rerun with -h for 
copyright info
==3854== Command: ./verifier /tmp/Finances.newcash
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x21D7C6: nothrow scope void 
gc.impl.conservative.gc.Gcx.collectRoots(void*, void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23BD4B: nothrow void 
core.thread.threadbase.thread_scanAll(scope void delegate(void*, 
void*) nothrow).__lambda2!(core.thread.threadbase.ScanType, 
void*, void*).__lambda2(core.thread.threadbase.ScanType, void*, 
void*) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2472CF: nothrow void 
core.thread.threadbase.scanAllTypeImpl(scope void 
delegate(core.thread.threadbase.ScanType, void*, void*) nothrow, 
void*) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x24720C: nothrow void 
core.thread.threadbase.thread_scanAllType(scope void 
delegate(core.thread.threadbase.ScanType, void*, void*) 
nothrow).__lambda2!(void*).__lambda2(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x211EEF: nothrow void 
core.thread.osthread.callWithStackShell(scope void 
delegate(void*) nothrow) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2471E1: thread_scanAllType (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23BD11: thread_scanAll (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21D901: nothrow void 
gc.impl.conservative.gc.Gcx.collectAllRoots(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21EFF9: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x211EBC: nothrow void 
core.thread.osthread.callWithStackShell(scope void 
delegate(void*) nothrow) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Thread 2:
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224995: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249A8: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249B7: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249BE: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224A0A: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224A4F: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x23C1CB: nothrow  nogc ulong 
gc.bits.GCBits.setLocked(ulong) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x224A6C: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224A8D: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249D6: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249E3: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224987: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224C56: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Thread 1:
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224995: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249A8: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249B7: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249BE: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224A0A: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    b
Oct 22 2020
next sibling parent reply donallen <donaldcallen gmail.com> writes:
Here's the rest of it:

y 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224A4F: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x23C1CB: nothrow  nogc ulong 
gc.bits.GCBits.setLocked(ulong) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x224A6C: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224A8D: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224987: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224C56: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
  requires a link to a commodity but doesn't have one.
A commodity with the same name as the account does exist. The 
account will be linked to it.
  [snip]
  ==3854== Thread 4:
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224AF4: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x23C1CB: nothrow  nogc ulong 
gc.bits.GCBits.setLocked(ulong) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x224B39: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224B3D: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224B5A: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224B67: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224B75: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Thread 1:
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x21DA9C: nothrow ulong 
gc.impl.conservative.gc.Gcx.sweep() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21EB9F: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6446: pure nothrow core.memory.BlkInfo_ 
core.memory.GC.qalloc(ulong, uint, const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B90AF: pure nothrow  trusted 
core.memory.BlkInfo_ 
std.array.Appender!(immutable(char)[]).Appender.ensureAddabl
(ulong).__lambda3() (/usr/include/dlang/dmd/std/array.d:3386)
==3854==    by 0x1B8FCD: pure nothrow  safe void 
std.array.Appender!(immutable(char)[]).Appender.ensureAddable(ulong)
(/usr/include/dlang/dmd/std/array.d:3386)
==3854==    by 0x1BD1E6: pure nothrow  safe char[] 
std.array.Appender!(immutable(char)[]).Appender.put!(immutable(char)[]).put(immutable(char)[
).bigDataFun(ulong) (/usr/include/dlang/dmd/std/array.d:3488)
==3854==    by 0x1BD0D4: pure nothrow  safe void 
std.array.Appender!(immutable(char)[]).Appender.put!(immutable(char)[]).pu
(immutable(char)[]) (/usr/include/dlang/dmd/std/array.d:3491)
==3854==    by 0x1C3B92: pure nothrow  safe void 
std.range.primitives.doPut!(std.array.Appender!(immutable(char)[]).Appender,
immutable(char)[]).doPut(ref std.array.Appender!(immutable(char)[]).Appender,
ref immutable(char)[]) (/usr/include/dlang/dmd/std/range/primitives.d:277)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
but doesn't have one.
A commodity with the same name as the account does exist. The 
account will be linked to it.
  requires a link to a commodity but doesn't have one.
[snip]
==3854==
==3854== HEAP SUMMARY:
==3854==     in use at exit: 2,380,288 bytes in 2,013 blocks
==3854==   total heap usage: 339,984 allocs, 337,971 frees, 
439,341,276 bytes allocated
==3854==
==3854== 16 bytes in 1 blocks are definitely lost in loss record 
21 of 171
==3854==    at 0x483A77F: malloc (vg_replace_malloc.c:307)
==3854==    by 0x208B1B: nothrow  nogc void* rt.tlsgc.init() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x207181: thread_init (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1FEC3B: rt_init (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E76DB: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).runAll() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7678: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).tryExec(scope void 
delegate()) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E75E1: _d_run_main2 (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7359: _d_run_main (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8A89: main 
(/usr/include/dlang/dmd/core/internal/entrypoint.d:29)
==3854==
==3854== 32 bytes in 1 blocks are possibly lost in loss record 33 
of 171
==3854==    at 0x483A77F: malloc (vg_replace_malloc.c:307)
==3854==    by 0x2195E5: core.gc.gcinterface.GC 
gc.impl.conservative.gc.initialize() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x20CF7C: core.gc.gcinterface.GC 
core.gc.registry.createGCInstance(immutable(char)[]) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2074E2: gc_init_nothrow (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1FDA7F: nothrow void* 
gc.impl.proto.gc.ProtoGC.malloc(ulong, uint, const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C1A: gc_malloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E8514: _d_allocmemory (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B6A4F: _Dmain (verifier.d:8)
==3854==    by 0x1E77CA: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).runAll().__lambda1() 
(in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7678: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).tryExec(scope void 
delegate()) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7752: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).runAll() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7678: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).tryExec(scope void 
delegate()) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== 2,380,056 (768 direct, 2,379,288 indirect) bytes in 1 
blocks are definitely lost in loss record 171 of 171
==3854==    at 0x483A77F: malloc (vg_replace_malloc.c:307)
==3854==    by 0x4905F13: ??? (in /usr/lib/libsqlite3.so.0.8.6)
==3854==    by 0x4905489: sqlite3Malloc (in 
/usr/lib/libsqlite3.so.0.8.6)
==3854==    by 0x490599A: sqlite3MallocZero (in 
/usr/lib/libsqlite3.so.0.8.6)
==3854==    by 0x490454E: ??? (in /usr/lib/libsqlite3.so.0.8.6)
==3854==    by 0x1D77FA: void* lib.open_db(immutable(char)[]) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B6ACB: _Dmain (verifier.d:33)
==3854==    by 0x1E77CA: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).runAll().__lambda1() 
(in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7678: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).tryExec(scope void 
delegate()) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7752: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).runAll() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7678: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).tryExec(scope void 
delegate()) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E75E1: _d_run_main2 (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== LEAK SUMMARY:
==3854==    definitely lost: 784 bytes in 2 blocks
==3854==    indirectly lost: 2,379,288 bytes in 2,005 blocks
==3854==      possibly lost: 32 bytes in 1 blocks
==3854==    still reachable: 184 bytes in 5 blocks
==3854==         suppressed: 0 bytes in 0 blocks
==3854== Reachable blocks (those to which a pointer was found) 
are not shown.
==3854== To see them, rerun with: --leak-check=full 
--show-leak-kinds=all
==3854==
==3854== For lists of detected and suppressed errors, rerun with: 
-s
==3854== ERROR SUMMARY: 3213 errors from 33 contexts (suppressed: 
0 from 0)
Oct 22 2020
parent Jerry <labuurii gmail.com> writes:
Maybe the answer is in the valgrind logs, its alot to read. But 
isn't it possible to put a write breakpoint in the accounts 
arrays and step through the code until a zero is written.
Oct 22 2020
prev sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 23/10/2020 4:56 AM, donallen wrote:

 ==3854==    by 0x1B7E98: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, 
 int) (verifier.d:272)
 ==3854==  Uninitialised value was created by a stack allocation
 ==3854==    at 0x1B77FC: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, 
 int) (verifier.d:84)
 ==3854==
 ==3854== Conditional jump or move depends on uninitialised value(s)


That is coming up an awful lot.

We'd need walk_account_tree source, to know if its doing something funky.

But upon reviewing how the entire thing is working, I think we need 
one_row and next_row_available_p source as well. Oh and reset_stmt too.
Oct 22 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 21:58:04 UTC, rikki cattermole 
wrote:

 On 23/10/2020 4:56 AM, donallen wrote:

 ==3854==    by 0x1B7E98: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
 ==3854==  Uninitialised value was created by a stack allocation
 ==3854==    at 0x1B77FC: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
 ==3854==
 ==3854== Conditional jump or move depends on uninitialised 
 value(s)


 That is coming up an awful lot.

 We'd need walk_account_tree source, to know if its doing 
 something funky.

 But upon reviewing how the entire thing is working, I think we 
 need one_row and next_row_available_p source as well. Oh and 
 reset_stmt too.


Here's the source code you requested:

     void walk_account_tree(Account account, int ancestor_flags)
     {
         void fix_missing_commodity()
         {
             immutable string new_commodity_guid = 
one_row(new_guid, &get_string).string_value;
             // Create new commodity
             bind_text(insert_new_commodity, 1, 
new_commodity_guid);
             bind_text(insert_new_commodity, 2, account.name);
             run_dm_stmt(insert_new_commodity, &reset_stmt);

             // And link the account to the new commodity
             bind_text(link_to_commodity, 1, new_commodity_guid);
             bind_text(link_to_commodity, 2, account.name);
             run_dm_stmt(link_to_commodity, &reset_stmt);
         }

         void check_and_repair_commodity_link()
         {
             if (account.commodity_guid.length == 0)
             {
                 bind_text(get_possible_commodity_guid, 1, 
account.name);
                 immutable Maybe maybe_commodity_guid = 
maybe_one_row(get_possible_commodity_guid,
                         &get_string);
                 if (!maybe_commodity_guid.valid_p)
                 {
                     // No commodity exists with the same name as 
the account.  Create one and link the account to it.
                     writeln(account.path,
                             " requires a link to a commodity but 
doesn't have one. A commodity
with the same name as the account does not exist. One will be 
created
with the symbol **unknown** and the account will be linked to it. 
If
you wish to get quotes for this commodity, you will need to fix 
the
symbol in Newcash.");
                     fix_missing_commodity();
                 }
                 else
                 {
                     writeln(account.path, " requires a link to a 
commodity but doesn't have one.
A commodity with the same name as the account does exist. The 
account will be linked to it.");
                     bind_text(link_to_commodity, 1, 
maybe_commodity_guid.value.string_value);
                     bind_text(link_to_commodity, 2, account.guid);
                     run_dm_stmt(link_to_commodity, &reset_stmt);
                 }
             }
             else
             {
                 sqlite3_reset(get_possible_commodity_guid);
                 /* This marketable account has a commodity. Check 
to be sure that the guid is valid.
            If not, set the account's commodity_guid to NULL and 
process this account again. */
                 bind_text(verify_commodity_guid, 1, 
account.commodity_guid);
                 Maybe maybe_valid_guid = 
maybe_one_row(verify_commodity_guid, &get_string);
                 if (!maybe_valid_guid.valid_p)
                 {
                     writeln(account.path,
                             " requires a commodity link and has 
one, but the commodity guid is
invalid. Creating a new commodity with the same name as the 
account and
linking the account to it.");
                     fix_missing_commodity();
                 }
                 else
                 {
                     // Warn if the commodity name is not the same 
as the account name
                     bind_text(check_commodity_name, 1, 
account.guid);
                     MultiType n = one_row(check_commodity_name, 
&get_int);
                     if (n.integer_value == 0)
                     {
                         writeln(account.path, " requires a 
commodity link and has one, but the
commodity name is not the same as the account name. Is this 
intentional?");
                     }
                 }
             }
         }

         // Placeholder?
         if ((account.flags & account_flag_placeholder) == 0)
         {
             // No. Is this account an asset?
             if ((ancestor_flags & 
account_flag_descendents_are_assets) != 0)
             {
                 // Yes. Is it marketable?
                 if ((ancestor_flags & 
account_flag_descendents_are_marketable) != 0)
                 {
                     // Yes, account is a marketable asset. Check 
that it is linked to a proper commodity.
                     check_and_repair_commodity_link();
                 }
                 else
                 {
                     // No
                     if (account.commodity_guid.length > 0)
                     {
                         // This account is a non-marketable asset 
and has a non-null commodity guid.
                         // Set to NULL. Non-marketable accounts 
should not point to commodities.
                         writeln(account.path,
                                 " %s is a non-marketable asset, 
but it is associated with a
commodity. Removing the association by setting the account's 
commodity link to NULL.");
                         bind_text(nullify_commodity_guid, 1, 
account.guid);
                         run_dm_stmt(nullify_commodity_guid, 
&reset_stmt);
                     }
                     // Make sure the quantity is zero. Should not 
be otherwise for a non-marketable asset.
                     bind_text(check_quantities, 1, account.guid);
                     if (one_row(check_quantities, 
&get_int).integer_value > 0)
                     {
                         writeln(account.path,
                                 " is a non-marketable asset 
account but has splits with non-zero quantities.
These will be fixed.");
                         bind_text(fix_quantities, 1, 
account.guid);
                         run_dm_stmt(fix_quantities, &reset_stmt);
                     }
                 }
             }
             else
             {
                 // This account is not an asset. Make sure it 
doesn't point to a commodity,
                 // unless it is an Income account and inherits 
the descendents-need-commodity property.
                 // Does it need a commodity link?
                 if ((ancestor_flags & 
account_flag_descendents_need_commodity_link) != 0)
                 {
                     // Yes. Is it an income account?
                     if ((ancestor_flags & 
account_flag_descendents_are_income) != 0)
                     {
                         // Yes
                         check_and_repair_commodity_link();
                     }
                     else
                     {
                         writeln("The account ", account.path,
                                 " is not an Asset or Income 
account, but inherits the
'needs commodity' property. This should not be possible and
indicates a bug in Newcash or in the Verifier. Please report to 
Don Allen.");
                     }
                 }
                 else
                 {
                     if (account.commodity_guid.length != 0)
                     {
                         // This account is not an asset, doesn't 
have the needs-commodity-link property and has a non-null 
commodity guid, which we set to NULL.
                         writeln(account.path, " is not an asset, 
doesn't have the needs-commodity-link property, but it is 
associated with a commodity.
Removing the association by setting the account's commodity link 
to NULL.");
                         bind_text(nullify_commodity_guid, 1, 
account.guid);
                         run_dm_stmt(nullify_commodity_guid, 
&reset_stmt);
                     }
                     // The account is not an asset and does not 
have needs-commodity-link property.
                     // Just make sure it doesn't inherit the 
marketable property
                     if ((ancestor_flags & 
account_flag_descendents_are_marketable) != 0)
                     {
                         writeln(account.path,
                                 " is designated marketable, but 
is not an asset account. This
should not be possible and is indicative of a Newcash bug. Please 
report this to Don Allen.");
                     }
                 }
                 // Make sure the quantity is zero. Should not be 
otherwise for a non-asset.
                 bind_text(check_quantities, 1, account.guid);
                 int temp;
                 if ((temp = one_row(check_quantities, 
&get_int).integer_value) > 0)
                 {
                     writeln(account.path,
                             " %s is not an asset account but has 
splits with non-zero quantities. These will be fixed.");

                     bind_text(fix_quantities, 1, account.guid);
                     run_dm_stmt(fix_quantities, &reset_stmt);
                 }
             }
         }
         else
         {
             // This account is a place-holder. Verify that it has 
no transactions. Warn the user if that is not true.
             bind_text(count_transactions, 1, account.guid);
             int transaction_count = one_row(count_transactions, 
&get_int).integer_value;
             if (transaction_count > 0)
             {
                 writeln(account.path,
                         " is a placeholder account, but it has %d 
transactions. You should
re-assign them to an appropriate account with Newcash.");
             }
         }
         // Now do the children of this account
         // First determine how many there are
         bind_text(count_children, 1, account.guid);
         int n_children = one_row(count_children, 
&get_int).integer_value;
         if (n_children > 0)
         {
             Account[] children = new Account[n_children];
             bind_text(find_children, 1, account.guid);
             int i = 0;
             while (next_row_available_p(find_children, 
&reset_stmt))
             {
                 if (i > n_children-1)
                     panic("walk_account_tree: child index exceeds 
n_children");
                 children[i].name = 
fromStringz(sqlite3_column_text(find_children, 0)).dup;
                 children[i].path = format("%s:%s", account.path, 
children[i].name);
                 children[i].guid = 
fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid
                 children[i].commodity_guid = 
fromStringz(sqlite3_column_text(find_children, 2)).dup; // 
commodity_guid
                 children[i].flags = 
sqlite3_column_int(find_children,
                         3) | account.flags & 
(account_flag_descendents_are_assets
                         | 
account_flag_descendents_are_liabilities | 
account_flag_descendents_are_income
                         | account_flag_descendents_are_expenses | 
account_flag_descendents_are_marketable
                         | 
account_flag_self_and_descendents_are_tax_related
                         | 
account_flag_descendents_need_commodity_link); // flags
                 i = i + 1;
             }
             foreach (child; children) {
                 walk_account_tree(child, ancestor_flags | 
account.flags);
             }
         }
     }

alias one_row_value_delegate = bool delegate (sqlite3_stmt* stmt, 
MultiType* result);
alias one_row_value_function = bool function (sqlite3_stmt* stmt, 
MultiType* result);
MultiType one_row(T)(sqlite3_stmt* stmt, T get_values)
{
     MultiType result;
     if (next_row_available_p(stmt, &reset_stmt))
     {
         if (get_values(stmt, &result))
         { // Did we get a value of the type requested?
             // Yes. Now make sure we only got one row back. This 
will also reset the stmt.
             if (next_row_available_p(stmt, &reset_stmt))
                 panic("one_row: stmt returned more than one row");
             return result;
         }
         else
             panic("one_row: stmt returned a null value");
     }
     else
         panic("one_row: stmt returned no rows");
     return result; // Need this for compiler happiness -- can't 
get here
}

/// Check for availability of another row
bool next_row_available_p(sqlite3_stmt* stmt, int 
function(sqlite3_stmt* stmt) cleanup)
{
     switch (sqlite3_step(stmt))
     {
     case sqlite_row:
         return (true);
     case sqlite_done:
         /* Make the prepared stmt available for re-use. */
         cleanup(stmt);
         return (false);
     default:
         stderr.writeln("next_row_available_p failed",);
         stderr.writeln(sqlite3_errmsg(sqlite3_db_handle(stmt)), 
stderr);
         exit(EXIT_FAILURE);
     }
     return (false); // Need this for compiler happiness -- can't 
get here
}

/// Reset so statement can be re-used
int reset_stmt(sqlite3_stmt* stmt)
{
     return sqlite3_reset(stmt);
}
Oct 23 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Oct 23, 2020 at 12:47:11PM +0000, donallen via Digitalmars-d wrote:

 On Thursday, 22 October 2020 at 21:58:04 UTC, rikki cattermole wrote:

 On 23/10/2020 4:56 AM, donallen wrote:

 ==3854== by 0x1B7E98: void
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable(char)[][]).Account,
 int) (verifier.d:272)
 ==3854== Uninitialised value was created by a stack allocation
 ==3854== at 0x1B77FC: void
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable(char)[][]).Account,
 int) (verifier.d:84)
 ==3854==
 ==3854== Conditional jump or move depends on uninitialised value(s)






[...]

Hmm. Which lines in your posted code do lines 272 and 84 refer to?
Those seem to be the problematic spots, but it's not clear where in the
function they are, since your posted code doesn't indicate line numbers.


I also noticed this bit of code (which may or may not have anything to
do with it):


                 int temp;
                 if ((temp = one_row(check_quantities,
 &get_int).integer_value) > 0)
                 {
                     writeln(account.path,
                             " %s is not an asset account but has splits with
 non-zero quantities. These will be fixed.");
 
                     bind_text(fix_quantities, 1, account.guid);
                     run_dm_stmt(fix_quantities, &reset_stmt);
                 }


The value of `temp` is not used after assignment. This is a rather odd
way of writing it.  What's the reason you didn't just write:

	if (one_row(...) > 0)

instead?


[...]

 MultiType one_row(T)(sqlite3_stmt* stmt, T get_values)


What's the definition of MultiType?


[...]

     return result; // Need this for compiler happiness -- can't get here


Note for the future: you could write `assert(0);` here to indicate that
it should not be reachable. (It compiles to a single 'hlt' instruction,
which will abort if the program somehow reaches it anyway.)


T

-- 
My program has no bugs! Only unintentional features...
Oct 23 2020
parent donallen <donaldcallen gmail.com> writes:
On Friday, 23 October 2020 at 14:37:41 UTC, H. S. Teoh wrote:

 On Fri, Oct 23, 2020 at 12:47:11PM +0000, donallen via 
 Digitalmars-d wrote:

 On Thursday, 22 October 2020 at 21:58:04 UTC, rikki cattermole 
 wrote:

 On 23/10/2020 4:56 AM, donallen wrote:

 ==3854==    by 0x1B7E98: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable(char)[][]).Account,
 int) (verifier.d:272)
 ==3854==  Uninitialised value was created by a stack 
 allocation
 ==3854==    at 0x1B77FC: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable(char)[][]).Account,
 int) (verifier.d:84)
 ==3854==
 ==3854== Conditional jump or move depends on uninitialised 
 value(s)






 [...]

 Hmm. Which lines in your posted code do lines 272 and 84 refer 
 to? Those seem to be the problematic spots, but it's not clear 
 where in the function they are, since your posted code doesn't 
 indicate line numbers.


 I also noticed this bit of code (which may or may not have 
 anything to
 do with it):


                 int temp;
                 if ((temp = one_row(check_quantities,
 &get_int).integer_value) > 0)
                 {
                     writeln(account.path,
                             " %s is not an asset account but 
 has splits with
 non-zero quantities. These will be fixed.");
 
                     bind_text(fix_quantities, 1, account.guid);
                     run_dm_stmt(fix_quantities, &reset_stmt);
                 }


 The value of `temp` is not used after assignment. This is a 
 rather odd way of writing it.  What's the reason you didn't 
 just write:

 	if (one_row(...) > 0)

 instead?


This code was ported from the original C. In the C code, I 
printf-ed the quantity in the error message and so needed to 
stash it somewhere (temp). When I transformed the code to D, I 
think I hadn't discovered format at that point, so just used 
writeln's ability to print the account.path prepended to the 
fixed text of the message. I forgot to take out the temporary and 
write it as you suggest above. So what you see in my code is a 
vestige of the C that is no longer applicable, but not harmful.


 [...]

 MultiType one_row(T)(sqlite3_stmt* stmt, T get_values)


 What's the definition of MultiType?


union MultiType
{
     int integer_value;
     sqlite3_int64 long_integer_value;
     double double_value;
     string string_value;
     void* pointer_value;
}


 [...]

     return result; // Need this for compiler happiness -- 
 can't get here


 Note for the future: you could write `assert(0);` here to 
 indicate that it should not be reachable. (It compiles to a 
 single 'hlt' instruction, which will abort if the program 
 somehow reaches it anyway.)


Thanks for the tip, useful if there's D in my future (which there 
will be if you guys can find something wrong with my code rather 
than the system).


 T
Oct 23 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/23/20 8:47 AM, donallen wrote:

 On Thursday, 22 October 2020 at 21:58:04 UTC, rikki cattermole wrote:

 On 23/10/2020 4:56 AM, donallen wrote:

 ==3854==    by 0x1B7E98: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, 
 int) (verifier.d:272)
 ==3854==  Uninitialised value was created by a stack allocation
 ==3854==    at 0x1B77FC: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, 
 int) (verifier.d:84)
 ==3854==
 ==3854== Conditional jump or move depends on uninitialised value(s)


 That is coming up an awful lot.

 We'd need walk_account_tree source, to know if its doing something funky.

 But upon reviewing how the entire thing is working, I think we need 
 one_row and next_row_available_p source as well. Oh and reset_stmt too.


 
 Here's the source code you requested:


This all looks ok. The only think I can possibly consider is that you 
are using bind_text a lot, to presumably bind sqlite prepared statements 
to D strings.

This means it's *possible* that the prepared statements have pointers to 
the strings, but nothing else does (hard to work it out from just 
reading the code, I can't tell when things go out of scope easily, and 
your prepared statements clearly exist outside these functions). If a 
prepared statement is a C-malloc'd item, then it's not scanned by the 
GC. This means that what possibly might be happening is that you bind a 
D string to a text parameter, the D string goes out of scope, the GC 
collects it and allocates it elsewhere, and then sqlite tries to use it 
to send some requests.

To test this theory, maybe try a version of the code that when calling 
bind_text, you append the string itself to a GC allocated array stored 
in a global. Might not prove anything, but if it's fixing the problem, 
maybe that's where the GC is collecting things that it shouldn't.

-Steve
Oct 23 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Friday, 23 October 2020 at 15:49:49 UTC, Steven Schveighoffer 
wrote:

 On 10/23/20 8:47 AM, donallen wrote:

 On Thursday, 22 October 2020 at 21:58:04 UTC, rikki cattermole 
 wrote:

 On 23/10/2020 4:56 AM, donallen wrote:

 ==3854==    by 0x1B7E98: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
 ==3854==  Uninitialised value was created by a stack 
 allocation
 ==3854==    at 0x1B77FC: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
 ==3854==
 ==3854== Conditional jump or move depends on uninitialised 
 value(s)


 That is coming up an awful lot.

 We'd need walk_account_tree source, to know if its doing 
 something funky.

 But upon reviewing how the entire thing is working, I think 
 we need one_row and next_row_available_p source as well. Oh 
 and reset_stmt too.


 
 Here's the source code you requested:


 This all looks ok. The only think I can possibly consider is 
 that you are using bind_text a lot, to presumably bind sqlite 
 prepared statements to D strings.

 This means it's *possible* that the prepared statements have 
 pointers to the strings, but nothing else does (hard to work it 
 out from just reading the code, I can't tell when things go out 
 of scope easily, and your prepared statements clearly exist 
 outside these functions). If a prepared statement is a 
 C-malloc'd item, then it's not scanned by the GC. This means 
 that what possibly might be happening is that you bind a D 
 string to a text parameter, the D string goes out of scope, the 
 GC collects it and allocates it elsewhere, and then sqlite 
 tries to use it to send some requests.

 To test this theory, maybe try a version of the code that when 
 calling bind_text, you append the string itself to a GC 
 allocated array stored in a global. Might not prove anything, 
 but if it's fixing the problem, maybe that's where the GC is 
 collecting things that it shouldn't.


It's a good theory, but I don't think it explains this.

What you are hypothesizing is that there's an instance where the 
lifetime of a D string that is used in a sqlite bind_text is 
shorter than the lifetime of the sqlite binding. I don't think 
that's the case.

The calls to bind_text all immediately precede calls to 
run_dm_stmt, one_row, etc. Those calls either run an insert or 
update, or in the case of one_row, a select. When those
calls return, the statement has been reset, which means, among 
other things, that the
bindings have been released, ending the binding's life.

The other case to look at is the use of next_row_available_p in a 
loop. There, bindings are established before entering the loop, 
which processes multiple rows coming back from a select. When 
next_row_available_p runs out of rows, it calls the cleanup 
function, usually reset_stmt (which just calls sqlite3_reset; I 
found that I could not pass an extern (c) function to 
next_row_available_p, so used this workaround; maybe there is a 
way to do this, but I just used this simple expedient). So at 
that point the bindings are cleared and next_row_available_p 
returns false, ending the loop and life of the bindings. Unless 
I've missed something, I think you will find that all the strings 
passed to bind_text (which get massaged by toStringz, so they 
look like C strings) live beyond the point where the statements 
they are bound to are reset.

Most of the calls to bind_text in the tree walker bind fields in 
account, which is an argument to the tree walker and thus lives 
until it returns. The others are to local variables that are 
declared before they are used in bind-text calls and the 
statement gets reset before the scope ends.

Yes, your theory could be tested using some variant of what you 
describe, in case my there's a flaw in my hand-waving above.

Some food for thought:

If I insert a writeln at the very beginning of walk_account_tree, 
printing, say, the name and guid of the account, the problem goes 
away -- the program behaves correctly.

If I change
children[i].path = format("%s:%s", account.path, 
children[i].name);
to
children[i].path = format("%s:%s", account.path, 
children[i].name).dup;

in the loop that fills in the child Account structs in children, 
same thing -- correct behavior. The dup of the format should not 
be necessary, since format returns a new GC-allocated string.

I suspect all of these odd "cures" have the effect they do 
because they change the allocation pattern of this section of the 
code. Without any of them, it appears that an allocation is 
triggering a GC at exactly the wrong moment. What is not 
understood is what is wrong with the moment.


 -Steve
Oct 23 2020
next sibling parent matheus <matheus gmail.com> writes:
On Friday, 23 October 2020 at 18:31:41 UTC, donallen wrote:

 ...


Have you tried Dustmite?

https://dlang.org/blog/2020/04/13/dustmite-the-general-purpose-data-reduction-tool/

Matheus.
Oct 23 2020
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Oct 23, 2020 at 06:31:41PM +0000, donallen via Digitalmars-d wrote:
[...]

 Some food for thought:
 
 If I insert a writeln at the very beginning of walk_account_tree,
 printing, say, the name and guid of the account, the problem goes away
 -- the program behaves correctly.


Curious.  IME, this is usually the sign of memory corruption elsewhere
in the code, and it's just pure happenstance that changing the
allocation pattern causes the corruption to overwrite something else
that doesn't lead to observable symptoms.

I wonder what might happen if instead of inserting a writeln, you
inserted a dummy allocation? Like:

	int[] dummy = [ 1, 2, 3 ];
	printf("%d\n", dummy.length); // don't let optimizer elide it



 If I change
 children[i].path = format("%s:%s", account.path, children[i].name);
 to
 children[i].path = format("%s:%s", account.path, children[i].name).dup;
 
 in the loop that fills in the child Account structs in children, same
 thing -- correct behavior. The dup of the format should not be
 necessary, since format returns a new GC-allocated string.


Indeed.  So again it looks like some kind of memory corruption.  Only,
without a reproducible test case we're just shooting in the dark here as
to what exactly is causing it.  Have you looked into Dustmite?  Once you
set it up, it's fully automated, and you can leave it running in the
background until it has reduced the code to a minimal(-ish) test case,
so it won't consume too much of your time.

Another random shot in the dark is, what compiler flag(s) are you using
to compile the program?  If you're using dmd with -O or -inline (or
both), there's a small chance you might be seeing a codegen bug.  (I've
had bad experiences with dmd's backend before, both in terms of codegen
bugs and poor optimization quality, and nowadays only use dmd for quick
prototyping or fast turnaround during development; for production I
avoid dmd and use LDC's much better and more reliable backend.)



 I suspect all of these odd "cures" have the effect they do because
 they change the allocation pattern of this section of the code.
 Without any of them, it appears that an allocation is triggering a GC
 at exactly the wrong moment. What is not understood is what is wrong
 with the moment.


[...]

Indeed.  If you could get dustmite to produce a minimal testcase that we
can use to reproduce the problem locally, we'd be able to help you
pinpoint exactly what is wrong with your code.  Or determine that it's a
GC bug, should it come to that.


T

-- 
Give a man a fish, and he eats once. Teach a man to fish, and he will sit
forever.
Oct 23 2020
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/23/20 2:31 PM, donallen wrote:

 On Friday, 23 October 2020 at 15:49:49 UTC, Steven Schveighoffer wrote:

 On 10/23/20 8:47 AM, donallen wrote:





 This all looks ok. The only think I can possibly consider is that you 
 are using bind_text a lot, to presumably bind sqlite prepared 
 statements to D strings.

 This means it's *possible* that the prepared statements have pointers 
 to the strings, but nothing else does (hard to work it out from just 
 reading the code, I can't tell when things go out of scope easily, and 
 your prepared statements clearly exist outside these functions). If a 
 prepared statement is a C-malloc'd item, then it's not scanned by the 
 GC. This means that what possibly might be happening is that you bind 
 a D string to a text parameter, the D string goes out of scope, the GC 
 collects it and allocates it elsewhere, and then sqlite tries to use 
 it to send some requests.

 To test this theory, maybe try a version of the code that when calling 
 bind_text, you append the string itself to a GC allocated array stored 
 in a global. Might not prove anything, but if it's fixing the problem, 
 maybe that's where the GC is collecting things that it shouldn't.


 
 It's a good theory, but I don't think it explains this.
 
 What you are hypothesizing is that there's an instance where the 
 lifetime of a D string that is used in a sqlite bind_text is shorter 
 than the lifetime of the sqlite binding. I don't think that's the case.
 
 The calls to bind_text all immediately precede calls to run_dm_stmt, 
 one_row, etc. Those calls either run an insert or update, or in the case 
 of one_row, a select. When those
 calls return, the statement has been reset, which means, among other 
 things, that the
 bindings have been released, ending the binding's life.


OK, I didn't know how that worked.


 
 The other case to look at is the use of next_row_available_p in a loop. 
 There, bindings are established before entering the loop, which 
 processes multiple rows coming back from a select. When 
 next_row_available_p runs out of rows, it calls the cleanup function, 
 usually reset_stmt (which just calls sqlite3_reset; I found that I could 
 not pass an extern (c) function to next_row_available_p, so used this 
 workaround; maybe there is a way to do this, but I just used this simple 
 expedient).


extern(C) function types are different than extern(D) function types. So 
it depends on what the parameter type is.


 So at that point the bindings are cleared and 
 next_row_available_p returns false, ending the loop and life of the 
 bindings. Unless I've missed something, I think you will find that all 
 the strings passed to bind_text (which get massaged by toStringz, so 
 they look like C strings) live beyond the point where the statements 
 they are bound to are reset.
 
 Most of the calls to bind_text in the tree walker bind fields in 
 account, which is an argument to the tree walker and thus lives until it 
 returns. The others are to local variables that are declared before they 
 are used in bind-text calls and the statement gets reset before the 
 scope ends.
 
 Yes, your theory could be tested using some variant of what you 
 describe, in case my there's a flaw in my hand-waving above.
 
 Some food for thought:
 
 If I insert a writeln at the very beginning of walk_account_tree, 
 printing, say, the name and guid of the account, the problem goes away 
 -- the program behaves correctly.


as with many memory corruption issues, introducing new code paths can 
change where the corruption occurs, or change that the corruption occurs 
at all.

You might want to try instead using printf, to avoid affecting anything 
in the D world. In the past to try and find corruption issues like this 
that are finnicky, I've also pre-allocated data to write to, and don't 
touch it for the duration of the problematic area, and then print it after.


 
 If I change
 children[i].path = format("%s:%s", account.path, children[i].name);
 to
 children[i].path = format("%s:%s", account.path, children[i].name).dup;
 
 in the loop that fills in the child Account structs in children, same 
 thing -- correct behavior. The dup of the format should not be 
 necessary, since format returns a new GC-allocated string.


Hm... that's interesting. You are correct that it allocates a new GC 
block. The dup is (should be) completely unnecessary. I don't think 
Appender (which is what format uses) has GC memory corruption issues, 
because it's used literally everywhere, it should be rock-solid. Always 
possible though.


 I suspect all of these odd "cures" have the effect they do because they 
 change the allocation pattern of this section of the code. Without any 
 of them, it appears that an allocation is triggering a GC at exactly the 
 wrong moment. What is not understood is what is wrong with the moment.


Yeah, I agree. For sure most of these constructs you are using are very 
well tested. There are many programs which use them, for many years, 
which still points me back at the sqlite library wrapper/binding. Not 
impossible that format and the GC have a bug, but it seems unlikely. Any 
time you have GC memory being prematurely collected, it smells like 
something was stored where it shouldn't be. The other major cause of 
this is generally alignment. But it doesn't seem like you are specifying 
alignment anywhere.

In any case, I understand if you don't want to go through any more 
exercises, but if you do, it might help find something that is 
super-obscure. Thanks for taking the time to investigate as much as you 
have!

-Steve
Oct 23 2020
prev sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
Thanks!

The problem is not in that block of code I believe.

Now how about bind_text?

Is that still using c variadic args?
Oct 23 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Friday, 23 October 2020 at 23:41:05 UTC, rikki cattermole 
wrote:

 Thanks!

 The problem is not in that block of code I believe.

 Now how about bind_text?

 Is that still using c variadic args?


No. Read on.

The detective in me couldn't resist (I really need a Peugeot and 
a basset hound) and so I did some more work on this despite not 
really having the time and I have made some progress.

The walk_account_tree function is defined inside the main 
function. Before it is defined,
I prepare all the sql statements needed to do the verification, 
stored in local variables. One of the queries is count_children:

"select count(guid) from accounts where parent_guid = ?1"

That query is run in the section of walk_account_tree that we've 
been looking at:
         // Now do the children of this account
         // First determine how many there are
         bind_text(count_children, 1, account.guid);
         int n_children = one_row(count_children, 
&get_int).integer_value;
         if (n_children > 0)
         {
             Account[] children = new Account[n_children];
             bind_text(find_children, 1, account.guid);

So the result of that query is saved in n_children and determines 
the size of the children array.
Recall that 'Account account' is the first argument to 
walk_account_tree, so n_children is the number of accounts that 
have account.guid,
an account's unique identifier, as its parent. In other words, 
its children.

If there are children, I allocate the children array and run the 
find_children query:

             "select name, guid, ifnull(commodity_guid, ''), flags
                 from accounts where parent_guid = ?1");

The number of rows that come back ought to be same as n_children. 
On a hunch, I put in a
test to check this:

         // Now do the children of this account
         // First determine how many there are
         bind_text(count_children, 1, account.guid);
         int n_children = one_row(count_children, 
&get_int).integer_value;
         if (n_children > 0)
         {
             Account[] children = new Account[n_children];
             bind_text(find_children, 1, account.guid);
             int i = 0;
             while (next_row_available_p(find_children, 
&reset_stmt))
             {
                 children[i].name = 
fromStringz(sqlite3_column_text(find_children, 0)).dup;
                 children[i].path = format("%s:%s", account.path, 
children[i].name);
                children[i].guid = 
fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid
                 children[i].commodity_guid = 
fromStringz(sqlite3_column_text(find_children, 2)).dup; // 
commodity_guid
                 children[i].flags = 
sqlite3_column_int(find_children,
                         3) | account.flags & 
(account_flag_descendents_are_assets
                         | 
account_flag_descendents_are_liabilities | 
account_flag_descendents_are_income
                         | account_flag_descendents_are_expenses | 
account_flag_descendents_are_marketable
                         | 
account_flag_self_and_descendents_are_tax_related
                         | 
account_flag_descendents_need_commodity_link); // flags
                 i = i + 1;
             }
             if (i != n_children)
                 panic(format("walk_account_tree: bad child index, 
%d, %d, %s", i, n_children, account.path));
             foreach (child; children) {
                 walk_account_tree(child, ancestor_flags | 
account.flags);
             }
         }

And, sure enough, the panic happens:
dca pangloss:~/Software/newcash_d/verifier$ ./verifier 
/tmp/Finances.newcash
walk_account_tree: bad child index, 231, 344, 
:Assets:Investments:Equities and derivatives:Taxable:Donald C. 
Allen 2003 Revocable Trust:TD Ameritrade
dca pangloss:~/Software/newcash_d/verifier$

Something has caused sqlite to end the return of rows 
prematurely. But next_row_p did not panic,
which it will if the the return code from sqlite3_step, which it 
calls, is other than SQLITE_ROW or SQLITE_DONE.
Since next_row_p just returned 'false', sqlite3_step must have 
returned SQLITE_DONE. sqlite returned 231 of 344 rows but is 
acting like everything is normal.

This explains why my earlier gdb session revealed that the error 
messages the verifier was spewing were due to being fed account 
structs that were all zero.
The recursive calls are being done on all 344 of the elements of 
the children array, but only 231 have data in them.
The rest are zero, as they were when the whole children array was 
allocated and those dataless "children" look bogus to the 
verifier.

Now, I know from prior experiment that if I disable the gc, the 
problem does not occur, so I tried this again with the newly 
installed panic:

dca pangloss:~/Software/newcash_d/verifier$ ./verifier 
"--DRT-gcopt=disable:1" /tmp/Finances.newcash
dca pangloss:~/Software/newcash_d/verifier$

This was run with the same executable as the above. No panic.

Next, I put in a call to GC.disable() just before the 'while' and 
a call to GC.enable{} after the loop returns. No panic.

So I moved GC.disable and enable calls into the loop, at first 
with the disable call at the top of the loop, which did not 
panic. I moved it down
one line at a time and when I get to this:

         // Now do the children of this account
         // First determine how many there are
         bind_text(count_children, 1, account.guid);
         int n_children = one_row(count_children, 
&get_int).integer_value;
         if (n_children > 0)
         {
             Account[] children = new Account[n_children];
             bind_text(find_children, 1, account.guid);
             int i = 0;
             while (next_row_available_p(find_children, 
&reset_stmt))
             {
                 children[i].name = 
fromStringz(sqlite3_column_text(find_children, 0)).dup;
                 children[i].path = format("%s:%s", account.path, 
children[i].name);
                 GC.disable();
                children[i].guid = 
fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid
                 children[i].commodity_guid = 
fromStringz(sqlite3_column_text(find_children, 2)).dup; // 
commodity_guid
                 children[i].flags = 
sqlite3_column_int(find_children,
                         3) | account.flags & 
(account_flag_descendents_are_assets
                         | 
account_flag_descendents_are_liabilities | 
account_flag_descendents_are_income
                         | account_flag_descendents_are_expenses | 
account_flag_descendents_are_marketable
                         | 
account_flag_self_and_descendents_are_tax_related
                         | 
account_flag_descendents_need_commodity_link); // flags
                 i = i + 1;
                 GC.enable();
             }
             if (i != n_children)
                 panic(format("walk_account_tree: bad child index, 
%d, %d, %s", i, n_children, account.path));
             foreach (child; children) {
                 walk_account_tree(child, ancestor_flags | 
account.flags);
             }
         }

the panic occurs.

Then I had another of my hunches. I re-wrote the line containing 
the call to format like so:

         // Now do the children of this account
         // First determine how many there are
         bind_text(count_children, 1, account.guid);
         int n_children = one_row(count_children, 
&get_int).integer_value;
         if (n_children > 0)
         {
             Account[] children = new Account[n_children];
             bind_text(find_children, 1, account.guid);
             int i = 0;
             while (next_row_available_p(find_children, 
&reset_stmt))
             {
                 children[i].name = 
fromStringz(sqlite3_column_text(find_children, 0)).dup;
                 size_t n_collections = 
GC.ProfileStats.numCollections;
                 children[i].path = (account.path ~ ":" ~ 
children[i].name);
                 bool collection_occured = 
GC.ProfileStats.numCollections != n_collections;
                children[i].guid = 
fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid
                 children[i].commodity_guid = 
fromStringz(sqlite3_column_text(find_children, 2)).dup; // 
commodity_guid
                 children[i].flags = 
sqlite3_column_int(find_children,
                         3) | account.flags & 
(account_flag_descendents_are_assets
                         | 
account_flag_descendents_are_liabilities | 
account_flag_descendents_are_income
                         | account_flag_descendents_are_expenses | 
account_flag_descendents_are_marketable
                         | 
account_flag_self_and_descendents_are_tax_related
                         | 
account_flag_descendents_need_commodity_link); // flags
                 i = i + 1;
             }
             if (i != n_children)
                 panic(format("walk_account_tree: bad child index, 
%d, %d, %s, %s", i, collection_occurred, n_children, 
account.path));
             foreach (child; children) {
                 walk_account_tree(child, ancestor_flags | 
account.flags);
             }
         }

This does not panic! In theory, this should consume the same 
amount of GC memory as the format call (though I have not looked 
at the format code; there may well
be intermediate allocations as it does its work).

Now I decided to instrument the code so I could determine whether 
GCs were occurring as a result of the call to format:

         // Now do the children of this account
         // First determine how many there are
         bind_text(count_children, 1, account.guid);
         int n_children = one_row(count_children, 
&get_int).integer_value;
         if (n_children > 0)
         {
             size_t collections_before = 0;
             size_t collections_after = 0;
             Account[] children = new Account[n_children];
             bind_text(find_children, 1, account.guid);
             int i = 0;
             while (next_row_available_p(find_children, 
&reset_stmt))
             {
                 children[i].name = 
fromStringz(sqlite3_column_text(find_children, 0)).dup;
                 collections_before += 
GC.profileStats().numCollections;
                 //children[i].path = (account.path ~ ":" ~ 
children[i].name);
                 children[i].path = format("%s:%s", account.path, 
children[i].name);
                 collections_after += 
GC.profileStats().numCollections;
               children[i].guid = 
fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid
                 children[i].commodity_guid = 
fromStringz(sqlite3_column_text(find_children, 2)).dup; // 
commodity_guid
                 children[i].flags = 
sqlite3_column_int(find_children,
                         3) | account.flags & 
(account_flag_descendents_are_assets
                         | 
account_flag_descendents_are_liabilities | 
account_flag_descendents_are_income
                         | account_flag_descendents_are_expenses | 
account_flag_descendents_are_marketable
                         | 
account_flag_self_and_descendents_are_tax_related
                         | 
account_flag_descendents_need_commodity_link); // flags
                 i = i + 1;
             }
             if (i != n_children)
                 panic(format("walk_account_tree: bad child index, 
%d, %d, %s, %d, %d", i, n_children, account.path, 
collections_before, collections_after));
             if (account.path == ":Assets:Investments:Equities and 
derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD 
Ameritrade") {
                 writeln(format("walk_account_tree: %d, %d, %s, 
%d, %d", i, n_children, account.path, collections_before, 
collections_after));
             }
             foreach (child; children) {
                 walk_account_tree(child, ancestor_flags | 
account.flags);
             }
         }

Running this produces:
dca pangloss:~/Software/newcash_d/verifier$ ./verifier 
"--DRT-gcopt=profile:1" /tmp/Finances.newcash
walk_account_tree: bad child index, 231, 344, 
:Assets:Investments:Equities and derivatives:Taxable:Donald C. 
Allen 2003 Revocable Trust:TD Ameritrade, 33, 34
dca pangloss:~/Software/newcash_d/verifier$

This proves that at least once during the loop that failed, a GC 
occurred as a result of the format call.

Next I tried uncommenting the concatenation line and commenting 
out the format call. That produces:
dca pangloss:~/Software/newcash_d/verifier$ ./verifier 
"--DRT-gcopt=profile:1" /tmp/Finances.newcash
walk_account_tree: 344, 344, :Assets:Investments:Equities and 
derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD 
Ameritrade, 0, 0
         Number of collections:  2
         Total GC prep time:  0 milliseconds
         Total mark time:  0 milliseconds
         Total sweep time:  0 milliseconds
         Max Pause Time:  0 milliseconds
         Grand total GC time:  0 milliseconds
GC summary:    5 MB,    2 GC    0 ms, Pauses    0 ms <    0 ms
dca pangloss:~/Software/newcash_d/verifier$

So again, using concatenation instead of format fixes the 
problem. And no GCs have occurred after finishing the loop on the 
account at issue.

And one other tidbit. If I add a precision to %s for the path in 
the format call, like so and change the test for the problem 
account by checking for its
guid rather than the truncated path:
         // Now do the children of this account
         // First determine how many there are
         bind_text(count_children, 1, account.guid);
         int n_children = one_row(count_children, 
&get_int).integer_value;
         if (n_children > 0)
         {
             size_t collections_before = 0;
             size_t collections_after = 0;
             Account[] children = new Account[n_children];
             bind_text(find_children, 1, account.guid);
             int i = 0;
             while (next_row_available_p(find_children, 
&reset_stmt))
             {
                 children[i].name = 
fromStringz(sqlite3_column_text(find_children, 0)).dup;
                 collections_before += 
GC.profileStats().numCollections;
                 //children[i].path = (account.path ~ ":" ~ 
children[i].name);
                 children[i].path = format("%s:%.10s", 
account.path, children[i].name);
                 collections_after += 
GC.profileStats().numCollections;
               children[i].guid = 
fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid
                 children[i].commodity_guid = 
fromStringz(sqlite3_column_text(find_children, 2)).dup; // 
commodity_guid
                 children[i].flags = 
sqlite3_column_int(find_children,
                         3) | account.flags & 
(account_flag_descendents_are_assets
                         | 
account_flag_descendents_are_liabilities | 
account_flag_descendents_are_income
                         | account_flag_descendents_are_expenses | 
account_flag_descendents_are_marketable
                         | 
account_flag_self_and_descendents_are_tax_related
                         | 
account_flag_descendents_need_commodity_link); // flags
                 i = i + 1;
             }
             if (i != n_children)
                 panic(format("walk_account_tree: bad child index, 
%d, %d, %s, %d, %d", i, n_children, account.path, 
collections_before, collections_after));
             if (account.guid == 
"c369abf6ec2c7222e3fdd174ce2c0c9a") {
                 writeln(format("walk_account_tree: %d, %d, %s, 
%d, %d", i, n_children, account.path, collections_before, 
collections_after));
             }
             foreach (child; children) {
                 walk_account_tree(child, ancestor_flags | 
account.flags);
             }
         }

I get:
dca pangloss:~/Software/newcash_d/verifier$ ./verifier 
"--DRT-gcopt=profile:1" /tmp/Finances.newcash
walk_account_tree: 344, 344, :Assets:Investment:Equities 
a:Taxable:Donald C. :TD Ameritr, 0, 0
         Number of collections:  3
         Total GC prep time:  0 milliseconds
         Total mark time:  0 milliseconds
         Total sweep time:  0 milliseconds
         Max Pause Time:  0 milliseconds
         Grand total GC time:  1 milliseconds
GC summary:    5 MB,    3 GC    1 ms, Pauses    1 ms <    0 ms
dca pangloss:~/Software/newcash_d/verifier$

No panic. So there appears to be a problem in format when dealing 
with strings fed to an unconstrained %s, probably when 
interacting the garbage collector. My guess is that the issue is
in format, not the gc.

That's as far as I've gotten. But I think we know more now than 
before.
Oct 24 2020
next sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Saturday, 24 October 2020 at 14:21:52 UTC, donallen wrote:

 On Friday, 23 October 2020 at 23:41:05 UTC, rikki cattermole 
 wrote:

 [...]


 No. Read on.

 [...]


Great writeup. Thanks for taking the time me to do it! (Iirc 
there's a nogc format somewhere in 
https://code.dlang.org/packages/ocean)
Oct 24 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/24/20 10:21 AM, donallen wrote:

 On Friday, 23 October 2020 at 23:41:05 UTC, rikki cattermole wrote:

 Thanks!

 The problem is not in that block of code I believe.

 Now how about bind_text?

 Is that still using c variadic args?


 
 No. Read on.
 
 The detective in me couldn't resist (I really need a Peugeot and a 
 basset hound) and so I did some more work on this despite not really 
 having the time and I have made some progress.


Very nice!

I wonder if the corruption is happening on the account.guid. Since we 
are talking about sqlite, which is a local DB, and not sending 
information to a server (where it can't be corrupted), it's possible the 
guid is being corrupted causing the premature exit of the loop.

Can you print out the guid after in the premature exit case to make sure 
it hasn't changed?


 This does not panic! In theory, this should consume the same amount of 
 GC memory as the format call (though I have not looked at the format 
 code; there may well
 be intermediate allocations as it does its work).


Just for informational purposes, an append statement between N strings 
allocates one block that holds all of them. Format appends, so it may 
involve multiple allocations, though it's possible it does a similar 
shortcut. I'm very interested in this code, I will look at the case for 
format, and see if I can sleuth out any possible corruption possibilities.


 
 No panic. So there appears to be a problem in format when dealing with 
 strings fed to an unconstrained %s, probably when interacting the 
 garbage collector. My guess is that the issue is
 in format, not the gc.


Yeah, I tend to agree. More specifically, it could be in appender as 
well (which is what format uses to allocate memory).


 That's as far as I've gotten. But I think we know more now than before.


Good work, and I think we can probably get to the bottom of it, lots of 
great information!

-Steve
Oct 24 2020
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/24/20 11:04 AM, Steven Schveighoffer wrote:

 On 10/24/20 10:21 AM, donallen wrote:

 This does not panic! In theory, this should consume the same amount of 
 GC memory as the format call (though I have not looked at the format 
 code; there may well
 be intermediate allocations as it does its work).


 
 Just for informational purposes, an append statement between N strings 
 allocates one block that holds all of them. Format appends, so it may 
 involve multiple allocations, though it's possible it does a similar 
 shortcut. I'm very interested in this code, I will look at the case for 
 format, and see if I can sleuth out any possible corruption possibilities.


Can you try this? This will remove format from the picture, but still 
use appender in the same way format does (to focus on whether the 
problem is in std.format or appender):

// instead of this
// children[i].path = format("%s:%s", account.path, children[i].name);

{ // need a scope to ensure appender goes out of scope
    import std.array;
    appender!string app;
    app.put(account.path);
    app.put(":");
    app.put(children[i].name);
    children[i].path = app[];
}

If this still panics, then I can eliminate all of std.format. If it 
doesn't panic, then it's most likely in std.format.

A closer look at appender, and it seems all correct. I'm kinda hoping 
this doesn't panic, but at the same time, the appender code is a lot 
less complex, so easier to examine closely.

-Steve
Oct 24 2020
prev sibling parent reply donallen <donaldcallen gmail.com> writes:
On Saturday, 24 October 2020 at 15:04:53 UTC, Steven 
Schveighoffer wrote:

 On 10/24/20 10:21 AM, donallen wrote:

 On Friday, 23 October 2020 at 23:41:05 UTC, rikki cattermole 
 wrote:

 Thanks!

 The problem is not in that block of code I believe.

 Now how about bind_text?

 Is that still using c variadic args?


 
 No. Read on.
 
 The detective in me couldn't resist (I really need a Peugeot 
 and a basset hound) and so I did some more work on this 
 despite not really having the time and I have made some 
 progress.


 Very nice!

 I wonder if the corruption is happening on the account.guid. 
 Since we are talking about sqlite, which is a local DB, and not 
 sending information to a server (where it can't be corrupted), 
 it's possible the guid is being corrupted causing the premature 
 exit of the loop.

 Can you print out the guid after in the premature exit case to 
 make sure it hasn't changed?


A good thought, but it doesn't change:

         // Now do the children of this account
         // First determine how many there are
         string guid_before = account.guid;
         bind_text(count_children, 1, account.guid);
         int n_children = one_row(count_children, 
&get_int).integer_value;
         if (n_children > 0)
         {
             size_t collections_before = 0;
             size_t collections_after = 0;
             Account[] children = new Account[n_children];
             string guid_after = account.guid;
             bind_text(find_children, 1, account.guid);
             int i = 0;
             while (next_row_available_p(find_children, 
&reset_stmt))
             {
                 children[i].name = 
fromStringz(sqlite3_column_text(find_children, 0)).dup;
                 collections_before += 
GC.profileStats().numCollections;
                 //children[i].path = (account.path ~ ":" ~ 
children[i].name);
                 children[i].path = format("%s:%s", account.path, 
children[i].name);
                 collections_after += 
GC.profileStats().numCollections;
               children[i].guid = 
fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid
                 children[i].commodity_guid = 
fromStringz(sqlite3_column_text(find_children, 2)).dup; // 
commodity_guid
                 children[i].flags = 
sqlite3_column_int(find_children,
                         3) | account.flags & 
(account_flag_descendents_are_assets
                         | 
account_flag_descendents_are_liabilities | 
account_flag_descendents_are_income
                         | account_flag_descendents_are_expenses | 
account_flag_descendents_are_marketable
                         | 
account_flag_self_and_descendents_are_tax_related
                         | 
account_flag_descendents_need_commodity_link); // flags
                 i = i + 1;
             }
             if (i != n_children)
                 panic(format("walk_account_tree: bad child index, 
%d, %d, %s, %s, %s, %d, %d", i, n_children, account.path, 
guid_before, guid_after, collections_before, collections_after));
             if (account.guid == 
"c369abf6ec2c7222e3fdd174ce2c0c9a") {
                 writeln(format("walk_account_tree: %d, %d, %s, 
%d, %d", i, n_children, account.path, collections_before, 
collections_after));
             }
             foreach (child; children) {
                 walk_account_tree(child, ancestor_flags | 
account.flags);
             }


dca pangloss:~/Software/newcash_d/verifier$ ./verifier 
"--DRT-gcopt=profile:1" /tmp/Finances.newcash
walk_account_tree: bad child index, 231, 344, 
:Assets:Investments:Equities and derivatives:Taxable:Donald C. 
Allen 2003 Revocable Trust:TD Ameritrade, 
c369abf6ec2c7222e3fdd174ce2c0c9a, 
c369abf6ec2c7222e3fdd174ce2c0c9a, 33, 34


 This does not panic! In theory, this should consume the same 
 amount of GC memory as the format call (though I have not 
 looked at the format code; there may well
 be intermediate allocations as it does its work).


 Just for informational purposes, an append statement between N 
 strings allocates one block that holds all of them. Format 
 appends, so it may involve multiple allocations, though it's 
 possible it does a similar shortcut. I'm very interested in 
 this code, I will look at the case for format, and see if I can 
 sleuth out any possible corruption possibilities.


 
 No panic. So there appears to be a problem in format when 
 dealing with strings fed to an unconstrained %s, probably when 
 interacting the garbage collector. My guess is that the issue 
 is
 in format, not the gc.


 Yeah, I tend to agree. More specifically, it could be in 
 appender as well (which is what format uses to allocate memory).


 That's as far as I've gotten. But I think we know more now 
 than before.


 Good work, and I think we can probably get to the bottom of it, 
 lots of great information!

 -Steve
Oct 24 2020
parent reply Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Saturday, 24 October 2020 at 16:00:24 UTC, donallen wrote:

 On Saturday, 24 October 2020 at 15:04:53 UTC, Steven 
 Schveighoffer wrote:

 [...]


 A good thought, but it doesn't change:

 [...]


Have you tried formattedWrite or the nogc package?
Oct 24 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Saturday, 24 October 2020 at 20:00:42 UTC, Imperatorn wrote:

 On Saturday, 24 October 2020 at 16:00:24 UTC, donallen wrote:

 On Saturday, 24 October 2020 at 15:04:53 UTC, Steven 
 Schveighoffer wrote:

 [...]


 A good thought, but it doesn't change:

 [...]


 Have you tried formattedWrite or the nogc package?


No. At this point, the verifier works and I have stopped further 
porting to D of my financial tools.
Oct 25 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Sunday, 25 October 2020 at 17:03:00 UTC, donallen wrote:

 On Saturday, 24 October 2020 at 20:00:42 UTC, Imperatorn wrote:

 On Saturday, 24 October 2020 at 16:00:24 UTC, donallen wrote:

 On Saturday, 24 October 2020 at 15:04:53 UTC, Steven 
 Schveighoffer wrote:

 [...]


 A good thought, but it doesn't change:

 [...]


 Have you tried formattedWrite or the nogc package?


 No. At this point, the verifier works and I have stopped 
 further porting to D of my financial tools.


I should clarify that I've stopped further work with D because of 
the issue that came up in trying to port my verifier application. 
I am hoping that the information I provided over the weekend will 
enable someone to come up with a definitive explanation of what 
happened. If that occurs, I will then decide whether I want to 
continue porting my code to D.

I will add that I am impressed with the language -- a big 
improvement over C and C++, and with the documentation and the 
helpfulness of the community. I'm just concerned, based on the 
verifier experience, that D doesn't have the critical mass needed 
to make a complex system rock solid. Perhaps the explanation of 
the verifier problem will change that.
Oct 26 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/26/20 10:17 AM, donallen wrote:

 I should clarify that I've stopped further work with D because of the 
 issue that came up in trying to port my verifier application. I am 
 hoping that the information I provided over the weekend will enable 
 someone to come up with a definitive explanation of what happened. If 
 that occurs, I will then decide whether I want to continue porting my 
 code to D.


Thanks for your reports. Unfortunately, without something reproducible 
locally, there's not much we can do without your help. As you know I'm 
sure, I can't fix problems without being able to test solutions. So far, 
nothing has jumped out as an obvious error in your code.

If you have time to run some more tests, then I would appreciate it. In 
particular, if you could try this test, it could narrow it down a bit: 
https://forum.dlang.org/post/rn1ie3$2363$1 digitalmars.com

Alternatively, if you feel comfortable with it, you can send me the 
code/data that fails (or maybe dummy data that also fails), and I can 
keep looking at it.

My email is schveiguy at gmail dot com.

-Steve
Oct 26 2020
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
Basically what Steven said, is what I was going to reply.

We need to work with the full code, I understand that it is probably 
proprietary. If this is a concern, perhaps somebody associated with the 
D Language Foundation will be willing to offer some help, assuming that 
is some comfort.
Oct 26 2020
prev sibling parent reply donallen <donaldcallen gmail.com> writes:
On Monday, 26 October 2020 at 14:33:15 UTC, Steven Schveighoffer 
wrote:

 On 10/26/20 10:17 AM, donallen wrote:

 I should clarify that I've stopped further work with D because 
 of the issue that came up in trying to port my verifier 
 application. I am hoping that the information I provided over 
 the weekend will enable someone to come up with a definitive 
 explanation of what happened. If that occurs, I will then 
 decide whether I want to continue porting my code to D.


 Thanks for your reports. Unfortunately, without something 
 reproducible locally, there's not much we can do without your 
 help. As you know I'm sure, I can't fix problems without being 
 able to test solutions. So far, nothing has jumped out as an 
 obvious error in your code.

 If you have time to run some more tests, then I would 
 appreciate it. In particular, if you could try this test, it 
 could narrow it down a bit: 
 https://forum.dlang.org/post/rn1ie3$2363$1 digitalmars.com


Sorry, I missed this suggestion in your earlier message. I'll 
give it a try. I'll let you know what happens.


 Alternatively, if you feel comfortable with it, you can send me 
 the code/data that fails (or maybe dummy data that also fails), 
 and I can keep looking at it.


As I said earlier, the code is not the issue, it's the data. If 
we get to this bridge and have to cross it, I'll look trying to 
cook up a dummy database that provokes the problem.


 My email is schveiguy at gmail dot com.

 -Steve
Oct 26 2020
parent reply donallen <donaldcallen gmail.com> writes:
With the help of an excellent suggestion by Steve Schveighoffer, 
I finally identified this problem yesterday. It was my error, not 
D's.

I will spare you all the details. The problem is in this function:

void bind_text(sqlite3_stmt* stmt, int index, const char[] 
the_text)
{
     if (sqlite3_bind_text(stmt, index, toStringz(the_text), -1, 
null)
!= sqlite_ok)
     {
         stderr.writeln("bind_text failed");
         auto db = sqlite3_db_handle(stmt);
         stderr.writeln("Error: %s", 
fromStringz(sqlite3_errmsg(db)));
         exit(EXIT_FAILURE);
     }
}

which takes a sqlite prepared statement, a query parameter index 
and a D string and attempts to convert the D string to a 
null-terminated C string and call sqlite3_bind_text to bind the C 
string to the indicated query parameter.

The culprit is the call to toStringz. The documentation for that
function give some good advice that I failed to follow:
"Important Note: When passing a char* to a C function, and the C
function keeps it around for any reason, make sure that you keep a
reference to it in your D code. Otherwise, it may become invalid
during a garbage collection cycle and cause a nasty bug when the C
code tries to use it."

That's precisely what is happening here. There is no reference to 
the
string returned by the toStringz call retained and if a GC happens
during a loop retrieving all the rows of a query, the C version 
of the
string that has been passed to sqlite by the sqlite3_bind_text 
call gets
freed by the GC before sqlite is done with it. So now the sqlite 
query
no longer has a proper value for the query parameter, and all 
bets are
off as far as the query behaving correctly. This also explains 
the mystery of why the loop
terminated prematurely when I first discovered this problem.

I fixed it by having bind_text return the C string to its caller, 
to be stored in a variable to protect it from the garbage 
collector. The original error, which was reproducible, is now 
gone. Steve's suggestion, which was to insert frequent calls to 
the garbage collector, provoked a different error reliably that 
is also gone with the fix.

Thanks to everyone who tried to help figure this one out.
Nov 15 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Nov 15, 2020 at 01:43:14PM +0000, donallen via Digitalmars-d wrote:

 With the help of an excellent suggestion by Steve Schveighoffer, I
 finally identified this problem yesterday. It was my error, not D's.


[...]

 void bind_text(sqlite3_stmt* stmt, int index, const char[] the_text)
 {
     if (sqlite3_bind_text(stmt, index, toStringz(the_text), -1, null)
 != sqlite_ok)
     {
         stderr.writeln("bind_text failed");
         auto db = sqlite3_db_handle(stmt);
         stderr.writeln("Error: %s", fromStringz(sqlite3_errmsg(db)));
         exit(EXIT_FAILURE);
     }
 }


[...]

 The culprit is the call to toStringz. The documentation for that
 function give some good advice that I failed to follow:
 "Important Note: When passing a char* to a C function, and the C
 function keeps it around for any reason, make sure that you keep a
 reference to it in your D code. Otherwise, it may become invalid
 during a garbage collection cycle and cause a nasty bug when the C
 code tries to use it."


Ouch.  I'm dismayed that I missed this when I last looked at this code
(which I believe you did post here). ;-)  Passing a GC-allocated string
to C code without retaining a reference somewhere on the D side is a
known gotcha when interfacing with C.  In fact, I'm almost certain I ran
into this very problem with toStringz myself in the past, yet I still
failed to notice it when I looked at your code.

In any case, I'm glad that this means you'll be around here more. ;-)


--T
Nov 15 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/15/20 10:05 AM, H. S. Teoh wrote:

 On Sun, Nov 15, 2020 at 01:43:14PM +0000, donallen via Digitalmars-d wrote:

 With the help of an excellent suggestion by Steve Schveighoffer, I
 finally identified this problem yesterday. It was my error, not D's.


 [...]

 void bind_text(sqlite3_stmt* stmt, int index, const char[] the_text)
 {
      if (sqlite3_bind_text(stmt, index, toStringz(the_text), -1, null)
 != sqlite_ok)
      {
          stderr.writeln("bind_text failed");
          auto db = sqlite3_db_handle(stmt);
          stderr.writeln("Error: %s", fromStringz(sqlite3_errmsg(db)));
          exit(EXIT_FAILURE);
      }
 }


 [...]

 The culprit is the call to toStringz. The documentation for that
 function give some good advice that I failed to follow:
 "Important Note: When passing a char* to a C function, and the C
 function keeps it around for any reason, make sure that you keep a
 reference to it in your D code. Otherwise, it may become invalid
 during a garbage collection cycle and cause a nasty bug when the C
 code tries to use it."


 
 Ouch.  I'm dismayed that I missed this when I last looked at this code
 (which I believe you did post here). ;-)  Passing a GC-allocated string
 to C code without retaining a reference somewhere on the D side is a
 known gotcha when interfacing with C.  In fact, I'm almost certain I ran
 into this very problem with toStringz myself in the past, yet I still
 failed to notice it when I looked at your code.


I had the same reaction! I also did not see the toStringz as a problem 
when I reviewed.

I will note, also, that toStringz has this quirk where in certain cases 
it just returns the pointer of the array, which is why I think the .idup 
fix worked (that was puzzling to me and I think pulled me away from it 
being a problem with storing a pointer in C-land only).

https://github.com/dlang/phobos/blob/e89d9d9ce072269e80da4f901ce3e953d736cb82/std/string.d#L361-L367


 In any case, I'm glad that this means you'll be around here more. ;-)


Likewise!

-Steve
Nov 15 2020
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 11/15/20 10:43 AM, Steven Schveighoffer wrote:


 I also did not see the toStringz as a problem
 when I reviewed.


While I have everbody's attention, :) toStringz will come up during one 
of my DConf presentations. I will claim that it is safe to pass to a C 
function for their immediate consumption (but they can't save the pointer).

So, I will claim that the following is safe and no GC collection on 
another thread can mess this up. (The C side is not allowed to store for 
later use but they are taking the pointer into a local variable on their 
function call stack.)

nothrow extern(C) void bar(const(char) ** name) {
   // ...
   *name = makeString(42).toStringz;
   // ...
}

My claim is based on my understanding on the following thread. (Thank 
you, Rikki Cattermole.)

   https://forum.dlang.org/thread/rn3ii0$jjj$1 digitalmars.com

Ali
Nov 15 2020
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Nov 15, 2020 at 01:46:52PM -0800, Ali ehreli via Digitalmars-d wrote:
[...]

 While I have everbody's attention, :) toStringz will come up during
 one of my DConf presentations. I will claim that it is safe to pass to
 a C function for their immediate consumption (but they can't save the
 pointer).
 
 So, I will claim that the following is safe and no GC collection on
 another thread can mess this up. (The C side is not allowed to store
 for later use but they are taking the pointer into a local variable on
 their function call stack.)
 
 nothrow extern(C) void bar(const(char) ** name) {
   // ...
   *name = makeString(42).toStringz;
   // ...
 }


[...]

AFAIK, as long as a reference exists on the stack, you should be safe,
because the GC does scan the stack.  But if the C side stores it
anywhere else past the point where the on-stack reference goes out of
scope, then you're in trouble.


T

-- 
Chance favours the prepared mind. -- Louis Pasteur
Nov 16 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/16/20 2:26 PM, H. S. Teoh wrote:

 On Sun, Nov 15, 2020 at 01:46:52PM -0800, Ali Çehreli via Digitalmars-d wrote:
 [...]

 While I have everbody's attention, :) toStringz will come up during
 one of my DConf presentations. I will claim that it is safe to pass to
 a C function for their immediate consumption (but they can't save the
 pointer).

 So, I will claim that the following is safe and no GC collection on
 another thread can mess this up. (The C side is not allowed to store
 for later use but they are taking the pointer into a local variable on
 their function call stack.)

 nothrow extern(C) void bar(const(char) ** name) {
    // ...
    *name = makeString(42).toStringz;
    // ...
 }


 [...]
 
 AFAIK, as long as a reference exists on the stack, you should be safe,
 because the GC does scan the stack.  But if the C side stores it
 anywhere else past the point where the on-stack reference goes out of
 scope, then you're in trouble.


I see a problem here, in that the stack is changeable by the C function.

I can imagine a situation where it removes the item from the stack to 
put it somewhere else (e.g. global memory or on the C heap) 
*temporarily*, and by the time the function exits, the pointer is no 
longer in use in C-land. Thus, the function doesn't "save the pointer".

This kind of problem would still be susceptible to the GC collecting it 
(via another thread), and so I would recommend still storing locally in 
the calling function a copy of the pointer.

-Steve
Nov 16 2020
next sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 11/16/20 1:25 PM, Steven Schveighoffer wrote:


 I can imagine a situation where it removes the item from the stack to
 put it somewhere else (e.g. global memory or on the C heap)
 *temporarily*


Good point. For example, they can provide the address of the member of a 
heap-allocated object:

   Obj * obj = malloc(/* ... */);
   D_function(&obj.member);
                                // <-- Another thread triggers a
                                //     GC collection at this point.

   printf("%d", obj.member);    // <-- Potentially too late.

This is hard to communicate to the callers. I said "safe to use for 
immediate consumption" but still not safe. Documenting like "pointer to 
stack allocated data only please" may be too confusing for casual users. 
So, the safest thing would be to make a copy with toStringz.

Of course, strings are just one type. I pass GC allocated large buffers 
to the C side without copying. Luckily, I am the only consumer so far 
and things seem to work. :)

Ali
Nov 16 2020
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 16 November 2020 at 21:25:03 UTC, Steven Schveighoffer 
wrote:

 I can imagine a situation where it removes the item from the 
 stack to put it somewhere else (e.g. global memory or on the C 
 heap) *temporarily*, and by the time the function exits, the 
 pointer is no longer in use in C-land. Thus, the function 
 doesn't "save the pointer".

 This kind of problem would still be susceptible to the GC 
 collecting it (via another thread), and so I would recommend 
 still storing locally in the calling function a copy of the 
 pointer.


How do you express that? LLVM can remove unused pointers? There 
are intrinsics to combat that, but is there something portable in 
D?
Nov 16 2020
prev sibling next sibling parent mw <mingwu gmail.com> writes:
On Sunday, 15 November 2020 at 13:43:14 UTC, donallen wrote:

 With the help of an excellent suggestion by Steve 
 Schveighoffer, I finally identified this problem yesterday. It 
 was my error, not D's.


Glad you fixed the problem.

And it looks like you missed my post:

https://forum.dlang.org/post/hqpppumvdtkbmghavsoq forum.dlang.org


D has been around for ~20 years, and there are 1899 packages, 
some of them under heavy industrial use, it's very unlikely it's 
a GC bug, if there is a bug, people may run into it long before 
you do esp you are new to D.

https://code.dlang.org/
Nov 15 2020
prev sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
Wahoo!
Nov 15 2020
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Oct 24, 2020 at 02:21:52PM +0000, donallen via Digitalmars-d wrote:
[...]

 The detective in me couldn't resist (I really need a Peugeot and a
 basset hound) and so I did some more work on this despite not really
 having the time and I have made some progress.


[...]

 Something has caused sqlite to end the return of rows prematurely. But
 next_row_p did not panic, which it will if the the return code from
 sqlite3_step, which it calls, is other than SQLITE_ROW or SQLITE_DONE.
 Since next_row_p just returned 'false', sqlite3_step must have
 returned SQLITE_DONE. sqlite returned 231 of 344 rows but is acting
 like everything is normal.


[...]

Very interesting finding indeed!



 Then I had another of my hunches. I re-wrote the line containing the
 call to format like so:


[...]

                 children[i].path = (account.path ~ ":" ~ children[i].name);


[...]

 This does not panic! In theory, this should consume the same amount of
 GC memory as the format call (though I have not looked at the format
 code; there may well be intermediate allocations as it does its work).


Am I right in assuming that account.path and children[i].name are both
of type `string`?  As opposed to some other object that defined a
toString method?

AFAIK, format does allocate intermediate buffers for its work; that may
explain the difference you observed here.



 Now I decided to instrument the code so I could determine whether GCs
 were occurring as a result of the call to format:


[...]

 Running this produces:
 dca pangloss:~/Software/newcash_d/verifier$ ./verifier
 "--DRT-gcopt=profile:1" /tmp/Finances.newcash
 walk_account_tree: bad child index, 231, 344, :Assets:Investments:Equities
 and derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade,
 33, 34
 dca pangloss:~/Software/newcash_d/verifier$
 
 This proves that at least once during the loop that failed, a GC
 occurred as a result of the format call.


[...]

Here, there appears to have been 33 GC collections before the format
call, and 34 after. Am I reading this right?



 Next I tried uncommenting the concatenation line and commenting out
 the format call. That produces:
 dca pangloss:~/Software/newcash_d/verifier$ ./verifier
 "--DRT-gcopt=profile:1" /tmp/Finances.newcash
 walk_account_tree: 344, 344, :Assets:Investments:Equities and
 derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade, 0, 0


[...]

 So again, using concatenation instead of format fixes the problem. And
 no GCs have occurred after finishing the loop on the account at issue.


However, here there appears to have been no collections at all (0
before, 0 after)!  So based on this we still cannot rule out a problem
in the GC.

Which makes me wonder: what would happen if you inserted:

        if (account.path == ":Assets:Investments:Equities and
derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade") 
		GC.collect();

right after the concatenation line to trigger a collection?  Would it
reintroduce the panic?



 And one other tidbit. If I add a precision to %s for the path in the
 format call, like so and change the test for the problem account by
 checking for its guid rather than the truncated path:


[...]

                 children[i].path = format("%s:%.10s", account.path,
 children[i].name);


[...]

 I get:
 dca pangloss:~/Software/newcash_d/verifier$ ./verifier
 "--DRT-gcopt=profile:1" /tmp/Finances.newcash
 walk_account_tree: 344, 344, :Assets:Investment:Equities a:Taxable:Donald C.
 :TD Ameritr, 0, 0


[...]

 No panic. So there appears to be a problem in format when dealing with
 strings fed to an unconstrained %s, probably when interacting the
 garbage collector. My guess is that the issue is in format, not the
 gc.


Interestingly enough, again there appears to be no collections done at
all (0 before, 0 after).  So I'm still not sure if we could entirely
rule out the GC at this point.  It would be more interesting to find out
what happens if you insert the GC.collect() line to manually trigger a
collection with the constrained version of the format call.



 That's as far as I've gotten. But I think we know more now than
 before.


This is extremely informative, and thank you for taking the time to do
this in spite of being very busy.  I hope we can eventually ferret out
the cause of the problem, whether it's in the GC, in format, or in your
code somehow.


T

-- 
Error: Keyboard not attached. Press F1 to continue. -- Yoon Ha Lee, CONLANG
Oct 24 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/21/20 12:24 PM, donallen wrote:

 The problem is that at some point, the verifier starts spewing bogus 
 error messages about what it is seeing in the tree. Oddly, putting in 
 debugging writelns results in the error messages not occurring -- a 
 Heisenbug.  But working with gdb, I found that the account structs after 
 the error messages start are zeroed. Turning on gc profiling tells me 
 that 3 gcs have occurred. Disabling the gc results in the program 
 running correctly -- no error messages (and I know the database has no 
 errors because the C version, which has been around for awhile, confirms 
 that the db is well formed).
 
 I could post a PR, but I'm not sure that this is a bug. It could easily 
 be a misunderstanding by me of D and its memory management. So I thought 
 I'd try this post first, hoping that one of you who knows the language 
 better than I do could point out a problem with my code. I do need to 
 resolve this or abandon this project.
 
 Thanks in advance for any help.


There seems to be nothing wrong in this code that I can see.

But of course, it's out of context, so it's hard to see if there are 
issues elsewhere.

These kinds of corruption bugs are really hard to track down.

What does your Account struct look like?

-Steve
Oct 22 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 18:24:57 UTC, Steven 
Schveighoffer wrote:

 On 10/21/20 12:24 PM, donallen wrote:

 The problem is that at some point, the verifier starts spewing 
 bogus error messages about what it is seeing in the tree. 
 Oddly, putting in debugging writelns results in the error 
 messages not occurring -- a Heisenbug.  But working with gdb, 
 I found that the account structs after the error messages 
 start are zeroed. Turning on gc profiling tells me that 3 gcs 
 have occurred. Disabling the gc results in the program running 
 correctly -- no error messages (and I know the database has no 
 errors because the C version, which has been around for 
 awhile, confirms that the db is well formed).
 
 I could post a PR, but I'm not sure that this is a bug. It 
 could easily be a misunderstanding by me of D and its memory 
 management. So I thought I'd try this post first, hoping that 
 one of you who knows the language better than I do could point 
 out a problem with my code. I do need to resolve this or 
 abandon this project.
 
 Thanks in advance for any help.


 There seems to be nothing wrong in this code that I can see.

 But of course, it's out of context, so it's hard to see if 
 there are issues elsewhere.

 These kinds of corruption bugs are really hard to track down.

 What does your Account struct look like?


     struct Account
     {
         string name;
         string path;
         string guid;
         string commodity_guid;
         int flags;
     }



 -Steve
Oct 22 2020
next sibling parent reply mw <mingwu gmail.com> writes:
On Thursday, 22 October 2020 at 19:34:18 UTC, donallen wrote:

 What does your Account struct look like?


     struct Account
     {
         string name;
         string path;
         string guid;
         string commodity_guid;
         int flags;
     }


Do you pass any variable from D to C? and how does those 
variables look like?

And if you are passing any D object to C, do you always hold a 
reference to them before the C returns? I mean: the D's gc think 
the object is no longer used, but actually is used by the C side.
Oct 22 2020
parent mw <mingwu gmail.com> writes:
On Thursday, 22 October 2020 at 19:57:08 UTC, mw wrote:

 Do you pass any variable from D to C? and how does those 
 variables look like?

 And if you are passing any D object to C, do you always hold a 
 reference to them before the C returns? I mean: the D's gc


s/before/after


 think the object is no longer used, but actually is used by the 
 C side.
Oct 22 2020
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 22 October 2020 at 19:34:18 UTC, donallen wrote:

     struct Account
     {
         string name;
         string path;
         string guid;
         string commodity_guid;
         int flags;
     }


Why did you call walk_account_tree on this, what does it do? 
Account has no pointers?
Oct 22 2020
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/21/20 12:24 PM, donallen wrote:


 The problem is that at some point, the verifier starts spewing bogus 
 error messages about what it is seeing in the tree. Oddly, putting in 
 debugging writelns results in the error messages not occurring -- a 
 Heisenbug.  But working with gdb, I found that the account structs after 
 the error messages start are zeroed. Turning on gc profiling tells me 
 that 3 gcs have occurred. Disabling the gc results in the program 
 running correctly -- no error messages (and I know the database has no 
 errors because the C version, which has been around for awhile, confirms 
 that the db is well formed).


Something I thought of - try running manual GC collections:

import core.memory;

GC.collect(); // peppered around your code

It might make your code fail closer to the true problem.

-Steve
Oct 23 2020