www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Either I'm confused or the gc is

reply donallen <donaldcallen gmail.com> writes:
I'm new to D, but not to programming (I wrote my first line of 
code 60 years ago and am retired from a long career as a 
developer and project manager).

I have a suite of personal finance applications that I wrote in 
C. I'd like to port them to D. The financial data is stored in a 
sqlite database. I spent yesterday re-writing the verifier 
application in D and have run into a perplexing issue.

The verifier does a recursive descent of the tree of accounts, 
checking a bunch of things at each node. After the checks are 
done, a query is done to discover how many children the current 
account has, at which point it allocates, with 'new', a dynamic 
array of Account structs. It then goes into a loop, collecting 
information from the db about each of the children, which gets 
stored in one of the Account structs in the array. Finally, it 
does a
foreach through the array, doing a recursive call on the tree 
walker.

Here's the relevant code:
````
         // Now do the children of this account
         // First determine how many there are
         bind_text(count_children, 1, account.guid);
         int n_children = one_row(count_children, 
&get_int).integer_value;
         if (n_children > 0)
         {
             Account[] children = new Account[n_children];
             bind_text(find_children, 1, account.guid);
             int i = 0;
             while (next_row_available_p(find_children, 
&reset_stmt))
             {
                 if (i > n_children-1)
                     panic("walk_account_tree: child index exceeds 
n_children");
                 children[i].name = 
fromStringz(sqlite3_column_text(find_children, 0)).dup;
                 children[i].path = format("%s:%s", account.path, 
children[i].name);
                 children[i].guid = 
fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid
                 children[i].commodity_guid = 
fromStringz(sqlite3_column_text(find_children, 2)).dup; // 
commodity_guid
                 children[i].flags = 
sqlite3_column_int(find_children,
                         3) | account.flags & 
(account_flag_descendents_are_assets
                         | 
account_flag_descendents_are_liabilities | 
account_flag_descendents_are_income
                         | account_flag_descendents_are_expenses | 
account_flag_descendents_are_marketable
                         | 
account_flag_self_and_descendents_are_tax_related
                         | 
account_flag_descendents_need_commodity_link); // flags
                 i = i + 1;
             }
             foreach (k, ref child; children)
                 walk_account_tree(&child, ancestor_flags | 
account.flags);
         }
````

The problem is that at some point, the verifier starts spewing 
bogus error messages about what it is seeing in the tree. Oddly, 
putting in debugging writelns results in the error messages not 
occurring -- a Heisenbug.  But working with gdb, I found that the 
account structs after the error messages start are zeroed. 
Turning on gc profiling tells me that 3 gcs have occurred. 
Disabling the gc results in the program running correctly -- no 
error messages (and I know the database has no errors because the 
C version, which has been around for awhile, confirms that the db 
is well formed).

I could post a PR, but I'm not sure that this is a bug. It could 
easily be a misunderstanding by me of D and its memory 
management. So I thought I'd try this post first, hoping that one 
of you who knows the language better than I do could point out a 
problem with my code. I do need to resolve this or abandon this 
project.

Thanks in advance for any help.
Oct 21 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Oct 21, 2020 at 04:24:41PM +0000, donallen via Digitalmars-d wrote:
 I'm new to D, but not to programming (I wrote my first line of code 60
 years ago and am retired from a long career as a developer and project
 manager).
Welcome aboard! ;-) [...]
 ````
         // Now do the children of this account
         // First determine how many there are
         bind_text(count_children, 1, account.guid);
         int n_children = one_row(count_children, &get_int).integer_value;
         if (n_children > 0)
         {
             Account[] children = new Account[n_children];
             bind_text(find_children, 1, account.guid);
             int i = 0;
             while (next_row_available_p(find_children, &reset_stmt))
             {
                 if (i > n_children-1)
                     panic("walk_account_tree: child index exceeds
 n_children");
                 children[i].name =
 fromStringz(sqlite3_column_text(find_children, 0)).dup;
                 children[i].path = format("%s:%s", account.path,
 children[i].name);
                 children[i].guid =
 fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid
                 children[i].commodity_guid =
 fromStringz(sqlite3_column_text(find_children, 2)).dup; // commodity_guid
                 children[i].flags = sqlite3_column_int(find_children,
                         3) | account.flags &
 (account_flag_descendents_are_assets
                         | account_flag_descendents_are_liabilities |
 account_flag_descendents_are_income
                         | account_flag_descendents_are_expenses |
 account_flag_descendents_are_marketable
                         | account_flag_self_and_descendents_are_tax_related
                         | account_flag_descendents_need_commodity_link); //
 flags
                 i = i + 1;
             }
             foreach (k, ref child; children)
                 walk_account_tree(&child, ancestor_flags | account.flags);
         }
 ````
 
 The problem is that at some point, the verifier starts spewing bogus
 error messages about what it is seeing in the tree. Oddly, putting in
 debugging writelns results in the error messages not occurring -- a
 Heisenbug.  But working with gdb, I found that the account structs
 after the error messages start are zeroed. Turning on gc profiling
 tells me that 3 gcs have occurred.  Disabling the gc results in the
 program running correctly -- no error messages (and I know the
 database has no errors because the C version, which has been around
 for awhile, confirms that the db is well formed).
Without seeing the rest of your code, or a minimal (runnable) failing test case, it's hard to say for certain what the problem is. But if I were to guess, it looks like the GC is prematurely collecting your arrays, though I can't imagine why it would if you still retain references to it. One potential gotcha, since you mention that this is being ported from C, is that if you pass pointers to GC-allocated objects to C code which stores it somewhere, you need to make sure you retain a reference somewhere on the D side of things, or else inform the GC of the reference using core.memory.GC.addRoot.[1] Otherwise, since the GC may not be aware of where the C code stores the pointer, it may fail to find it and wrongly believe that the object is dead, and thereby collect it prematurely. [1] See: https://dlang.org/library/core/memory/gc.add_root.html Another potential problem is if your C code (or C-style code ported to D) obscures pointers, e.g., using the XOR trick to implement a doubly-linked list with only a single pointer field, the GC will not be able to discover the reference, and thus may wrongly mark the referenced object as dead. Another thing that stuck out to me while glancing over your code snippet, is that the `children` array doesn't seem to be stored anywhere; this means it will go out of scope at the end of the scope and possibly be collected, unless you retain at least one reference to an array element somewhere. I can't tell if this is important without knowing the rest of your code, but it's something to look into. (Though it puzzles me why this would be a problem, since obviously your code still has a reference to *something* in that array in order for the verifier code to check it. But it's something to look into if there are no other clues.)
 I could post a PR, but I'm not sure that this is a bug. It could
 easily be a misunderstanding by me of D and its memory management. So
 I thought I'd try this post first, hoping that one of you who knows
 the language better than I do could point out a problem with my code.
 I do need to resolve this or abandon this project.
[...] Have you identified what's causing the problem? I.e., what would you put in your PR? T -- He who does not appreciate the beauty of language is not worthy to bemoan its flaws.
Oct 21 2020
next sibling parent donallen <donaldcallen gmail.com> writes:
On Wednesday, 21 October 2020 at 18:55:50 UTC, H. S. Teoh wrote:
 On Wed, Oct 21, 2020 at 04:24:41PM +0000, donallen via 
 Digitalmars-d wrote:
 [...]
Welcome aboard! ;-) [...]
Essentially what I wrote in my post, if I became convinced by you folks that the problem was not mine.
 [...]
Oct 21 2020
prev sibling next sibling parent reply donallen <donaldcallen gmail.com> writes:
On Wednesday, 21 October 2020 at 18:55:50 UTC, H. S. Teoh wrote:
 On Wed, Oct 21, 2020 at 04:24:41PM +0000, donallen via 
 Digitalmars-d wrote:
 [...]
Welcome aboard! ;-) [...]
 [...]
Without seeing the rest of your code, or a minimal (runnable) failing test case, it's hard to say for certain what the problem is. But if I were to guess, it looks like the GC is prematurely collecting your arrays, though I can't imagine why it would if you still retain references to it. One potential gotcha, since you mention that this is being ported from C, is that if you pass pointers to GC-allocated objects to C code which stores it somewhere, you need to make sure you retain a reference somewhere on the D side of things, or else inform the GC of the reference using core.memory.GC.addRoot.[1] Otherwise, since the GC may not be aware of where the C code stores the pointer, it may fail to find it and wrongly believe that the object is dead, and thereby collect it prematurely.
I am not passing pointers to GC-allocated objects to C code. The only C code involved here is libsqlite3 and there the pointers I pass to it are those I obtained from it.
 [1] See: https://dlang.org/library/core/memory/gc.add_root.html

 Another potential problem is if your C code (or C-style code 
 ported to D) obscures pointers, e.g., using the XOR trick to 
 implement a doubly-linked list with only a single pointer 
 field, the GC will not be able to discover the reference, and 
 thus may wrongly mark the referenced object as dead.
Nope.
 Another thing that stuck out to me while glancing over your 
 code snippet, is that the `children` array doesn't seem to be 
 stored anywhere; this means it will go out of scope at the end 
 of the scope and possibly be collected, unless you retain at 
 least one reference to an array element somewhere.  I can't 
 tell if this is important without knowing the rest of your 
 code, but it's something to look into.  (Though it puzzles me 
 why this would be a problem, since obviously your code still 
 has a reference to *something* in that array in order for the 
 verifier code to check it. But it's something to look into if 
 there are no other clues.)
As I understand it from the documentation, dynamic arrays are represented as fat pointers that contain the array size and a pointer to the actual data in the heap. I assume, though the documentation doesn't say, that the size/pointer structure is on the stack. If so, every one of the children arrays has a pointer to it on the stack as the descent through the tree proceeds and therefore should not be garbage collectible until a particular call to the tree walker (of which I've provided the last bit of code, which deals with descending into the child accounts) returns to its caller. It does look to me like those arrays are being collected prematurely and incorrectly.
Oct 21 2020
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Oct 21, 2020 at 09:35:43PM +0000, donallen via Digitalmars-d wrote:
[...]
 As I understand it from the documentation, dynamic arrays are
 represented as fat pointers that contain the array size and a pointer
 to the actual data in the heap. I assume, though the documentation
 doesn't say, that the size/pointer structure is on the stack.
That is correct, since you declared the `children` array as a local variable.
 If so, every one of the children arrays has a pointer to it on the
 stack as the descent through the tree proceeds and therefore should
 not be garbage collectible until a particular call to the tree walker
 (of which I've provided the last bit of code, which deals with
 descending into the child accounts) returns to its caller.
Also correct. Barring a bug in the GC.
 It does look to me like those arrays are being collected prematurely
 and incorrectly.
It would appear to be so, but it's really hard to say without being able to reproduce the problem locally. Another possibility is memory corruption caused by incorrect translation of C code to D. Generally, this should be pretty rare: Walter designed D syntax to be similar to C syntax such that if it compiles, it should work exactly as it does in C, otherwise there should be a compile error. But there *are* corner cases where this may inadvertently happen. There are also some gotchas with interfacing D code with C libraries (sqlite, as you mentioned), which may appear to work at first but may introduce subtle memory problems if not used correctly. Of course, there *is* the possibility that there's a bug in the GC that causes it to collect prematurely, but IMO the chances of that are pretty low, because D arrays are one of its most-used core features, and if something so basic doesn't work correctly, we would have seen (and fixed) the problem a long time ago. At least, it would have surfaced in some of the large D projects out there. But I haven't heard of any such problem recently, except in connection with code that interfaces with C, where sometimes mistakes can happen and memory corruption results. But on the off-chance there *is* a bug in the GC: we'd greatly appreciate it if you could somehow reduce the problem to a minimal test case that we can reproduce locally so that we can investigate. T -- Не дорог подарок, дорога любовь.
Oct 21 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Wednesday, 21 October 2020 at 22:28:48 UTC, H. S. Teoh wrote:
 On Wed, Oct 21, 2020 at 09:35:43PM +0000, donallen via 
 Digitalmars-d wrote: [...]
 [...]
That is correct, since you declared the `children` array as a local variable.
 [...]
Also correct. Barring a bug in the GC.
 [...]
It would appear to be so, but it's really hard to say without being able to reproduce the problem locally. Another possibility is memory corruption caused by incorrect translation of C code to D. Generally, this should be pretty rare: Walter designed D syntax to be similar to C syntax such that if it compiles, it should work exactly as it does in C, otherwise there should be a compile error. But there *are* corner cases where this may inadvertently happen. There are also some gotchas with interfacing D code with C libraries (sqlite, as you mentioned), which may appear to work at first but may introduce subtle memory problems if not used correctly. Of course, there *is* the possibility that there's a bug in the GC that causes it to collect prematurely, but IMO the chances of that are pretty low, because D arrays are one of its most-used core features, and if something so basic doesn't work correctly, we would have seen (and fixed) the problem a long time ago. At least, it would have surfaced in some of the large D projects out there. But I haven't heard of any such problem recently, except in connection with code that interfaces with C, where sometimes mistakes can happen and memory corruption results. But on the off-chance there *is* a bug in the GC: we'd greatly appreciate it if you could somehow reduce the problem to a minimal test case that we can reproduce locally so that we can investigate.
I will try. My suspicion that there's a gc problem largely rests on the fact that when I run the program with the gc disabled, the problem goes away -- it behaves as it should. Do you know if there's any way to configure the gc so it tells you what it's doing? It would be great if it could be told to announce when it's running and what it's collecting. I did a little looking in the documentation yesterday for that and did not find anything. If you can't help in that area, I'll have a look at the gc source code.
 T
Oct 21 2020
parent Max Haughton <maxhaton gmail.com> writes:
On Wednesday, 21 October 2020 at 23:02:07 UTC, donallen wrote:
 On Wednesday, 21 October 2020 at 22:28:48 UTC, H. S. Teoh wrote:
 [...]
I will try. My suspicion that there's a gc problem largely rests on the fact that when I run the program with the gc disabled, the problem goes away -- it behaves as it should. Do you know if there's any way to configure the gc so it tells you what it's doing? It would be great if it could be told to announce when it's running and what it's collecting. I did a little looking in the documentation yesterday for that and did not find anything. If you can't help in that area, I'll have a look at the gc source code.
 [...]
There are some flags and options, but if you can't find one that fixes your bug I would use something like bpf or gdb to watch it manually (if possible). GDB supports d demangling.
Oct 21 2020
prev sibling parent Max Haughton <maxhaton gmail.com> writes:
On Wednesday, 21 October 2020 at 18:55:50 UTC, H. S. Teoh wrote:
 On Wed, Oct 21, 2020 at 04:24:41PM +0000, donallen via 
 Digitalmars-d wrote:
 [...]
Welcome aboard! ;-) [...]
I have some D bolted into unreal engine, to avoid upsetting the beast I malloc and copy every GC'd (not performance code so the GC is fine) buffer into the games allocator.
Oct 21 2020
prev sibling next sibling parent reply Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Wednesday, 21 October 2020 at 16:24:41 UTC, donallen wrote:
 I'm new to D, but not to programming (I wrote my first line of 
 code 60 years ago and am retired from a long career as a 
 developer and project manager).

 [...]
What's your i and n_children on panic?
Oct 21 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Wednesday, 21 October 2020 at 19:30:40 UTC, Imperatorn wrote:
 On Wednesday, 21 October 2020 at 16:24:41 UTC, donallen wrote:
 I'm new to D, but not to programming (I wrote my first line of 
 code 60 years ago and am retired from a long career as a 
 developer and project manager).

 [...]
What's your i and n_children on panic?
There is no panic. The program starts complaining about the account data that it is checking, spewing bogus error messages. This is because the Account structures have been zeroed at some point in the recursive descent. But I did set a breakpoint in gdb at the place where the error messages are issued, and i was something like 230 of an n_children of something like 350. I don't think either number is particularly significant, other than that this was an account that had a lot of children and so was likely to trigger a gc.
Oct 21 2020
parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Wednesday, 21 October 2020 at 21:26:36 UTC, donallen wrote:
 On Wednesday, 21 October 2020 at 19:30:40 UTC, Imperatorn wrote:
 On Wednesday, 21 October 2020 at 16:24:41 UTC, donallen wrote:
 I'm new to D, but not to programming (I wrote my first line 
 of code 60 years ago and am retired from a long career as a 
 developer and project manager).

 [...]
What's your i and n_children on panic?
There is no panic. The program starts complaining about the account data that it is checking, spewing bogus error messages. This is because the Account structures have been zeroed at some point in the recursive descent. But I did set a breakpoint in gdb at the place where the error messages are issued, and i was something like 230 of an n_children of something like 350. I don't think either number is particularly significant, other than that this was an account that had a lot of children and so was likely to trigger a gc.
Oh, sorry, I mis-read something 🐣 A divide and conquer approach would otherwise be to enable() and disable() GC at various places to narrow it down. Or even do a collect() and look at addrOf as stated earlier. Would be good to know if there really is a bug in the GC or not. I suspect not, but you can't know for sure until you try.
Oct 22 2020
prev sibling next sibling parent ikod <igor.khasilev gmail.com> writes:
On Wednesday, 21 October 2020 at 16:24:41 UTC, donallen wrote:
 I'm new to D, but not to programming (I wrote my first line of 
 code 60 years ago and am retired from a long career as a 
 developer and project manager).
Very cool!
 Thanks in advance for any help.
Most likely GC clears prematurely some of your data (as H.S.Teoh pointed out). You can try to disable GC for a single program run with --DRT-gcopt="disable:1" option. Just add it to your command line args and see what happens. Another investigation can be done using memory checker like 'valgrind'. Regards,
Oct 21 2020
prev sibling next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
Try adding:

import core.memory : GC;
Account[] children = new Account[n_children];

GC.addRoot(children.ptr);
scope(exit)
	GC.removeRoot(children.ptr);

With the GC turned on, does it still cause issues?
Also are you compiling for 64bit or 32bit?
Oct 21 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 00:53:55 UTC, rikki cattermole 
wrote:
 Try adding:

 import core.memory : GC;
 Account[] children = new Account[n_children];

 GC.addRoot(children.ptr);
 scope(exit)
 	GC.removeRoot(children.ptr);

 With the GC turned on, does it still cause issues?
 Also are you compiling for 64bit or 32bit?
I tried this. With the gc on, the problem still occurs. I am working on 64-bit Arch Linux systems, up-to-date, DMD64 D Compiler v2.094.0
Oct 21 2020
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 22/10/2020 5:02 PM, donallen wrote:
 I tried this. With the gc on, the problem still occurs.
It won't be the GC then. I would look at stack corruption, so maybe look into valgrind.
Oct 21 2020
prev sibling parent reply tsbockman <thomas.bockman gmail.com> writes:
On Thursday, 22 October 2020 at 04:02:10 UTC, donallen wrote:
 I tried this. With the gc on, the problem still occurs.

 I am working on 64-bit Arch Linux systems, up-to-date, DMD64 D 
 Compiler v2.094.0
Have you tried using a different compiler? I know with the big languages like C or Java the meme is "it's never a compiler bug", but D is developed with *far* fewer resources, and compiler bugs are, sadly, not rare. For example, DMD 2.094+ have this still-unfixed back-end regression that may possibly be causing your problem: https://issues.dlang.org/show_bug.cgi?id=21325 The LDC and GDC back-ends are much more reliable (and better in other ways) than DMD's because they are tested and maintained by much larger communities. I recommend giving the latest LDC a try, as well as a significantly older compiler of your choice, before assuming that the problem is in your code, or in the garbage collector.
Oct 22 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 18:54:24 UTC, tsbockman wrote:
 On Thursday, 22 October 2020 at 04:02:10 UTC, donallen wrote:
 I tried this. With the gc on, the problem still occurs.

 I am working on 64-bit Arch Linux systems, up-to-date, DMD64 D 
 Compiler v2.094.0
Have you tried using a different compiler?
Yes -- to no avail: dca franz:~/Software/newcash_d/verifier$ make cd ../library ; make make[1]: Entering directory '/home/dca/Software/newcash_d/library' make[1]: Nothing to be done for 'all'. make[1]: Leaving directory '/home/dca/Software/newcash_d/library' #dmd -g -profile=gc -I=../library -L='-lsqlite3' verifier.d ../library/lib.o ldc -g -I=../library -L='-lsqlite3' verifier.d ../library/lib.o ../library/lib.o:lib.d:function _D3std9exception__T7bailOutHTC9ExceptionZQwFNaNfAyamMAxaZv: error: undefined reference to '_d_throwdwarf' ../library/lib.o:lib.d:function _D3std3utf__T10decodeImplVbi1VEQBd8typecons__T4FlagVAyaa19_7573655265706c6163656d656e744463686172ZQCa 0TAxwZQDrFNaQkKmZw: error: undefined reference to '_d_throwdwarf' ../library/lib.o:lib.d:function _D3std6format__T14formattedWriteTSQBg5array__T8AppenderTAyaZQoTaTAaZQ aFNaNfKQBsMxAaQtZk: error: undefined reference to '_d_throwdwarf' ../library/lib.o:lib.d:function _D3std6format__T6getNthVAyaa13_696e7465676572207769647468SQCe6traits10isIntegral iTAaZQCsFNaNfkQmZi: error: undefined reference to '_d_throwdwarf' ../library/lib.o:lib.d:function _D3std6format__T13formatElementTSQBf5stdio4File17LockingTextWriterTAyaTaZQCfFNfKQBwQqMKxSQDjQDi__T 0FormatSpecTaZQpZv: error: undefined reference to '__dmd_begin_catch' ../library/lib.o:lib.d:function _D3std6format__T15formatValueImplTSQBh5stdio4File17LockingTextWriterTwTaZQCfFNfKQBuwMKxSQDiQDh__T 0FormatSpecTaZQpZv: error: undefined reference to '_memset32' ../library/lib.o:lib.d:DW.ref.__dmd_personality_v0: error: undefined reference to '__dmd_personality_v0' ../library/lib.o:lib.d:function _D3std9algorithm9iteration__T8splitterVAyaa6_61203d3d2062TQtTQwZQBjFQBdQBgZ6Result11__xopEqualsFKxSQDtQDsQDl__TQDeVQCya6_61203d3d2062TQDrTQDvZQE FQEdQEgZQDaKxQCiZb: error: undefined reference to '_D4core8internal5array8equality__T8__equalsTyaTyaZQqFNaNbNiNfMAyaMQeZb' ../library/lib.o:lib.d:function _D3std9algorithm9iteration__T8splitterVAyaa6_61203d3d2062TQtTQwZQBjFQBdQBgZ6Result11__xopEqualsFKxSQDtQDsQDl__TQDeVQCya6_61203d3d2062TQDrTQDvZQE FQEdQEgZQDaKxQCiZb: error: undefined reference to '_D4core8internal5array8equality__T8__equalsTyaTyaZQqFNaNbNiNfMAyaMQeZb' ../library/lib.o:lib.d:function _D3std4conv__T10emplaceRefTAyaTQeTQhZQxFKQoKQrZ1S11__xopEqualsFKxSQCmQCl__TQCjTQCaTQCeTQCiZQCzF QCrKQCvZQCfKxQBsZb: error: undefined reference to '_D4core8internal5array8equality__T8__equalsTyaTyaZQqFNaNbNiNfMAyaMQeZb' ../library/lib.o:lib.d:function _D4core8internal5array8equality__T8__equalsTxAyaTxQfZQtFNaNbNiNfMAxQwMQfZb: error: undefined reference to '_D4core8internal5array8equality__T8__equalsTyaTyaZQqFNaNbNiNfMAyaMQeZb' collect2: error: ld returned 1 exit status Error: /usr/bin/cc failed with status: 1 make: *** [makefile:11: verifier] Error 1 dca franz:~/Software/newcash_d/verifier$ make cd ../library ; make make[1]: Entering directory '/home/dca/Software/newcash_d/library' make[1]: Nothing to be done for 'all'. make[1]: Leaving directory '/home/dca/Software/newcash_d/library' #dmd -g -profile=gc -I=../library -L='-lsqlite3' verifier.d ../library/lib.o gdc -g -I=../library -L='-lsqlite3' verifier.d ../library/lib.o verifier.d:4:8: error: module lib is in file 'lib.d' which cannot be read 4 | import lib; | ^ import path[0] = /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/d make: *** [makefile:11: verifier] Error 1 dca franz:~/Software/newcash_d/verifier$ ls -l ../library/lib.d -rw-r--r-- 1 dca allen 12244 Oct 20 11:19 ../library/lib.d dca franz:~/Software/newcash_d/verifier$ I hate to say this, because so much of this project makes sense to me -- the design of the language, the effort that has been put into documentation, the helpfulness of the community -- but I think I have to throw in the towel. I can't spend any more time on this and it appears that D is just not solid enough for the work I'm trying to do with it.
Oct 22 2020
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 22 October 2020 at 19:38:42 UTC, donallen wrote:
 [snip]

 I hate to say this, because so much of this project makes sense 
 to me -- the design of the language, the effort that has been 
 put into documentation, the helpfulness of the community -- but 
 I think I have to throw in the towel. I can't spend any more 
 time on this and it appears that D is just not solid enough for 
 the work I'm trying to do with it.
From the community's perspective, if there is a compiler bug that is causing your problem, then it will neither be found nor fixed, which is disappointing. In general, you will always get better help when you are able to provide either the complete code or a reduced example and what commands you used to run it, as was mentioned above.
Oct 22 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 21:00:41 UTC, jmh530 wrote:
 On Thursday, 22 October 2020 at 19:38:42 UTC, donallen wrote:
 [snip]

 I hate to say this, because so much of this project makes 
 sense to me -- the design of the language, the effort that has 
 been put into documentation, the helpfulness of the community 
 -- but I think I have to throw in the towel. I can't spend any 
 more time on this and it appears that D is just not solid 
 enough for the work I'm trying to do with it.
From the community's perspective, if there is a compiler bug that is causing your problem, then it will neither be found nor fixed, which is disappointing. In general, you will always get better help when you are able to provide either the complete code or a reduced example and what commands you used to run it, as was mentioned above.
I've already provided valgrind output that shows the gc referencing an uninitialized variable. Please understand -- I don't have infinite time to spend on this and I do have alternatives for this work, e.g., nim or go or chez scheme. Given the valgrind output indicating what looks like a serious error, I'm not going to put the effort into trying to reduce this error to an example that I can share fully unless and until someone convinces me that what valgrind is saying doesn't indicate a bug in the gc.
Oct 22 2020
parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 10/22/20 2:28 PM, donallen wrote:

 I've already provided valgrind output that shows the gc referencing an
 uninitialized variable.
I had similar problems. Luckily, all my problems were issues with my code. * In one case, the program was linked with dmd but I was loading a shared library (written again in D) but I was calling dlopen(). Instead, I had to call Runtime.loadLibrary. * In another case, my D library was being loaded by Python. I had forgotten to initialize the D runtime. I had to call Runtime.initialize. * In yet another case, I was calling myObject.close() in destructors, forgetting that myObject was already finalized by the GC.
 Please understand -- I don't have infinite time to spend on this
It's some more work but are you aware of Dustmite, which may be able to magically reduce your issue to a minimal case? https://dlang.org/blog/2020/04/13/dustmite-the-general-purpose-data-reduction-tool/ Ali
Oct 22 2020
prev sibling parent reply user1234 <user1234 12.de> writes:
On Thursday, 22 October 2020 at 19:38:42 UTC, donallen wrote:
 On Thursday, 22 October 2020 at 18:54:24 UTC, tsbockman wrote:
 On Thursday, 22 October 2020 at 04:02:10 UTC, donallen wrote:
 I tried this. With the gc on, the problem still occurs.

 I am working on 64-bit Arch Linux systems, up-to-date, DMD64 
 D Compiler v2.094.0
Have you tried using a different compiler?
Yes -- to no avail: dca franz:~/Software/newcash_d/verifier$ make cd ../library ; make make[1]: Entering directory '/home/dca/Software/newcash_d/library' make[1]: Nothing to be done for 'all'. make[1]: Leaving directory '/home/dca/Software/newcash_d/library' #dmd -g -profile=gc -I=../library -L='-lsqlite3' verifier.d ../library/lib.o ldc -g -I=../library -L='-lsqlite3' verifier.d ../library/lib.o ../library/lib.o:lib.d:function _D3std9exception__T7bailOutHTC9ExceptionZQwFNaNfAyamMAxaZv: error: undefined reference to '_d_throwdwarf' ../library/lib.o:lib.d:function [...]
The errors you get are because you must compile *everything* with ldc (program name should be ldc2 BTW), but the lib.o file is still produced with dmd as it contains DMD specific symbols.
 I hate to say this, because so much of this project makes sense 
 to me -- the design of the language, the effort that has been 
 put into documentation, the helpfulness of the community -- but 
 I think I have to throw in the towel. I can't spend any more 
 time on this and it appears that D is just not solid enough for 
 the work I'm trying to do with it.
Yeah, that's perfectly understandable. As you've been said in one of the first answer this must be because D GC sees a pointer allocated with malloc, and this pointer looks orphan to the GC as it has no root. Would you put something that's mallocated in a D struct or in a D array ? From the memory point of view the code you have shown is correct (the `fromStringz().dup` is exactly what's must be done to prevent problems when mixin manual and D managed memory) so the origin of the problem really *has* to be somewhere else.
Oct 22 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Oct 22, 2020 at 10:05:42PM +0000, user1234 via Digitalmars-d wrote:
[...]
 As you've been said in one of the first answer this must be because D
 GC sees a pointer allocated with malloc, and this pointer looks orphan
 to the GC as it has no root.
The GC does not collect objects allocated by malloc. (That would not make any sense, since if you allocated it with malloc, obviously you intend to deallocate it with free!) T -- I see that you JS got Bach.
Oct 22 2020
parent user1234 <user1234 12.de> writes:
On Thursday, 22 October 2020 at 23:07:28 UTC, H. S. Teoh wrote:
 On Thu, Oct 22, 2020 at 10:05:42PM +0000, user1234 via 
 Digitalmars-d wrote: [...]
 As you've been said in one of the first answer this must be 
 because D GC sees a pointer allocated with malloc, and this 
 pointer looks orphan to the GC as it has no root.
The GC does not collect objects allocated by malloc. (That would not make any sense, since if you allocated it with malloc, obviously you intend to deallocate it with free!) T
Yeah you're right. I was actually thinking to the opposite case: the parent is mallocated but the members are not, eg --- import core.memory; import std.experimental.allocator; import std.experimental.allocator.mallocator; import std.experimental.allocator.common; enum fill = "azertyuiopqsdfghjklm"; struct Node { this(string c){content = c;} string content; Node*[] nodes; } void main() { Node* root = make!Node(Mallocator.instance, fill); foreach(immutable i; 0 .. 10000) { root.nodes ~= make!Node(Mallocator.instance, fill); foreach(immutable j; 0 .. 100) root.nodes[i].nodes ~= make!Node(Mallocator.instance, fill); } assert(root.content == fill); foreach(immutable i; 0 .. root.nodes.length) { assert(root.nodes[i].content == fill); foreach(immutable j; 0 .. root.nodes[i].nodes.length) assert(root.nodes[i].nodes[j].content == fill); } } --- which crashes because of that (it "should" crash but I dont have much memory installed so maybe this is not the case for everyone)
Oct 22 2020
prev sibling parent reply donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 22:05:42 UTC, user1234 wrote:
 On Thursday, 22 October 2020 at 19:38:42 UTC, donallen wrote:
 On Thursday, 22 October 2020 at 18:54:24 UTC, tsbockman wrote:
 On Thursday, 22 October 2020 at 04:02:10 UTC, donallen wrote:
 I tried this. With the gc on, the problem still occurs.

 I am working on 64-bit Arch Linux systems, up-to-date, DMD64 
 D Compiler v2.094.0
Have you tried using a different compiler?
Yes -- to no avail: dca franz:~/Software/newcash_d/verifier$ make cd ../library ; make make[1]: Entering directory '/home/dca/Software/newcash_d/library' make[1]: Nothing to be done for 'all'. make[1]: Leaving directory '/home/dca/Software/newcash_d/library' #dmd -g -profile=gc -I=../library -L='-lsqlite3' verifier.d ../library/lib.o ldc -g -I=../library -L='-lsqlite3' verifier.d ../library/lib.o ../library/lib.o:lib.d:function _D3std9exception__T7bailOutHTC9ExceptionZQwFNaNfAyamMAxaZv: error: undefined reference to '_d_throwdwarf' ../library/lib.o:lib.d:function [...]
The errors you get are because you must compile *everything* with ldc (program name should be ldc2 BTW), but the lib.o file is still produced with dmd as it contains DMD specific symbols.
Yes, you are right. I deleted the .o files and rebuilt everything with ldc and it linked properly. But it not only produces the same error as dmd, but a new one as well (see the last line: A commodity with the same name as the account does exist. The account will be linked to it. requires a link to a commodity but doesn't have one. A commodity with the same name as the account does exist. The account will be linked to it. requires a link to a commodity but doesn't have one. A commodity with the same name as the account does exist. The account will be linked to it. requires a link to a commodity but doesn't have one. A commodity with the same name as the account does exist. The account will be linked to it. requires a link to a commodity but doesn't have one. A commodity with the same name as the account does exist. The account will be linked to it. requires a link to a commodity but doesn't have one. A commodity with the same name as the account does exist. The account will be linked to it. requires a link to a commodity but doesn't have one. A commodity with the same name as the account does exist. The account will be linked to it. walk_account_tree: child index exceeds n_children dca igor:~/Software/newcash_d/verifier$
 I hate to say this, because so much of this project makes 
 sense to me -- the design of the language, the effort that has 
 been put into documentation, the helpfulness of the community 
 -- but I think I have to throw in the towel. I can't spend any 
 more time on this and it appears that D is just not solid 
 enough for the work I'm trying to do with it.
Yeah, that's perfectly understandable. As you've been said in one of the first answer this must be because D GC sees a pointer allocated with malloc, and this pointer looks orphan to the GC as it has no root. Would you put something that's mallocated in a D struct or in a D array ? From the memory point of view the code you have shown is correct (the `fromStringz().dup` is exactly what's must be done to prevent problems when mixin manual and D managed memory) so the origin of the problem really *has* to be somewhere else.
Oct 22 2020
parent user1234 <user1234 12.de> writes:
On Thursday, 22 October 2020 at 23:48:22 UTC, donallen wrote:
 I hate to say this, because so much of this project makes 
 sense to me -- the design of the language, the effort that 
 has been put into documentation, the helpfulness of the 
 community -- but I think I have to throw in the towel. I 
 can't spend any more time on this and it appears that D is 
 just not solid enough for the work I'm trying to do with it.
BTW I forgot to mention, even if it's too late I suppose, that to convert a C program to D a way to go is: 1. convert minimally and use -betterC 2. leave the -betterC mode 3. incrementally add more idiomatic D feature see https://dlang.org/blog/2018/06/11/dasbetterc-converting-make-c-to-d/ for more detail on the methodolgy.
Oct 23 2020
prev sibling next sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
donallen wrote:

 But working with gdb, I found that the account structs after 
 the error messages start are zeroed.
GC should not zero any memory by itself (it doesn't "clear" freed chunks). it looks like the GC is reusing the allocated memory. you can use `core.memory.GC.addrOf()` to check if your pointers are still "alive". you can add calls to `addrOf()` before processing a data, it may help to catch the moment data is incorrectly freed (if that happens).
Oct 21 2020
parent donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 01:30:41 UTC, ketmar wrote:
 donallen wrote:

 But working with gdb, I found that the account structs after 
 the error messages start are zeroed.
GC should not zero any memory by itself (it doesn't "clear" freed chunks). it looks like the GC is reusing the allocated memory.
The thought did occur to me as well that the zeroing probably occurs on allocation, but I don't see how you get from that to the behavior I'm observing, since the children array gets populated with data right after allocation. It appears that the Account structs being passed in the recursive calls to walk_account_tree, which are elements of the children array, are zeroed.
 you can use `core.memory.GC.addrOf()` to check if your pointers 
 are still "alive". you can add calls to `addrOf()` before 
 processing a data, it may help to catch the moment data is 
 incorrectly freed (if that happens).
I'll give this a try tomorrow. Thanks.
Oct 21 2020
prev sibling next sibling parent reply donallen <donaldcallen gmail.com> writes:
Some new things:

1. I tried building with the gc-profile option, like so:

dmd -g -profile=gc -I=../library -L='-lsqlite3' verifier.d 
../library/lib.o

When I run the program built this way, the problem does not 
occur. There is also no gc profiling output. Makes me wonder if 
the compile-time option somehow altered things (increased heap 
size?) that resulted in no gcs? Pure speculation on my part, but 
it is consistent with the behavior (program works correctly when 
the gc doesn't run, no profiling output suggests no gc).

But if I enable gc profiling at the command line:
verifier "--DRT-gcopt=profile:1" /tmp/Finances.newcash

the problem does occur and I get profiling output:
         Number of collections:  3
         Total GC prep time:  0 milliseconds
         Total mark time:  0 milliseconds
         Total sweep time:  0 milliseconds
         Max Pause Time:  0 milliseconds
         Grand total GC time:  0 milliseconds
GC summary:    5 MB,    3 GC    0 ms, Pauses    0 ms <    0 ms

2. I tried adding a call to addrOf(children.ptr), panicking on a 
null return, just before the recursive calls to the tree walker. 
No panic.

I will look into a way to reproduce this that I can give to the 
project. The problem is not the code -- it's the data. The 
database that the D version of the verifier is failing on 
contains literally decades of our financial transactions. So 
while there is nothing illegal, immoral or fattening in it, it's 
nobody's business but ours. So I need to create a new database 
with no personal info in it that provokes the problem.

I will also look into running the verifier under Valgrind.

I don't know when I will be able to devote more time to this. 
Like everyone, I've got other things on my plate.

Thanks to everyone who has tried to help.
Oct 22 2020
parent reply donallen <donaldcallen gmail.com> writes:
Realizing that running this thing under valgrind was easy, I did 
so:

valgrind --leak-check=yes --track-origins=yes ./verifier 
/tmp/Finances.newcash > /tmp/vg.output 2>&1

and then

ddemangle /tmp/vg.output > /tmp/vg-demangled.output

Here is the first half of the output (posting in two parts 
because it's big), which suggests a problem in the gc (the 
messages beginning "requires a link to a commodity ..." are the 
bogus errors produced by the verifier due to looking at zeroed 
Account structs; I've removed the duplicates):

==3854== Memcheck, a memory error detector
==3854== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward 
et al.
==3854== Using Valgrind-3.16.1 and LibVEX; rerun with -h for 
copyright info
==3854== Command: ./verifier /tmp/Finances.newcash
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x21D7C6: nothrow scope void 
gc.impl.conservative.gc.Gcx.collectRoots(void*, void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23BD4B: nothrow void 
core.thread.threadbase.thread_scanAll(scope void delegate(void*, 
void*) nothrow).__lambda2!(core.thread.threadbase.ScanType, 
void*, void*).__lambda2(core.thread.threadbase.ScanType, void*, 
void*) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2472CF: nothrow void 
core.thread.threadbase.scanAllTypeImpl(scope void 
delegate(core.thread.threadbase.ScanType, void*, void*) nothrow, 
void*) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x24720C: nothrow void 
core.thread.threadbase.thread_scanAllType(scope void 
delegate(core.thread.threadbase.ScanType, void*, void*) 
nothrow).__lambda2!(void*).__lambda2(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x211EEF: nothrow void 
core.thread.osthread.callWithStackShell(scope void 
delegate(void*) nothrow) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2471E1: thread_scanAllType (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23BD11: thread_scanAll (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21D901: nothrow void 
gc.impl.conservative.gc.Gcx.collectAllRoots(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21EFF9: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x211EBC: nothrow void 
core.thread.osthread.callWithStackShell(scope void 
delegate(void*) nothrow) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Thread 2:
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224995: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249A8: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249B7: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249BE: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224A0A: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224A4F: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x23C1CB: nothrow  nogc ulong 
gc.bits.GCBits.setLocked(ulong) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x224A6C: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224A8D: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1E9048: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249D6: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249E3: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224987: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224C56: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Thread 1:
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224995: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249A8: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249B7: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x2249BE: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224A0A: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    b
Oct 22 2020
next sibling parent reply donallen <donaldcallen gmail.com> writes:
Here's the rest of it:

y 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224A4F: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x23C1CB: nothrow  nogc ulong 
gc.bits.GCBits.setLocked(ulong) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x224A6C: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224A8D: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224987: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224C56: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F137: nothrow void 
gc.impl.conservative.gc.Gcx.markParallel(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21E959: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E90CB: _d_newitemT (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8D4B: pure nothrow ref  trusted 
std.array.Appender!(immutable(char)[]).Appender 
std.array.Appender!(immutable(char)[]).Appender.__ctor(immutable(char)[])
(/usr/include/dlang/dmd/std/array.d:3261)
==3854==    by 0x1B8C0F: pure nothrow  safe 
std.array.Appender!(immutable(char)[]).Appender 
std.array.appender!(immutable(char)[]).appender() 
(/usr/include/dlang/dmd/std/array.d:3886)
==3854==    by 0x1C81AC: pure  safe immutable(char)[] 
std.format.format!(char, immutable(char)[], 
immutable(char)[]).format(in char[], immutable(char)[], 
immutable(char)[]) (/usr/include/dlang/dmd/std/format.d:6642)
==3854==    by 0x1B7E98: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77FC: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
  requires a link to a commodity but doesn't have one.
A commodity with the same name as the account does exist. The 
account will be linked to it.
  [snip]
  ==3854== Thread 4:
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224AF4: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x23C1CB: nothrow  nogc ulong 
gc.bits.GCBits.setLocked(ulong) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x224B39: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224B3D: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224B5A: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x224B67: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Use of uninitialised value of size 8
==3854==    at 0x224B75: nothrow scope void 
gc.impl.conservative.gc.Gcx.mark!(false, 
true).mark(gc.impl.conservative.gc.Gcx.ScanRange!(false).ScanRange) (in
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x225007: nothrow void 
gc.impl.conservative.gc.Gcx.pullFromScanStackImpl!(false).pul
FromScanStackImpl() (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21F645: nothrow void 
gc.impl.conservative.gc.Gcx.scanBackground() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x23C0D2: extern (C) nothrow void* 
core.thread.osthread.createLowLevelThread(void delegate() 
nothrow, uint, void delegate() 
nothrow).thread_lowlevelEntry(void*) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x49B03E8: start_thread (in 
/usr/lib/libpthread-2.32.so)
==3854==    by 0x4C3C292: clone (in /usr/lib/libc-2.32.so)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
==3854== Thread 1:
==3854== Conditional jump or move depends on uninitialised 
value(s)
==3854==    at 0x21DA9C: nothrow ulong 
gc.impl.conservative.gc.Gcx.sweep() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21EB9F: nothrow ulong 
gc.impl.conservative.gc.Gcx.fullcollect(bool) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21CBFC: nothrow void* 
gc.impl.conservative.gc.Gcx.smallAlloc(ulong, ref ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2227C0: nothrow void* 
gc.impl.conservative.gc.ConservativeGC.runLocked!(gc.impl.conservative.gc.ConservativeGC
mallocNoSync(ulong, uint, ref ulong, const(TypeInfo)),
gc.impl.conservative.gc.mallocTime, gc.impl.conservative.gc.numMallocs, ulong,
uint, ulong, const(TypeInfo)).runLocked(ref ulong, ref uint, ref ulong, ref
const(TypeInfo)) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x21A279: nothrow core.memory.BlkInfo_ 
gc.impl.conservative.gc.ConservativeGC.qalloc(ulong, uint, 
const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C5E: gc_qalloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6446: pure nothrow core.memory.BlkInfo_ 
core.memory.GC.qalloc(ulong, uint, const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B90AF: pure nothrow  trusted 
core.memory.BlkInfo_ 
std.array.Appender!(immutable(char)[]).Appender.ensureAddabl
(ulong).__lambda3() (/usr/include/dlang/dmd/std/array.d:3386)
==3854==    by 0x1B8FCD: pure nothrow  safe void 
std.array.Appender!(immutable(char)[]).Appender.ensureAddable(ulong)
(/usr/include/dlang/dmd/std/array.d:3386)
==3854==    by 0x1BD1E6: pure nothrow  safe char[] 
std.array.Appender!(immutable(char)[]).Appender.put!(immutable(char)[]).put(immutable(char)[
).bigDataFun(ulong) (/usr/include/dlang/dmd/std/array.d:3488)
==3854==    by 0x1BD0D4: pure nothrow  safe void 
std.array.Appender!(immutable(char)[]).Appender.put!(immutable(char)[]).pu
(immutable(char)[]) (/usr/include/dlang/dmd/std/array.d:3491)
==3854==    by 0x1C3B92: pure nothrow  safe void 
std.range.primitives.doPut!(std.array.Appender!(immutable(char)[]).Appender,
immutable(char)[]).doPut(ref std.array.Appender!(immutable(char)[]).Appender,
ref immutable(char)[]) (/usr/include/dlang/dmd/std/range/primitives.d:277)
==3854==  Uninitialised value was created by a stack allocation
==3854==    at 0x1B77F8: void 
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
==3854==
but doesn't have one.
A commodity with the same name as the account does exist. The 
account will be linked to it.
  requires a link to a commodity but doesn't have one.
[snip]
==3854==
==3854== HEAP SUMMARY:
==3854==     in use at exit: 2,380,288 bytes in 2,013 blocks
==3854==   total heap usage: 339,984 allocs, 337,971 frees, 
439,341,276 bytes allocated
==3854==
==3854== 16 bytes in 1 blocks are definitely lost in loss record 
21 of 171
==3854==    at 0x483A77F: malloc (vg_replace_malloc.c:307)
==3854==    by 0x208B1B: nothrow  nogc void* rt.tlsgc.init() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x207181: thread_init (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1FEC3B: rt_init (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E76DB: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).runAll() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7678: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).tryExec(scope void 
delegate()) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E75E1: _d_run_main2 (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7359: _d_run_main (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B8A89: main 
(/usr/include/dlang/dmd/core/internal/entrypoint.d:29)
==3854==
==3854== 32 bytes in 1 blocks are possibly lost in loss record 33 
of 171
==3854==    at 0x483A77F: malloc (vg_replace_malloc.c:307)
==3854==    by 0x2195E5: core.gc.gcinterface.GC 
gc.impl.conservative.gc.initialize() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x20CF7C: core.gc.gcinterface.GC 
core.gc.registry.createGCInstance(immutable(char)[]) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x2074E2: gc_init_nothrow (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1FDA7F: nothrow void* 
gc.impl.proto.gc.ProtoGC.malloc(ulong, uint, const(TypeInfo)) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E6C1A: gc_malloc (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E8514: _d_allocmemory (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B6A4F: _Dmain (verifier.d:8)
==3854==    by 0x1E77CA: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).runAll().__lambda1() 
(in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7678: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).tryExec(scope void 
delegate()) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7752: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).runAll() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7678: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).tryExec(scope void 
delegate()) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== 2,380,056 (768 direct, 2,379,288 indirect) bytes in 1 
blocks are definitely lost in loss record 171 of 171
==3854==    at 0x483A77F: malloc (vg_replace_malloc.c:307)
==3854==    by 0x4905F13: ??? (in /usr/lib/libsqlite3.so.0.8.6)
==3854==    by 0x4905489: sqlite3Malloc (in 
/usr/lib/libsqlite3.so.0.8.6)
==3854==    by 0x490599A: sqlite3MallocZero (in 
/usr/lib/libsqlite3.so.0.8.6)
==3854==    by 0x490454E: ??? (in /usr/lib/libsqlite3.so.0.8.6)
==3854==    by 0x1D77FA: void* lib.open_db(immutable(char)[]) (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1B6ACB: _Dmain (verifier.d:33)
==3854==    by 0x1E77CA: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).runAll().__lambda1() 
(in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7678: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).tryExec(scope void 
delegate()) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7752: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).runAll() (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E7678: void rt.dmain2._d_run_main2(char[][], 
ulong, extern (C) int function(char[][])*).tryExec(scope void 
delegate()) (in /home/dca/Software/newcash_d/verifier/verifier)
==3854==    by 0x1E75E1: _d_run_main2 (in 
/home/dca/Software/newcash_d/verifier/verifier)
==3854==
==3854== LEAK SUMMARY:
==3854==    definitely lost: 784 bytes in 2 blocks
==3854==    indirectly lost: 2,379,288 bytes in 2,005 blocks
==3854==      possibly lost: 32 bytes in 1 blocks
==3854==    still reachable: 184 bytes in 5 blocks
==3854==         suppressed: 0 bytes in 0 blocks
==3854== Reachable blocks (those to which a pointer was found) 
are not shown.
==3854== To see them, rerun with: --leak-check=full 
--show-leak-kinds=all
==3854==
==3854== For lists of detected and suppressed errors, rerun with: 
-s
==3854== ERROR SUMMARY: 3213 errors from 33 contexts (suppressed: 
0 from 0)
Oct 22 2020
parent Jerry <labuurii gmail.com> writes:
Maybe the answer is in the valgrind logs, its alot to read. But 
isn't it possible to put a write breakpoint in the accounts 
arrays and step through the code until a zero is written.
Oct 22 2020
prev sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 23/10/2020 4:56 AM, donallen wrote:
 ==3854==    by 0x1B7E98: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, 
 int) (verifier.d:272)
 ==3854==  Uninitialised value was created by a stack allocation
 ==3854==    at 0x1B77FC: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, 
 int) (verifier.d:84)
 ==3854==
 ==3854== Conditional jump or move depends on uninitialised value(s)
That is coming up an awful lot. We'd need walk_account_tree source, to know if its doing something funky. But upon reviewing how the entire thing is working, I think we need one_row and next_row_available_p source as well. Oh and reset_stmt too.
Oct 22 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 21:58:04 UTC, rikki cattermole 
wrote:
 On 23/10/2020 4:56 AM, donallen wrote:
 ==3854==    by 0x1B7E98: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
 ==3854==  Uninitialised value was created by a stack allocation
 ==3854==    at 0x1B77FC: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
 ==3854==
 ==3854== Conditional jump or move depends on uninitialised 
 value(s)
That is coming up an awful lot. We'd need walk_account_tree source, to know if its doing something funky. But upon reviewing how the entire thing is working, I think we need one_row and next_row_available_p source as well. Oh and reset_stmt too.
Here's the source code you requested: void walk_account_tree(Account account, int ancestor_flags) { void fix_missing_commodity() { immutable string new_commodity_guid = one_row(new_guid, &get_string).string_value; // Create new commodity bind_text(insert_new_commodity, 1, new_commodity_guid); bind_text(insert_new_commodity, 2, account.name); run_dm_stmt(insert_new_commodity, &reset_stmt); // And link the account to the new commodity bind_text(link_to_commodity, 1, new_commodity_guid); bind_text(link_to_commodity, 2, account.name); run_dm_stmt(link_to_commodity, &reset_stmt); } void check_and_repair_commodity_link() { if (account.commodity_guid.length == 0) { bind_text(get_possible_commodity_guid, 1, account.name); immutable Maybe maybe_commodity_guid = maybe_one_row(get_possible_commodity_guid, &get_string); if (!maybe_commodity_guid.valid_p) { // No commodity exists with the same name as the account. Create one and link the account to it. writeln(account.path, " requires a link to a commodity but doesn't have one. A commodity with the same name as the account does not exist. One will be created with the symbol **unknown** and the account will be linked to it. If you wish to get quotes for this commodity, you will need to fix the symbol in Newcash."); fix_missing_commodity(); } else { writeln(account.path, " requires a link to a commodity but doesn't have one. A commodity with the same name as the account does exist. The account will be linked to it."); bind_text(link_to_commodity, 1, maybe_commodity_guid.value.string_value); bind_text(link_to_commodity, 2, account.guid); run_dm_stmt(link_to_commodity, &reset_stmt); } } else { sqlite3_reset(get_possible_commodity_guid); /* This marketable account has a commodity. Check to be sure that the guid is valid. If not, set the account's commodity_guid to NULL and process this account again. */ bind_text(verify_commodity_guid, 1, account.commodity_guid); Maybe maybe_valid_guid = maybe_one_row(verify_commodity_guid, &get_string); if (!maybe_valid_guid.valid_p) { writeln(account.path, " requires a commodity link and has one, but the commodity guid is invalid. Creating a new commodity with the same name as the account and linking the account to it."); fix_missing_commodity(); } else { // Warn if the commodity name is not the same as the account name bind_text(check_commodity_name, 1, account.guid); MultiType n = one_row(check_commodity_name, &get_int); if (n.integer_value == 0) { writeln(account.path, " requires a commodity link and has one, but the commodity name is not the same as the account name. Is this intentional?"); } } } } // Placeholder? if ((account.flags & account_flag_placeholder) == 0) { // No. Is this account an asset? if ((ancestor_flags & account_flag_descendents_are_assets) != 0) { // Yes. Is it marketable? if ((ancestor_flags & account_flag_descendents_are_marketable) != 0) { // Yes, account is a marketable asset. Check that it is linked to a proper commodity. check_and_repair_commodity_link(); } else { // No if (account.commodity_guid.length > 0) { // This account is a non-marketable asset and has a non-null commodity guid. // Set to NULL. Non-marketable accounts should not point to commodities. writeln(account.path, " %s is a non-marketable asset, but it is associated with a commodity. Removing the association by setting the account's commodity link to NULL."); bind_text(nullify_commodity_guid, 1, account.guid); run_dm_stmt(nullify_commodity_guid, &reset_stmt); } // Make sure the quantity is zero. Should not be otherwise for a non-marketable asset. bind_text(check_quantities, 1, account.guid); if (one_row(check_quantities, &get_int).integer_value > 0) { writeln(account.path, " is a non-marketable asset account but has splits with non-zero quantities. These will be fixed."); bind_text(fix_quantities, 1, account.guid); run_dm_stmt(fix_quantities, &reset_stmt); } } } else { // This account is not an asset. Make sure it doesn't point to a commodity, // unless it is an Income account and inherits the descendents-need-commodity property. // Does it need a commodity link? if ((ancestor_flags & account_flag_descendents_need_commodity_link) != 0) { // Yes. Is it an income account? if ((ancestor_flags & account_flag_descendents_are_income) != 0) { // Yes check_and_repair_commodity_link(); } else { writeln("The account ", account.path, " is not an Asset or Income account, but inherits the 'needs commodity' property. This should not be possible and indicates a bug in Newcash or in the Verifier. Please report to Don Allen."); } } else { if (account.commodity_guid.length != 0) { // This account is not an asset, doesn't have the needs-commodity-link property and has a non-null commodity guid, which we set to NULL. writeln(account.path, " is not an asset, doesn't have the needs-commodity-link property, but it is associated with a commodity. Removing the association by setting the account's commodity link to NULL."); bind_text(nullify_commodity_guid, 1, account.guid); run_dm_stmt(nullify_commodity_guid, &reset_stmt); } // The account is not an asset and does not have needs-commodity-link property. // Just make sure it doesn't inherit the marketable property if ((ancestor_flags & account_flag_descendents_are_marketable) != 0) { writeln(account.path, " is designated marketable, but is not an asset account. This should not be possible and is indicative of a Newcash bug. Please report this to Don Allen."); } } // Make sure the quantity is zero. Should not be otherwise for a non-asset. bind_text(check_quantities, 1, account.guid); int temp; if ((temp = one_row(check_quantities, &get_int).integer_value) > 0) { writeln(account.path, " %s is not an asset account but has splits with non-zero quantities. These will be fixed."); bind_text(fix_quantities, 1, account.guid); run_dm_stmt(fix_quantities, &reset_stmt); } } } else { // This account is a place-holder. Verify that it has no transactions. Warn the user if that is not true. bind_text(count_transactions, 1, account.guid); int transaction_count = one_row(count_transactions, &get_int).integer_value; if (transaction_count > 0) { writeln(account.path, " is a placeholder account, but it has %d transactions. You should re-assign them to an appropriate account with Newcash."); } } // Now do the children of this account // First determine how many there are bind_text(count_children, 1, account.guid); int n_children = one_row(count_children, &get_int).integer_value; if (n_children > 0) { Account[] children = new Account[n_children]; bind_text(find_children, 1, account.guid); int i = 0; while (next_row_available_p(find_children, &reset_stmt)) { if (i > n_children-1) panic("walk_account_tree: child index exceeds n_children"); children[i].name = fromStringz(sqlite3_column_text(find_children, 0)).dup; children[i].path = format("%s:%s", account.path, children[i].name); children[i].guid = fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid children[i].commodity_guid = fromStringz(sqlite3_column_text(find_children, 2)).dup; // commodity_guid children[i].flags = sqlite3_column_int(find_children, 3) | account.flags & (account_flag_descendents_are_assets | account_flag_descendents_are_liabilities | account_flag_descendents_are_income | account_flag_descendents_are_expenses | account_flag_descendents_are_marketable | account_flag_self_and_descendents_are_tax_related | account_flag_descendents_need_commodity_link); // flags i = i + 1; } foreach (child; children) { walk_account_tree(child, ancestor_flags | account.flags); } } } alias one_row_value_delegate = bool delegate (sqlite3_stmt* stmt, MultiType* result); alias one_row_value_function = bool function (sqlite3_stmt* stmt, MultiType* result); MultiType one_row(T)(sqlite3_stmt* stmt, T get_values) { MultiType result; if (next_row_available_p(stmt, &reset_stmt)) { if (get_values(stmt, &result)) { // Did we get a value of the type requested? // Yes. Now make sure we only got one row back. This will also reset the stmt. if (next_row_available_p(stmt, &reset_stmt)) panic("one_row: stmt returned more than one row"); return result; } else panic("one_row: stmt returned a null value"); } else panic("one_row: stmt returned no rows"); return result; // Need this for compiler happiness -- can't get here } /// Check for availability of another row bool next_row_available_p(sqlite3_stmt* stmt, int function(sqlite3_stmt* stmt) cleanup) { switch (sqlite3_step(stmt)) { case sqlite_row: return (true); case sqlite_done: /* Make the prepared stmt available for re-use. */ cleanup(stmt); return (false); default: stderr.writeln("next_row_available_p failed",); stderr.writeln(sqlite3_errmsg(sqlite3_db_handle(stmt)), stderr); exit(EXIT_FAILURE); } return (false); // Need this for compiler happiness -- can't get here } /// Reset so statement can be re-used int reset_stmt(sqlite3_stmt* stmt) { return sqlite3_reset(stmt); }
Oct 23 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Oct 23, 2020 at 12:47:11PM +0000, donallen via Digitalmars-d wrote:
 On Thursday, 22 October 2020 at 21:58:04 UTC, rikki cattermole wrote:
 On 23/10/2020 4:56 AM, donallen wrote:
 ==3854== by 0x1B7E98: void
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable(char)[][]).Account,
 int) (verifier.d:272)
 ==3854== Uninitialised value was created by a stack allocation
 ==3854== at 0x1B77FC: void
verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable(char)[][]).Account,
 int) (verifier.d:84)
 ==3854==
 ==3854== Conditional jump or move depends on uninitialised value(s)
[...] Hmm. Which lines in your posted code do lines 272 and 84 refer to? Those seem to be the problematic spots, but it's not clear where in the function they are, since your posted code doesn't indicate line numbers. I also noticed this bit of code (which may or may not have anything to do with it):
                 int temp;
                 if ((temp = one_row(check_quantities,
 &get_int).integer_value) > 0)
                 {
                     writeln(account.path,
                             " %s is not an asset account but has splits with
 non-zero quantities. These will be fixed.");
 
                     bind_text(fix_quantities, 1, account.guid);
                     run_dm_stmt(fix_quantities, &reset_stmt);
                 }
The value of `temp` is not used after assignment. This is a rather odd way of writing it. What's the reason you didn't just write: if (one_row(...) > 0) instead? [...]
 MultiType one_row(T)(sqlite3_stmt* stmt, T get_values)
What's the definition of MultiType? [...]
     return result; // Need this for compiler happiness -- can't get here
Note for the future: you could write `assert(0);` here to indicate that it should not be reachable. (It compiles to a single 'hlt' instruction, which will abort if the program somehow reaches it anyway.) T -- My program has no bugs! Only unintentional features...
Oct 23 2020
parent donallen <donaldcallen gmail.com> writes:
On Friday, 23 October 2020 at 14:37:41 UTC, H. S. Teoh wrote:
 On Fri, Oct 23, 2020 at 12:47:11PM +0000, donallen via 
 Digitalmars-d wrote:
 On Thursday, 22 October 2020 at 21:58:04 UTC, rikki cattermole 
 wrote:
 On 23/10/2020 4:56 AM, donallen wrote:
 ==3854==    by 0x1B7E98: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable(char)[][]).Account,
 int) (verifier.d:272)
 ==3854==  Uninitialised value was created by a stack 
 allocation
 ==3854==    at 0x1B77FC: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable(char)[][]).Account,
 int) (verifier.d:84)
 ==3854==
 ==3854== Conditional jump or move depends on uninitialised 
 value(s)
[...] Hmm. Which lines in your posted code do lines 272 and 84 refer to? Those seem to be the problematic spots, but it's not clear where in the function they are, since your posted code doesn't indicate line numbers. I also noticed this bit of code (which may or may not have anything to do with it):
                 int temp;
                 if ((temp = one_row(check_quantities,
 &get_int).integer_value) > 0)
                 {
                     writeln(account.path,
                             " %s is not an asset account but 
 has splits with
 non-zero quantities. These will be fixed.");
 
                     bind_text(fix_quantities, 1, account.guid);
                     run_dm_stmt(fix_quantities, &reset_stmt);
                 }
The value of `temp` is not used after assignment. This is a rather odd way of writing it. What's the reason you didn't just write: if (one_row(...) > 0) instead?
This code was ported from the original C. In the C code, I printf-ed the quantity in the error message and so needed to stash it somewhere (temp). When I transformed the code to D, I think I hadn't discovered format at that point, so just used writeln's ability to print the account.path prepended to the fixed text of the message. I forgot to take out the temporary and write it as you suggest above. So what you see in my code is a vestige of the C that is no longer applicable, but not harmful.
 [...]
 MultiType one_row(T)(sqlite3_stmt* stmt, T get_values)
What's the definition of MultiType?
union MultiType { int integer_value; sqlite3_int64 long_integer_value; double double_value; string string_value; void* pointer_value; }
 [...]
     return result; // Need this for compiler happiness -- 
 can't get here
Note for the future: you could write `assert(0);` here to indicate that it should not be reachable. (It compiles to a single 'hlt' instruction, which will abort if the program somehow reaches it anyway.)
Thanks for the tip, useful if there's D in my future (which there will be if you guys can find something wrong with my code rather than the system).
 T
Oct 23 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/23/20 8:47 AM, donallen wrote:
 On Thursday, 22 October 2020 at 21:58:04 UTC, rikki cattermole wrote:
 On 23/10/2020 4:56 AM, donallen wrote:
 ==3854==    by 0x1B7E98: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, 
 int) (verifier.d:272)
 ==3854==  Uninitialised value was created by a stack allocation
 ==3854==    at 0x1B77FC: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, 
 int) (verifier.d:84)
 ==3854==
 ==3854== Conditional jump or move depends on uninitialised value(s)
That is coming up an awful lot. We'd need walk_account_tree source, to know if its doing something funky. But upon reviewing how the entire thing is working, I think we need one_row and next_row_available_p source as well. Oh and reset_stmt too.
Here's the source code you requested:
This all looks ok. The only think I can possibly consider is that you are using bind_text a lot, to presumably bind sqlite prepared statements to D strings. This means it's *possible* that the prepared statements have pointers to the strings, but nothing else does (hard to work it out from just reading the code, I can't tell when things go out of scope easily, and your prepared statements clearly exist outside these functions). If a prepared statement is a C-malloc'd item, then it's not scanned by the GC. This means that what possibly might be happening is that you bind a D string to a text parameter, the D string goes out of scope, the GC collects it and allocates it elsewhere, and then sqlite tries to use it to send some requests. To test this theory, maybe try a version of the code that when calling bind_text, you append the string itself to a GC allocated array stored in a global. Might not prove anything, but if it's fixing the problem, maybe that's where the GC is collecting things that it shouldn't. -Steve
Oct 23 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Friday, 23 October 2020 at 15:49:49 UTC, Steven Schveighoffer 
wrote:
 On 10/23/20 8:47 AM, donallen wrote:
 On Thursday, 22 October 2020 at 21:58:04 UTC, rikki cattermole 
 wrote:
 On 23/10/2020 4:56 AM, donallen wrote:
 ==3854==    by 0x1B7E98: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:272)
 ==3854==  Uninitialised value was created by a stack 
 allocation
 ==3854==    at 0x1B77FC: void 
 verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable
char)[][]).Account, int) (verifier.d:84)
 ==3854==
 ==3854== Conditional jump or move depends on uninitialised 
 value(s)
That is coming up an awful lot. We'd need walk_account_tree source, to know if its doing something funky. But upon reviewing how the entire thing is working, I think we need one_row and next_row_available_p source as well. Oh and reset_stmt too.
Here's the source code you requested:
This all looks ok. The only think I can possibly consider is that you are using bind_text a lot, to presumably bind sqlite prepared statements to D strings. This means it's *possible* that the prepared statements have pointers to the strings, but nothing else does (hard to work it out from just reading the code, I can't tell when things go out of scope easily, and your prepared statements clearly exist outside these functions). If a prepared statement is a C-malloc'd item, then it's not scanned by the GC. This means that what possibly might be happening is that you bind a D string to a text parameter, the D string goes out of scope, the GC collects it and allocates it elsewhere, and then sqlite tries to use it to send some requests. To test this theory, maybe try a version of the code that when calling bind_text, you append the string itself to a GC allocated array stored in a global. Might not prove anything, but if it's fixing the problem, maybe that's where the GC is collecting things that it shouldn't.
It's a good theory, but I don't think it explains this. What you are hypothesizing is that there's an instance where the lifetime of a D string that is used in a sqlite bind_text is shorter than the lifetime of the sqlite binding. I don't think that's the case. The calls to bind_text all immediately precede calls to run_dm_stmt, one_row, etc. Those calls either run an insert or update, or in the case of one_row, a select. When those calls return, the statement has been reset, which means, among other things, that the bindings have been released, ending the binding's life. The other case to look at is the use of next_row_available_p in a loop. There, bindings are established before entering the loop, which processes multiple rows coming back from a select. When next_row_available_p runs out of rows, it calls the cleanup function, usually reset_stmt (which just calls sqlite3_reset; I found that I could not pass an extern (c) function to next_row_available_p, so used this workaround; maybe there is a way to do this, but I just used this simple expedient). So at that point the bindings are cleared and next_row_available_p returns false, ending the loop and life of the bindings. Unless I've missed something, I think you will find that all the strings passed to bind_text (which get massaged by toStringz, so they look like C strings) live beyond the point where the statements they are bound to are reset. Most of the calls to bind_text in the tree walker bind fields in account, which is an argument to the tree walker and thus lives until it returns. The others are to local variables that are declared before they are used in bind-text calls and the statement gets reset before the scope ends. Yes, your theory could be tested using some variant of what you describe, in case my there's a flaw in my hand-waving above. Some food for thought: If I insert a writeln at the very beginning of walk_account_tree, printing, say, the name and guid of the account, the problem goes away -- the program behaves correctly. If I change children[i].path = format("%s:%s", account.path, children[i].name); to children[i].path = format("%s:%s", account.path, children[i].name).dup; in the loop that fills in the child Account structs in children, same thing -- correct behavior. The dup of the format should not be necessary, since format returns a new GC-allocated string. I suspect all of these odd "cures" have the effect they do because they change the allocation pattern of this section of the code. Without any of them, it appears that an allocation is triggering a GC at exactly the wrong moment. What is not understood is what is wrong with the moment.
 -Steve
Oct 23 2020
next sibling parent matheus <matheus gmail.com> writes:
On Friday, 23 October 2020 at 18:31:41 UTC, donallen wrote:
 ...
Have you tried Dustmite? https://dlang.org/blog/2020/04/13/dustmite-the-general-purpose-data-reduction-tool/ Matheus.
Oct 23 2020
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Oct 23, 2020 at 06:31:41PM +0000, donallen via Digitalmars-d wrote:
[...]
 Some food for thought:
 
 If I insert a writeln at the very beginning of walk_account_tree,
 printing, say, the name and guid of the account, the problem goes away
 -- the program behaves correctly.
Curious. IME, this is usually the sign of memory corruption elsewhere in the code, and it's just pure happenstance that changing the allocation pattern causes the corruption to overwrite something else that doesn't lead to observable symptoms. I wonder what might happen if instead of inserting a writeln, you inserted a dummy allocation? Like: int[] dummy = [ 1, 2, 3 ]; printf("%d\n", dummy.length); // don't let optimizer elide it
 If I change
 children[i].path = format("%s:%s", account.path, children[i].name);
 to
 children[i].path = format("%s:%s", account.path, children[i].name).dup;
 
 in the loop that fills in the child Account structs in children, same
 thing -- correct behavior. The dup of the format should not be
 necessary, since format returns a new GC-allocated string.
Indeed. So again it looks like some kind of memory corruption. Only, without a reproducible test case we're just shooting in the dark here as to what exactly is causing it. Have you looked into Dustmite? Once you set it up, it's fully automated, and you can leave it running in the background until it has reduced the code to a minimal(-ish) test case, so it won't consume too much of your time. Another random shot in the dark is, what compiler flag(s) are you using to compile the program? If you're using dmd with -O or -inline (or both), there's a small chance you might be seeing a codegen bug. (I've had bad experiences with dmd's backend before, both in terms of codegen bugs and poor optimization quality, and nowadays only use dmd for quick prototyping or fast turnaround during development; for production I avoid dmd and use LDC's much better and more reliable backend.)
 I suspect all of these odd "cures" have the effect they do because
 they change the allocation pattern of this section of the code.
 Without any of them, it appears that an allocation is triggering a GC
 at exactly the wrong moment. What is not understood is what is wrong
 with the moment.
[...] Indeed. If you could get dustmite to produce a minimal testcase that we can use to reproduce the problem locally, we'd be able to help you pinpoint exactly what is wrong with your code. Or determine that it's a GC bug, should it come to that. T -- Give a man a fish, and he eats once. Teach a man to fish, and he will sit forever.
Oct 23 2020
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/23/20 2:31 PM, donallen wrote:
 On Friday, 23 October 2020 at 15:49:49 UTC, Steven Schveighoffer wrote:
 On 10/23/20 8:47 AM, donallen wrote:
 This all looks ok. The only think I can possibly consider is that you 
 are using bind_text a lot, to presumably bind sqlite prepared 
 statements to D strings.

 This means it's *possible* that the prepared statements have pointers 
 to the strings, but nothing else does (hard to work it out from just 
 reading the code, I can't tell when things go out of scope easily, and 
 your prepared statements clearly exist outside these functions). If a 
 prepared statement is a C-malloc'd item, then it's not scanned by the 
 GC. This means that what possibly might be happening is that you bind 
 a D string to a text parameter, the D string goes out of scope, the GC 
 collects it and allocates it elsewhere, and then sqlite tries to use 
 it to send some requests.

 To test this theory, maybe try a version of the code that when calling 
 bind_text, you append the string itself to a GC allocated array stored 
 in a global. Might not prove anything, but if it's fixing the problem, 
 maybe that's where the GC is collecting things that it shouldn't.
It's a good theory, but I don't think it explains this. What you are hypothesizing is that there's an instance where the lifetime of a D string that is used in a sqlite bind_text is shorter than the lifetime of the sqlite binding. I don't think that's the case. The calls to bind_text all immediately precede calls to run_dm_stmt, one_row, etc. Those calls either run an insert or update, or in the case of one_row, a select. When those calls return, the statement has been reset, which means, among other things, that the bindings have been released, ending the binding's life.
OK, I didn't know how that worked.
 
 The other case to look at is the use of next_row_available_p in a loop. 
 There, bindings are established before entering the loop, which 
 processes multiple rows coming back from a select. When 
 next_row_available_p runs out of rows, it calls the cleanup function, 
 usually reset_stmt (which just calls sqlite3_reset; I found that I could 
 not pass an extern (c) function to next_row_available_p, so used this 
 workaround; maybe there is a way to do this, but I just used this simple 
 expedient).
extern(C) function types are different than extern(D) function types. So it depends on what the parameter type is.
 So at that point the bindings are cleared and 
 next_row_available_p returns false, ending the loop and life of the 
 bindings. Unless I've missed something, I think you will find that all 
 the strings passed to bind_text (which get massaged by toStringz, so 
 they look like C strings) live beyond the point where the statements 
 they are bound to are reset.
 
 Most of the calls to bind_text in the tree walker bind fields in 
 account, which is an argument to the tree walker and thus lives until it 
 returns. The others are to local variables that are declared before they 
 are used in bind-text calls and the statement gets reset before the 
 scope ends.
 
 Yes, your theory could be tested using some variant of what you 
 describe, in case my there's a flaw in my hand-waving above.
 
 Some food for thought:
 
 If I insert a writeln at the very beginning of walk_account_tree, 
 printing, say, the name and guid of the account, the problem goes away 
 -- the program behaves correctly.
as with many memory corruption issues, introducing new code paths can change where the corruption occurs, or change that the corruption occurs at all. You might want to try instead using printf, to avoid affecting anything in the D world. In the past to try and find corruption issues like this that are finnicky, I've also pre-allocated data to write to, and don't touch it for the duration of the problematic area, and then print it after.
 
 If I change
 children[i].path = format("%s:%s", account.path, children[i].name);
 to
 children[i].path = format("%s:%s", account.path, children[i].name).dup;
 
 in the loop that fills in the child Account structs in children, same 
 thing -- correct behavior. The dup of the format should not be 
 necessary, since format returns a new GC-allocated string.
Hm... that's interesting. You are correct that it allocates a new GC block. The dup is (should be) completely unnecessary. I don't think Appender (which is what format uses) has GC memory corruption issues, because it's used literally everywhere, it should be rock-solid. Always possible though.
 I suspect all of these odd "cures" have the effect they do because they 
 change the allocation pattern of this section of the code. Without any 
 of them, it appears that an allocation is triggering a GC at exactly the 
 wrong moment. What is not understood is what is wrong with the moment.
Yeah, I agree. For sure most of these constructs you are using are very well tested. There are many programs which use them, for many years, which still points me back at the sqlite library wrapper/binding. Not impossible that format and the GC have a bug, but it seems unlikely. Any time you have GC memory being prematurely collected, it smells like something was stored where it shouldn't be. The other major cause of this is generally alignment. But it doesn't seem like you are specifying alignment anywhere. In any case, I understand if you don't want to go through any more exercises, but if you do, it might help find something that is super-obscure. Thanks for taking the time to investigate as much as you have! -Steve
Oct 23 2020
prev sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
Thanks!

The problem is not in that block of code I believe.

Now how about bind_text?

Is that still using c variadic args?
Oct 23 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Friday, 23 October 2020 at 23:41:05 UTC, rikki cattermole 
wrote:
 Thanks!

 The problem is not in that block of code I believe.

 Now how about bind_text?

 Is that still using c variadic args?
No. Read on. The detective in me couldn't resist (I really need a Peugeot and a basset hound) and so I did some more work on this despite not really having the time and I have made some progress. The walk_account_tree function is defined inside the main function. Before it is defined, I prepare all the sql statements needed to do the verification, stored in local variables. One of the queries is count_children: "select count(guid) from accounts where parent_guid = ?1" That query is run in the section of walk_account_tree that we've been looking at: // Now do the children of this account // First determine how many there are bind_text(count_children, 1, account.guid); int n_children = one_row(count_children, &get_int).integer_value; if (n_children > 0) { Account[] children = new Account[n_children]; bind_text(find_children, 1, account.guid); So the result of that query is saved in n_children and determines the size of the children array. Recall that 'Account account' is the first argument to walk_account_tree, so n_children is the number of accounts that have account.guid, an account's unique identifier, as its parent. In other words, its children. If there are children, I allocate the children array and run the find_children query: "select name, guid, ifnull(commodity_guid, ''), flags from accounts where parent_guid = ?1"); The number of rows that come back ought to be same as n_children. On a hunch, I put in a test to check this: // Now do the children of this account // First determine how many there are bind_text(count_children, 1, account.guid); int n_children = one_row(count_children, &get_int).integer_value; if (n_children > 0) { Account[] children = new Account[n_children]; bind_text(find_children, 1, account.guid); int i = 0; while (next_row_available_p(find_children, &reset_stmt)) { children[i].name = fromStringz(sqlite3_column_text(find_children, 0)).dup; children[i].path = format("%s:%s", account.path, children[i].name); children[i].guid = fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid children[i].commodity_guid = fromStringz(sqlite3_column_text(find_children, 2)).dup; // commodity_guid children[i].flags = sqlite3_column_int(find_children, 3) | account.flags & (account_flag_descendents_are_assets | account_flag_descendents_are_liabilities | account_flag_descendents_are_income | account_flag_descendents_are_expenses | account_flag_descendents_are_marketable | account_flag_self_and_descendents_are_tax_related | account_flag_descendents_need_commodity_link); // flags i = i + 1; } if (i != n_children) panic(format("walk_account_tree: bad child index, %d, %d, %s", i, n_children, account.path)); foreach (child; children) { walk_account_tree(child, ancestor_flags | account.flags); } } And, sure enough, the panic happens: dca pangloss:~/Software/newcash_d/verifier$ ./verifier /tmp/Finances.newcash walk_account_tree: bad child index, 231, 344, :Assets:Investments:Equities and derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade dca pangloss:~/Software/newcash_d/verifier$ Something has caused sqlite to end the return of rows prematurely. But next_row_p did not panic, which it will if the the return code from sqlite3_step, which it calls, is other than SQLITE_ROW or SQLITE_DONE. Since next_row_p just returned 'false', sqlite3_step must have returned SQLITE_DONE. sqlite returned 231 of 344 rows but is acting like everything is normal. This explains why my earlier gdb session revealed that the error messages the verifier was spewing were due to being fed account structs that were all zero. The recursive calls are being done on all 344 of the elements of the children array, but only 231 have data in them. The rest are zero, as they were when the whole children array was allocated and those dataless "children" look bogus to the verifier. Now, I know from prior experiment that if I disable the gc, the problem does not occur, so I tried this again with the newly installed panic: dca pangloss:~/Software/newcash_d/verifier$ ./verifier "--DRT-gcopt=disable:1" /tmp/Finances.newcash dca pangloss:~/Software/newcash_d/verifier$ This was run with the same executable as the above. No panic. Next, I put in a call to GC.disable() just before the 'while' and a call to GC.enable{} after the loop returns. No panic. So I moved GC.disable and enable calls into the loop, at first with the disable call at the top of the loop, which did not panic. I moved it down one line at a time and when I get to this: // Now do the children of this account // First determine how many there are bind_text(count_children, 1, account.guid); int n_children = one_row(count_children, &get_int).integer_value; if (n_children > 0) { Account[] children = new Account[n_children]; bind_text(find_children, 1, account.guid); int i = 0; while (next_row_available_p(find_children, &reset_stmt)) { children[i].name = fromStringz(sqlite3_column_text(find_children, 0)).dup; children[i].path = format("%s:%s", account.path, children[i].name); GC.disable(); children[i].guid = fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid children[i].commodity_guid = fromStringz(sqlite3_column_text(find_children, 2)).dup; // commodity_guid children[i].flags = sqlite3_column_int(find_children, 3) | account.flags & (account_flag_descendents_are_assets | account_flag_descendents_are_liabilities | account_flag_descendents_are_income | account_flag_descendents_are_expenses | account_flag_descendents_are_marketable | account_flag_self_and_descendents_are_tax_related | account_flag_descendents_need_commodity_link); // flags i = i + 1; GC.enable(); } if (i != n_children) panic(format("walk_account_tree: bad child index, %d, %d, %s", i, n_children, account.path)); foreach (child; children) { walk_account_tree(child, ancestor_flags | account.flags); } } the panic occurs. Then I had another of my hunches. I re-wrote the line containing the call to format like so: // Now do the children of this account // First determine how many there are bind_text(count_children, 1, account.guid); int n_children = one_row(count_children, &get_int).integer_value; if (n_children > 0) { Account[] children = new Account[n_children]; bind_text(find_children, 1, account.guid); int i = 0; while (next_row_available_p(find_children, &reset_stmt)) { children[i].name = fromStringz(sqlite3_column_text(find_children, 0)).dup; size_t n_collections = GC.ProfileStats.numCollections; children[i].path = (account.path ~ ":" ~ children[i].name); bool collection_occured = GC.ProfileStats.numCollections != n_collections; children[i].guid = fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid children[i].commodity_guid = fromStringz(sqlite3_column_text(find_children, 2)).dup; // commodity_guid children[i].flags = sqlite3_column_int(find_children, 3) | account.flags & (account_flag_descendents_are_assets | account_flag_descendents_are_liabilities | account_flag_descendents_are_income | account_flag_descendents_are_expenses | account_flag_descendents_are_marketable | account_flag_self_and_descendents_are_tax_related | account_flag_descendents_need_commodity_link); // flags i = i + 1; } if (i != n_children) panic(format("walk_account_tree: bad child index, %d, %d, %s, %s", i, collection_occurred, n_children, account.path)); foreach (child; children) { walk_account_tree(child, ancestor_flags | account.flags); } } This does not panic! In theory, this should consume the same amount of GC memory as the format call (though I have not looked at the format code; there may well be intermediate allocations as it does its work). Now I decided to instrument the code so I could determine whether GCs were occurring as a result of the call to format: // Now do the children of this account // First determine how many there are bind_text(count_children, 1, account.guid); int n_children = one_row(count_children, &get_int).integer_value; if (n_children > 0) { size_t collections_before = 0; size_t collections_after = 0; Account[] children = new Account[n_children]; bind_text(find_children, 1, account.guid); int i = 0; while (next_row_available_p(find_children, &reset_stmt)) { children[i].name = fromStringz(sqlite3_column_text(find_children, 0)).dup; collections_before += GC.profileStats().numCollections; //children[i].path = (account.path ~ ":" ~ children[i].name); children[i].path = format("%s:%s", account.path, children[i].name); collections_after += GC.profileStats().numCollections; children[i].guid = fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid children[i].commodity_guid = fromStringz(sqlite3_column_text(find_children, 2)).dup; // commodity_guid children[i].flags = sqlite3_column_int(find_children, 3) | account.flags & (account_flag_descendents_are_assets | account_flag_descendents_are_liabilities | account_flag_descendents_are_income | account_flag_descendents_are_expenses | account_flag_descendents_are_marketable | account_flag_self_and_descendents_are_tax_related | account_flag_descendents_need_commodity_link); // flags i = i + 1; } if (i != n_children) panic(format("walk_account_tree: bad child index, %d, %d, %s, %d, %d", i, n_children, account.path, collections_before, collections_after)); if (account.path == ":Assets:Investments:Equities and derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade") { writeln(format("walk_account_tree: %d, %d, %s, %d, %d", i, n_children, account.path, collections_before, collections_after)); } foreach (child; children) { walk_account_tree(child, ancestor_flags | account.flags); } } Running this produces: dca pangloss:~/Software/newcash_d/verifier$ ./verifier "--DRT-gcopt=profile:1" /tmp/Finances.newcash walk_account_tree: bad child index, 231, 344, :Assets:Investments:Equities and derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade, 33, 34 dca pangloss:~/Software/newcash_d/verifier$ This proves that at least once during the loop that failed, a GC occurred as a result of the format call. Next I tried uncommenting the concatenation line and commenting out the format call. That produces: dca pangloss:~/Software/newcash_d/verifier$ ./verifier "--DRT-gcopt=profile:1" /tmp/Finances.newcash walk_account_tree: 344, 344, :Assets:Investments:Equities and derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade, 0, 0 Number of collections: 2 Total GC prep time: 0 milliseconds Total mark time: 0 milliseconds Total sweep time: 0 milliseconds Max Pause Time: 0 milliseconds Grand total GC time: 0 milliseconds GC summary: 5 MB, 2 GC 0 ms, Pauses 0 ms < 0 ms dca pangloss:~/Software/newcash_d/verifier$ So again, using concatenation instead of format fixes the problem. And no GCs have occurred after finishing the loop on the account at issue. And one other tidbit. If I add a precision to %s for the path in the format call, like so and change the test for the problem account by checking for its guid rather than the truncated path: // Now do the children of this account // First determine how many there are bind_text(count_children, 1, account.guid); int n_children = one_row(count_children, &get_int).integer_value; if (n_children > 0) { size_t collections_before = 0; size_t collections_after = 0; Account[] children = new Account[n_children]; bind_text(find_children, 1, account.guid); int i = 0; while (next_row_available_p(find_children, &reset_stmt)) { children[i].name = fromStringz(sqlite3_column_text(find_children, 0)).dup; collections_before += GC.profileStats().numCollections; //children[i].path = (account.path ~ ":" ~ children[i].name); children[i].path = format("%s:%.10s", account.path, children[i].name); collections_after += GC.profileStats().numCollections; children[i].guid = fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid children[i].commodity_guid = fromStringz(sqlite3_column_text(find_children, 2)).dup; // commodity_guid children[i].flags = sqlite3_column_int(find_children, 3) | account.flags & (account_flag_descendents_are_assets | account_flag_descendents_are_liabilities | account_flag_descendents_are_income | account_flag_descendents_are_expenses | account_flag_descendents_are_marketable | account_flag_self_and_descendents_are_tax_related | account_flag_descendents_need_commodity_link); // flags i = i + 1; } if (i != n_children) panic(format("walk_account_tree: bad child index, %d, %d, %s, %d, %d", i, n_children, account.path, collections_before, collections_after)); if (account.guid == "c369abf6ec2c7222e3fdd174ce2c0c9a") { writeln(format("walk_account_tree: %d, %d, %s, %d, %d", i, n_children, account.path, collections_before, collections_after)); } foreach (child; children) { walk_account_tree(child, ancestor_flags | account.flags); } } I get: dca pangloss:~/Software/newcash_d/verifier$ ./verifier "--DRT-gcopt=profile:1" /tmp/Finances.newcash walk_account_tree: 344, 344, :Assets:Investment:Equities a:Taxable:Donald C. :TD Ameritr, 0, 0 Number of collections: 3 Total GC prep time: 0 milliseconds Total mark time: 0 milliseconds Total sweep time: 0 milliseconds Max Pause Time: 0 milliseconds Grand total GC time: 1 milliseconds GC summary: 5 MB, 3 GC 1 ms, Pauses 1 ms < 0 ms dca pangloss:~/Software/newcash_d/verifier$ No panic. So there appears to be a problem in format when dealing with strings fed to an unconstrained %s, probably when interacting the garbage collector. My guess is that the issue is in format, not the gc. That's as far as I've gotten. But I think we know more now than before.
Oct 24 2020
next sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Saturday, 24 October 2020 at 14:21:52 UTC, donallen wrote:
 On Friday, 23 October 2020 at 23:41:05 UTC, rikki cattermole 
 wrote:
 [...]
No. Read on. [...]
Great writeup. Thanks for taking the time me to do it! (Iirc there's a nogc format somewhere in https://code.dlang.org/packages/ocean)
Oct 24 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/24/20 10:21 AM, donallen wrote:
 On Friday, 23 October 2020 at 23:41:05 UTC, rikki cattermole wrote:
 Thanks!

 The problem is not in that block of code I believe.

 Now how about bind_text?

 Is that still using c variadic args?
No. Read on. The detective in me couldn't resist (I really need a Peugeot and a basset hound) and so I did some more work on this despite not really having the time and I have made some progress.
Very nice! I wonder if the corruption is happening on the account.guid. Since we are talking about sqlite, which is a local DB, and not sending information to a server (where it can't be corrupted), it's possible the guid is being corrupted causing the premature exit of the loop. Can you print out the guid after in the premature exit case to make sure it hasn't changed?
 This does not panic! In theory, this should consume the same amount of 
 GC memory as the format call (though I have not looked at the format 
 code; there may well
 be intermediate allocations as it does its work).
Just for informational purposes, an append statement between N strings allocates one block that holds all of them. Format appends, so it may involve multiple allocations, though it's possible it does a similar shortcut. I'm very interested in this code, I will look at the case for format, and see if I can sleuth out any possible corruption possibilities.
 
 No panic. So there appears to be a problem in format when dealing with 
 strings fed to an unconstrained %s, probably when interacting the 
 garbage collector. My guess is that the issue is
 in format, not the gc.
Yeah, I tend to agree. More specifically, it could be in appender as well (which is what format uses to allocate memory).
 That's as far as I've gotten. But I think we know more now than before.
Good work, and I think we can probably get to the bottom of it, lots of great information! -Steve
Oct 24 2020
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/24/20 11:04 AM, Steven Schveighoffer wrote:
 On 10/24/20 10:21 AM, donallen wrote:
 This does not panic! In theory, this should consume the same amount of 
 GC memory as the format call (though I have not looked at the format 
 code; there may well
 be intermediate allocations as it does its work).
Just for informational purposes, an append statement between N strings allocates one block that holds all of them. Format appends, so it may involve multiple allocations, though it's possible it does a similar shortcut. I'm very interested in this code, I will look at the case for format, and see if I can sleuth out any possible corruption possibilities.
Can you try this? This will remove format from the picture, but still use appender in the same way format does (to focus on whether the problem is in std.format or appender): // instead of this // children[i].path = format("%s:%s", account.path, children[i].name); { // need a scope to ensure appender goes out of scope import std.array; appender!string app; app.put(account.path); app.put(":"); app.put(children[i].name); children[i].path = app[]; } If this still panics, then I can eliminate all of std.format. If it doesn't panic, then it's most likely in std.format. A closer look at appender, and it seems all correct. I'm kinda hoping this doesn't panic, but at the same time, the appender code is a lot less complex, so easier to examine closely. -Steve
Oct 24 2020
prev sibling parent reply donallen <donaldcallen gmail.com> writes:
On Saturday, 24 October 2020 at 15:04:53 UTC, Steven 
Schveighoffer wrote:
 On 10/24/20 10:21 AM, donallen wrote:
 On Friday, 23 October 2020 at 23:41:05 UTC, rikki cattermole 
 wrote:
 Thanks!

 The problem is not in that block of code I believe.

 Now how about bind_text?

 Is that still using c variadic args?
No. Read on. The detective in me couldn't resist (I really need a Peugeot and a basset hound) and so I did some more work on this despite not really having the time and I have made some progress.
Very nice! I wonder if the corruption is happening on the account.guid. Since we are talking about sqlite, which is a local DB, and not sending information to a server (where it can't be corrupted), it's possible the guid is being corrupted causing the premature exit of the loop. Can you print out the guid after in the premature exit case to make sure it hasn't changed?
A good thought, but it doesn't change: // Now do the children of this account // First determine how many there are string guid_before = account.guid; bind_text(count_children, 1, account.guid); int n_children = one_row(count_children, &get_int).integer_value; if (n_children > 0) { size_t collections_before = 0; size_t collections_after = 0; Account[] children = new Account[n_children]; string guid_after = account.guid; bind_text(find_children, 1, account.guid); int i = 0; while (next_row_available_p(find_children, &reset_stmt)) { children[i].name = fromStringz(sqlite3_column_text(find_children, 0)).dup; collections_before += GC.profileStats().numCollections; //children[i].path = (account.path ~ ":" ~ children[i].name); children[i].path = format("%s:%s", account.path, children[i].name); collections_after += GC.profileStats().numCollections; children[i].guid = fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid children[i].commodity_guid = fromStringz(sqlite3_column_text(find_children, 2)).dup; // commodity_guid children[i].flags = sqlite3_column_int(find_children, 3) | account.flags & (account_flag_descendents_are_assets | account_flag_descendents_are_liabilities | account_flag_descendents_are_income | account_flag_descendents_are_expenses | account_flag_descendents_are_marketable | account_flag_self_and_descendents_are_tax_related | account_flag_descendents_need_commodity_link); // flags i = i + 1; } if (i != n_children) panic(format("walk_account_tree: bad child index, %d, %d, %s, %s, %s, %d, %d", i, n_children, account.path, guid_before, guid_after, collections_before, collections_after)); if (account.guid == "c369abf6ec2c7222e3fdd174ce2c0c9a") { writeln(format("walk_account_tree: %d, %d, %s, %d, %d", i, n_children, account.path, collections_before, collections_after)); } foreach (child; children) { walk_account_tree(child, ancestor_flags | account.flags); } dca pangloss:~/Software/newcash_d/verifier$ ./verifier "--DRT-gcopt=profile:1" /tmp/Finances.newcash walk_account_tree: bad child index, 231, 344, :Assets:Investments:Equities and derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade, c369abf6ec2c7222e3fdd174ce2c0c9a, c369abf6ec2c7222e3fdd174ce2c0c9a, 33, 34
 This does not panic! In theory, this should consume the same 
 amount of GC memory as the format call (though I have not 
 looked at the format code; there may well
 be intermediate allocations as it does its work).
Just for informational purposes, an append statement between N strings allocates one block that holds all of them. Format appends, so it may involve multiple allocations, though it's possible it does a similar shortcut. I'm very interested in this code, I will look at the case for format, and see if I can sleuth out any possible corruption possibilities.
 
 No panic. So there appears to be a problem in format when 
 dealing with strings fed to an unconstrained %s, probably when 
 interacting the garbage collector. My guess is that the issue 
 is
 in format, not the gc.
Yeah, I tend to agree. More specifically, it could be in appender as well (which is what format uses to allocate memory).
 That's as far as I've gotten. But I think we know more now 
 than before.
Good work, and I think we can probably get to the bottom of it, lots of great information! -Steve
Oct 24 2020
parent reply Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Saturday, 24 October 2020 at 16:00:24 UTC, donallen wrote:
 On Saturday, 24 October 2020 at 15:04:53 UTC, Steven 
 Schveighoffer wrote:
 [...]
A good thought, but it doesn't change: [...]
Have you tried formattedWrite or the nogc package?
Oct 24 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Saturday, 24 October 2020 at 20:00:42 UTC, Imperatorn wrote:
 On Saturday, 24 October 2020 at 16:00:24 UTC, donallen wrote:
 On Saturday, 24 October 2020 at 15:04:53 UTC, Steven 
 Schveighoffer wrote:
 [...]
A good thought, but it doesn't change: [...]
Have you tried formattedWrite or the nogc package?
No. At this point, the verifier works and I have stopped further porting to D of my financial tools.
Oct 25 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Sunday, 25 October 2020 at 17:03:00 UTC, donallen wrote:
 On Saturday, 24 October 2020 at 20:00:42 UTC, Imperatorn wrote:
 On Saturday, 24 October 2020 at 16:00:24 UTC, donallen wrote:
 On Saturday, 24 October 2020 at 15:04:53 UTC, Steven 
 Schveighoffer wrote:
 [...]
A good thought, but it doesn't change: [...]
Have you tried formattedWrite or the nogc package?
No. At this point, the verifier works and I have stopped further porting to D of my financial tools.
I should clarify that I've stopped further work with D because of the issue that came up in trying to port my verifier application. I am hoping that the information I provided over the weekend will enable someone to come up with a definitive explanation of what happened. If that occurs, I will then decide whether I want to continue porting my code to D. I will add that I am impressed with the language -- a big improvement over C and C++, and with the documentation and the helpfulness of the community. I'm just concerned, based on the verifier experience, that D doesn't have the critical mass needed to make a complex system rock solid. Perhaps the explanation of the verifier problem will change that.
Oct 26 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/26/20 10:17 AM, donallen wrote:
 I should clarify that I've stopped further work with D because of the 
 issue that came up in trying to port my verifier application. I am 
 hoping that the information I provided over the weekend will enable 
 someone to come up with a definitive explanation of what happened. If 
 that occurs, I will then decide whether I want to continue porting my 
 code to D.
Thanks for your reports. Unfortunately, without something reproducible locally, there's not much we can do without your help. As you know I'm sure, I can't fix problems without being able to test solutions. So far, nothing has jumped out as an obvious error in your code. If you have time to run some more tests, then I would appreciate it. In particular, if you could try this test, it could narrow it down a bit: https://forum.dlang.org/post/rn1ie3$2363$1 digitalmars.com Alternatively, if you feel comfortable with it, you can send me the code/data that fails (or maybe dummy data that also fails), and I can keep looking at it. My email is schveiguy at gmail dot com. -Steve
Oct 26 2020
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
Basically what Steven said, is what I was going to reply.

We need to work with the full code, I understand that it is probably 
proprietary. If this is a concern, perhaps somebody associated with the 
D Language Foundation will be willing to offer some help, assuming that 
is some comfort.
Oct 26 2020
prev sibling parent reply donallen <donaldcallen gmail.com> writes:
On Monday, 26 October 2020 at 14:33:15 UTC, Steven Schveighoffer 
wrote:
 On 10/26/20 10:17 AM, donallen wrote:
 I should clarify that I've stopped further work with D because 
 of the issue that came up in trying to port my verifier 
 application. I am hoping that the information I provided over 
 the weekend will enable someone to come up with a definitive 
 explanation of what happened. If that occurs, I will then 
 decide whether I want to continue porting my code to D.
Thanks for your reports. Unfortunately, without something reproducible locally, there's not much we can do without your help. As you know I'm sure, I can't fix problems without being able to test solutions. So far, nothing has jumped out as an obvious error in your code. If you have time to run some more tests, then I would appreciate it. In particular, if you could try this test, it could narrow it down a bit: https://forum.dlang.org/post/rn1ie3$2363$1 digitalmars.com
Sorry, I missed this suggestion in your earlier message. I'll give it a try. I'll let you know what happens.
 Alternatively, if you feel comfortable with it, you can send me 
 the code/data that fails (or maybe dummy data that also fails), 
 and I can keep looking at it.
As I said earlier, the code is not the issue, it's the data. If we get to this bridge and have to cross it, I'll look trying to cook up a dummy database that provokes the problem.
 My email is schveiguy at gmail dot com.

 -Steve
Oct 26 2020
parent reply donallen <donaldcallen gmail.com> writes:
With the help of an excellent suggestion by Steve Schveighoffer, 
I finally identified this problem yesterday. It was my error, not 
D's.

I will spare you all the details. The problem is in this function:

void bind_text(sqlite3_stmt* stmt, int index, const char[] 
the_text)
{
     if (sqlite3_bind_text(stmt, index, toStringz(the_text), -1, 
null)
!= sqlite_ok)
     {
         stderr.writeln("bind_text failed");
         auto db = sqlite3_db_handle(stmt);
         stderr.writeln("Error: %s", 
fromStringz(sqlite3_errmsg(db)));
         exit(EXIT_FAILURE);
     }
}

which takes a sqlite prepared statement, a query parameter index 
and a D string and attempts to convert the D string to a 
null-terminated C string and call sqlite3_bind_text to bind the C 
string to the indicated query parameter.

The culprit is the call to toStringz. The documentation for that
function give some good advice that I failed to follow:
"Important Note: When passing a char* to a C function, and the C
function keeps it around for any reason, make sure that you keep a
reference to it in your D code. Otherwise, it may become invalid
during a garbage collection cycle and cause a nasty bug when the C
code tries to use it."

That's precisely what is happening here. There is no reference to 
the
string returned by the toStringz call retained and if a GC happens
during a loop retrieving all the rows of a query, the C version 
of the
string that has been passed to sqlite by the sqlite3_bind_text 
call gets
freed by the GC before sqlite is done with it. So now the sqlite 
query
no longer has a proper value for the query parameter, and all 
bets are
off as far as the query behaving correctly. This also explains 
the mystery of why the loop
terminated prematurely when I first discovered this problem.

I fixed it by having bind_text return the C string to its caller, 
to be stored in a variable to protect it from the garbage 
collector. The original error, which was reproducible, is now 
gone. Steve's suggestion, which was to insert frequent calls to 
the garbage collector, provoked a different error reliably that 
is also gone with the fix.

Thanks to everyone who tried to help figure this one out.
Nov 15 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Nov 15, 2020 at 01:43:14PM +0000, donallen via Digitalmars-d wrote:
 With the help of an excellent suggestion by Steve Schveighoffer, I
 finally identified this problem yesterday. It was my error, not D's.
[...]
 void bind_text(sqlite3_stmt* stmt, int index, const char[] the_text)
 {
     if (sqlite3_bind_text(stmt, index, toStringz(the_text), -1, null)
 != sqlite_ok)
     {
         stderr.writeln("bind_text failed");
         auto db = sqlite3_db_handle(stmt);
         stderr.writeln("Error: %s", fromStringz(sqlite3_errmsg(db)));
         exit(EXIT_FAILURE);
     }
 }
[...]
 The culprit is the call to toStringz. The documentation for that
 function give some good advice that I failed to follow:
 "Important Note: When passing a char* to a C function, and the C
 function keeps it around for any reason, make sure that you keep a
 reference to it in your D code. Otherwise, it may become invalid
 during a garbage collection cycle and cause a nasty bug when the C
 code tries to use it."
Ouch. I'm dismayed that I missed this when I last looked at this code (which I believe you did post here). ;-) Passing a GC-allocated string to C code without retaining a reference somewhere on the D side is a known gotcha when interfacing with C. In fact, I'm almost certain I ran into this very problem with toStringz myself in the past, yet I still failed to notice it when I looked at your code. In any case, I'm glad that this means you'll be around here more. ;-) --T
Nov 15 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/15/20 10:05 AM, H. S. Teoh wrote:
 On Sun, Nov 15, 2020 at 01:43:14PM +0000, donallen via Digitalmars-d wrote:
 With the help of an excellent suggestion by Steve Schveighoffer, I
 finally identified this problem yesterday. It was my error, not D's.
[...]
 void bind_text(sqlite3_stmt* stmt, int index, const char[] the_text)
 {
      if (sqlite3_bind_text(stmt, index, toStringz(the_text), -1, null)
 != sqlite_ok)
      {
          stderr.writeln("bind_text failed");
          auto db = sqlite3_db_handle(stmt);
          stderr.writeln("Error: %s", fromStringz(sqlite3_errmsg(db)));
          exit(EXIT_FAILURE);
      }
 }
[...]
 The culprit is the call to toStringz. The documentation for that
 function give some good advice that I failed to follow:
 "Important Note: When passing a char* to a C function, and the C
 function keeps it around for any reason, make sure that you keep a
 reference to it in your D code. Otherwise, it may become invalid
 during a garbage collection cycle and cause a nasty bug when the C
 code tries to use it."
Ouch. I'm dismayed that I missed this when I last looked at this code (which I believe you did post here). ;-) Passing a GC-allocated string to C code without retaining a reference somewhere on the D side is a known gotcha when interfacing with C. In fact, I'm almost certain I ran into this very problem with toStringz myself in the past, yet I still failed to notice it when I looked at your code.
I had the same reaction! I also did not see the toStringz as a problem when I reviewed. I will note, also, that toStringz has this quirk where in certain cases it just returns the pointer of the array, which is why I think the .idup fix worked (that was puzzling to me and I think pulled me away from it being a problem with storing a pointer in C-land only). https://github.com/dlang/phobos/blob/e89d9d9ce072269e80da4f901ce3e953d736cb82/std/string.d#L361-L367
 In any case, I'm glad that this means you'll be around here more. ;-)
Likewise! -Steve
Nov 15 2020
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 11/15/20 10:43 AM, Steven Schveighoffer wrote:

 I also did not see the toStringz as a problem
 when I reviewed.
While I have everbody's attention, :) toStringz will come up during one of my DConf presentations. I will claim that it is safe to pass to a C function for their immediate consumption (but they can't save the pointer). So, I will claim that the following is safe and no GC collection on another thread can mess this up. (The C side is not allowed to store for later use but they are taking the pointer into a local variable on their function call stack.) nothrow extern(C) void bar(const(char) ** name) { // ... *name = makeString(42).toStringz; // ... } My claim is based on my understanding on the following thread. (Thank you, Rikki Cattermole.) https://forum.dlang.org/thread/rn3ii0$jjj$1 digitalmars.com Ali
Nov 15 2020
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Nov 15, 2020 at 01:46:52PM -0800, Ali ehreli via Digitalmars-d wrote:
[...]
 While I have everbody's attention, :) toStringz will come up during
 one of my DConf presentations. I will claim that it is safe to pass to
 a C function for their immediate consumption (but they can't save the
 pointer).
 
 So, I will claim that the following is safe and no GC collection on
 another thread can mess this up. (The C side is not allowed to store
 for later use but they are taking the pointer into a local variable on
 their function call stack.)
 
 nothrow extern(C) void bar(const(char) ** name) {
   // ...
   *name = makeString(42).toStringz;
   // ...
 }
[...] AFAIK, as long as a reference exists on the stack, you should be safe, because the GC does scan the stack. But if the C side stores it anywhere else past the point where the on-stack reference goes out of scope, then you're in trouble. T -- Chance favours the prepared mind. -- Louis Pasteur
Nov 16 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/16/20 2:26 PM, H. S. Teoh wrote:
 On Sun, Nov 15, 2020 at 01:46:52PM -0800, Ali Çehreli via Digitalmars-d wrote:
 [...]
 While I have everbody's attention, :) toStringz will come up during
 one of my DConf presentations. I will claim that it is safe to pass to
 a C function for their immediate consumption (but they can't save the
 pointer).

 So, I will claim that the following is safe and no GC collection on
 another thread can mess this up. (The C side is not allowed to store
 for later use but they are taking the pointer into a local variable on
 their function call stack.)

 nothrow extern(C) void bar(const(char) ** name) {
    // ...
    *name = makeString(42).toStringz;
    // ...
 }
[...] AFAIK, as long as a reference exists on the stack, you should be safe, because the GC does scan the stack. But if the C side stores it anywhere else past the point where the on-stack reference goes out of scope, then you're in trouble.
I see a problem here, in that the stack is changeable by the C function. I can imagine a situation where it removes the item from the stack to put it somewhere else (e.g. global memory or on the C heap) *temporarily*, and by the time the function exits, the pointer is no longer in use in C-land. Thus, the function doesn't "save the pointer". This kind of problem would still be susceptible to the GC collecting it (via another thread), and so I would recommend still storing locally in the calling function a copy of the pointer. -Steve
Nov 16 2020
next sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 11/16/20 1:25 PM, Steven Schveighoffer wrote:

 I can imagine a situation where it removes the item from the stack to
 put it somewhere else (e.g. global memory or on the C heap)
 *temporarily*
Good point. For example, they can provide the address of the member of a heap-allocated object: Obj * obj = malloc(/* ... */); D_function(&obj.member); // <-- Another thread triggers a // GC collection at this point. printf("%d", obj.member); // <-- Potentially too late. This is hard to communicate to the callers. I said "safe to use for immediate consumption" but still not safe. Documenting like "pointer to stack allocated data only please" may be too confusing for casual users. So, the safest thing would be to make a copy with toStringz. Of course, strings are just one type. I pass GC allocated large buffers to the C side without copying. Luckily, I am the only consumer so far and things seem to work. :) Ali
Nov 16 2020
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 16 November 2020 at 21:25:03 UTC, Steven Schveighoffer 
wrote:
 I can imagine a situation where it removes the item from the 
 stack to put it somewhere else (e.g. global memory or on the C 
 heap) *temporarily*, and by the time the function exits, the 
 pointer is no longer in use in C-land. Thus, the function 
 doesn't "save the pointer".

 This kind of problem would still be susceptible to the GC 
 collecting it (via another thread), and so I would recommend 
 still storing locally in the calling function a copy of the 
 pointer.
How do you express that? LLVM can remove unused pointers? There are intrinsics to combat that, but is there something portable in D?
Nov 16 2020
prev sibling next sibling parent mw <mingwu gmail.com> writes:
On Sunday, 15 November 2020 at 13:43:14 UTC, donallen wrote:
 With the help of an excellent suggestion by Steve 
 Schveighoffer, I finally identified this problem yesterday. It 
 was my error, not D's.
Glad you fixed the problem. And it looks like you missed my post: https://forum.dlang.org/post/hqpppumvdtkbmghavsoq forum.dlang.org D has been around for ~20 years, and there are 1899 packages, some of them under heavy industrial use, it's very unlikely it's a GC bug, if there is a bug, people may run into it long before you do esp you are new to D. https://code.dlang.org/
Nov 15 2020
prev sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
Wahoo!
Nov 15 2020
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Oct 24, 2020 at 02:21:52PM +0000, donallen via Digitalmars-d wrote:
[...]
 The detective in me couldn't resist (I really need a Peugeot and a
 basset hound) and so I did some more work on this despite not really
 having the time and I have made some progress.
[...]
 Something has caused sqlite to end the return of rows prematurely. But
 next_row_p did not panic, which it will if the the return code from
 sqlite3_step, which it calls, is other than SQLITE_ROW or SQLITE_DONE.
 Since next_row_p just returned 'false', sqlite3_step must have
 returned SQLITE_DONE. sqlite returned 231 of 344 rows but is acting
 like everything is normal.
[...] Very interesting finding indeed!
 Then I had another of my hunches. I re-wrote the line containing the
 call to format like so:
[...]
                 children[i].path = (account.path ~ ":" ~ children[i].name);
[...]
 This does not panic! In theory, this should consume the same amount of
 GC memory as the format call (though I have not looked at the format
 code; there may well be intermediate allocations as it does its work).
Am I right in assuming that account.path and children[i].name are both of type `string`? As opposed to some other object that defined a toString method? AFAIK, format does allocate intermediate buffers for its work; that may explain the difference you observed here.
 Now I decided to instrument the code so I could determine whether GCs
 were occurring as a result of the call to format:
[...]
 Running this produces:
 dca pangloss:~/Software/newcash_d/verifier$ ./verifier
 "--DRT-gcopt=profile:1" /tmp/Finances.newcash
 walk_account_tree: bad child index, 231, 344, :Assets:Investments:Equities
 and derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade,
 33, 34
 dca pangloss:~/Software/newcash_d/verifier$
 
 This proves that at least once during the loop that failed, a GC
 occurred as a result of the format call.
[...] Here, there appears to have been 33 GC collections before the format call, and 34 after. Am I reading this right?
 Next I tried uncommenting the concatenation line and commenting out
 the format call. That produces:
 dca pangloss:~/Software/newcash_d/verifier$ ./verifier
 "--DRT-gcopt=profile:1" /tmp/Finances.newcash
 walk_account_tree: 344, 344, :Assets:Investments:Equities and
 derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade, 0, 0
[...]
 So again, using concatenation instead of format fixes the problem. And
 no GCs have occurred after finishing the loop on the account at issue.
However, here there appears to have been no collections at all (0 before, 0 after)! So based on this we still cannot rule out a problem in the GC. Which makes me wonder: what would happen if you inserted: if (account.path == ":Assets:Investments:Equities and derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade") GC.collect(); right after the concatenation line to trigger a collection? Would it reintroduce the panic?
 And one other tidbit. If I add a precision to %s for the path in the
 format call, like so and change the test for the problem account by
 checking for its guid rather than the truncated path:
[...]
                 children[i].path = format("%s:%.10s", account.path,
 children[i].name);
[...]
 I get:
 dca pangloss:~/Software/newcash_d/verifier$ ./verifier
 "--DRT-gcopt=profile:1" /tmp/Finances.newcash
 walk_account_tree: 344, 344, :Assets:Investment:Equities a:Taxable:Donald C.
 :TD Ameritr, 0, 0
[...]
 No panic. So there appears to be a problem in format when dealing with
 strings fed to an unconstrained %s, probably when interacting the
 garbage collector. My guess is that the issue is in format, not the
 gc.
Interestingly enough, again there appears to be no collections done at all (0 before, 0 after). So I'm still not sure if we could entirely rule out the GC at this point. It would be more interesting to find out what happens if you insert the GC.collect() line to manually trigger a collection with the constrained version of the format call.
 That's as far as I've gotten. But I think we know more now than
 before.
This is extremely informative, and thank you for taking the time to do this in spite of being very busy. I hope we can eventually ferret out the cause of the problem, whether it's in the GC, in format, or in your code somehow. T -- Error: Keyboard not attached. Press F1 to continue. -- Yoon Ha Lee, CONLANG
Oct 24 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/21/20 12:24 PM, donallen wrote:
 The problem is that at some point, the verifier starts spewing bogus 
 error messages about what it is seeing in the tree. Oddly, putting in 
 debugging writelns results in the error messages not occurring -- a 
 Heisenbug.  But working with gdb, I found that the account structs after 
 the error messages start are zeroed. Turning on gc profiling tells me 
 that 3 gcs have occurred. Disabling the gc results in the program 
 running correctly -- no error messages (and I know the database has no 
 errors because the C version, which has been around for awhile, confirms 
 that the db is well formed).
 
 I could post a PR, but I'm not sure that this is a bug. It could easily 
 be a misunderstanding by me of D and its memory management. So I thought 
 I'd try this post first, hoping that one of you who knows the language 
 better than I do could point out a problem with my code. I do need to 
 resolve this or abandon this project.
 
 Thanks in advance for any help.
There seems to be nothing wrong in this code that I can see. But of course, it's out of context, so it's hard to see if there are issues elsewhere. These kinds of corruption bugs are really hard to track down. What does your Account struct look like? -Steve
Oct 22 2020
parent reply donallen <donaldcallen gmail.com> writes:
On Thursday, 22 October 2020 at 18:24:57 UTC, Steven 
Schveighoffer wrote:
 On 10/21/20 12:24 PM, donallen wrote:
 The problem is that at some point, the verifier starts spewing 
 bogus error messages about what it is seeing in the tree. 
 Oddly, putting in debugging writelns results in the error 
 messages not occurring -- a Heisenbug.  But working with gdb, 
 I found that the account structs after the error messages 
 start are zeroed. Turning on gc profiling tells me that 3 gcs 
 have occurred. Disabling the gc results in the program running 
 correctly -- no error messages (and I know the database has no 
 errors because the C version, which has been around for 
 awhile, confirms that the db is well formed).
 
 I could post a PR, but I'm not sure that this is a bug. It 
 could easily be a misunderstanding by me of D and its memory 
 management. So I thought I'd try this post first, hoping that 
 one of you who knows the language better than I do could point 
 out a problem with my code. I do need to resolve this or 
 abandon this project.
 
 Thanks in advance for any help.
There seems to be nothing wrong in this code that I can see. But of course, it's out of context, so it's hard to see if there are issues elsewhere. These kinds of corruption bugs are really hard to track down. What does your Account struct look like?
struct Account { string name; string path; string guid; string commodity_guid; int flags; }
 -Steve
Oct 22 2020
next sibling parent reply mw <mingwu gmail.com> writes:
On Thursday, 22 October 2020 at 19:34:18 UTC, donallen wrote:
 What does your Account struct look like?
struct Account { string name; string path; string guid; string commodity_guid; int flags; }
Do you pass any variable from D to C? and how does those variables look like? And if you are passing any D object to C, do you always hold a reference to them before the C returns? I mean: the D's gc think the object is no longer used, but actually is used by the C side.
Oct 22 2020
parent mw <mingwu gmail.com> writes:
On Thursday, 22 October 2020 at 19:57:08 UTC, mw wrote:
 Do you pass any variable from D to C? and how does those 
 variables look like?

 And if you are passing any D object to C, do you always hold a 
 reference to them before the C returns? I mean: the D's gc
s/before/after
 think the object is no longer used, but actually is used by the 
 C side.
Oct 22 2020
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 22 October 2020 at 19:34:18 UTC, donallen wrote:
     struct Account
     {
         string name;
         string path;
         string guid;
         string commodity_guid;
         int flags;
     }
Why did you call walk_account_tree on this, what does it do? Account has no pointers?
Oct 22 2020
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/21/20 12:24 PM, donallen wrote:

 The problem is that at some point, the verifier starts spewing bogus 
 error messages about what it is seeing in the tree. Oddly, putting in 
 debugging writelns results in the error messages not occurring -- a 
 Heisenbug.  But working with gdb, I found that the account structs after 
 the error messages start are zeroed. Turning on gc profiling tells me 
 that 3 gcs have occurred. Disabling the gc results in the program 
 running correctly -- no error messages (and I know the database has no 
 errors because the C version, which has been around for awhile, confirms 
 that the db is well formed).
Something I thought of - try running manual GC collections: import core.memory; GC.collect(); // peppered around your code It might make your code fail closer to the true problem. -Steve
Oct 23 2020