www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - core.exception.InvalidMemoryOperationError (0)

reply "Bayan Rafeh" <bayan.rafeh92 gmail.com> writes:
This problem is a tough one. I've been getting this error when I 
run my unittests, and apparently it is caused by attempting an 
allocation in the destructor from what little I could find online 
about the subject.

The error is triggered once all my tests are complete, so I'm 
assuming the garbage collector is running before termination, but 
I tried placing logging messages in all my destructors and I 
didn't get anything, so none of them are being called.

Is there any other possible reason to get this error?
Jan 24 2015
next sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 01/24/2015 04:16 AM, Bayan Rafeh wrote:
 This problem is a tough one. I've been getting this error when I run my
 unittests, and apparently it is caused by attempting an allocation in
 the destructor from what little I could find online about the subject.

 The error is triggered once all my tests are complete, so I'm assuming
 the garbage collector is running before termination, but I tried placing
 logging messages in all my destructors and I didn't get anything, so
 none of them are being called.

 Is there any other possible reason to get this error?
The allocations may be indirect. For example, formatting a string to log a message may allocate. Try marking your destructors as nogc to see whether the compiler agrees. I am not sure whether it would cause the same error but another reason for the error is touching reference members of a class, which may have been finalized before the object. Ali
Jan 24 2015
parent =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
On Saturday, 24 January 2015 at 22:42:51 UTC, Ali Çehreli wrote:
 The allocations may be indirect. For example, formatting a 
 string to log a message may allocate. Try marking your 
 destructors as  nogc to see whether the compiler agrees.

 I am not sure whether it would cause the same error but another 
 reason for the error is touching reference members of a class, 
 which may have been finalized before the object.

 Ali
Strange, I have no explicit destructors declared in my code (classes/structs). I'm making rich usage of std.container.Array, thought...
Jan 24 2015
prev sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Saturday, 24 January 2015 at 12:16:38 UTC, Bayan Rafeh wrote:
 This problem is a tough one. I've been getting this error when 
 I run my unittests, and apparently it is caused by attempting 
 an allocation in the destructor from what little I could find 
 online about the subject.

 The error is triggered once all my tests are complete, so I'm 
 assuming the garbage collector is running before termination, 
 but I tried placing logging messages in all my destructors and 
 I didn't get anything, so none of them are being called.

 Is there any other possible reason to get this error?
Hi, I created a wiki page which I hope will help you solve this problem: http://wiki.dlang.org/InvalidMemoryOperationError Hope this helps.
Jan 24 2015
parent reply "Bayan Rafeh" <bayan.rafeh92 gmail.com> writes:
On Sunday, 25 January 2015 at 00:43:43 UTC, Vladimir Panteleev 
wrote:
 On Saturday, 24 January 2015 at 12:16:38 UTC, Bayan Rafeh wrote:
 This problem is a tough one. I've been getting this error when 
 I run my unittests, and apparently it is caused by attempting 
 an allocation in the destructor from what little I could find 
 online about the subject.

 The error is triggered once all my tests are complete, so I'm 
 assuming the garbage collector is running before termination, 
 but I tried placing logging messages in all my destructors and 
 I didn't get anything, so none of them are being called.

 Is there any other possible reason to get this error?
Hi, I created a wiki page which I hope will help you solve this problem: http://wiki.dlang.org/InvalidMemoryOperationError Hope this helps.
Thank you very much, this will certainly be useful in debugging it.
 The allocations may be indirect. For example, formatting a 
 string to log a message may allocate. Try marking your 
 destructors as  nogc to see whether the compiler agrees.
I tried what you said and I think I see the problem. I managed to create an example program that duplicates the problem: import std.stdio; class A { string path; this(string p) { path = p; } ~this() { } void a(){} void b(){} } class B { A a; this() { this.a = new A("laladiv"); } ~this() { delete a; } } void main() { B a = new B(); B b = new B(); delete b; }
Jan 25 2015
parent reply "Bayan Rafeh" <bayan.rafeh92 gmail.com> writes:
On Sunday, 25 January 2015 at 08:39:42 UTC, Bayan Rafeh wrote:
 On Sunday, 25 January 2015 at 00:43:43 UTC, Vladimir Panteleev 
 wrote:
 On Saturday, 24 January 2015 at 12:16:38 UTC, Bayan Rafeh 
 wrote:
 This problem is a tough one. I've been getting this error 
 when I run my unittests, and apparently it is caused by 
 attempting an allocation in the destructor from what little I 
 could find online about the subject.

 The error is triggered once all my tests are complete, so I'm 
 assuming the garbage collector is running before termination, 
 but I tried placing logging messages in all my destructors 
 and I didn't get anything, so none of them are being called.

 Is there any other possible reason to get this error?
Hi, I created a wiki page which I hope will help you solve this problem: http://wiki.dlang.org/InvalidMemoryOperationError Hope this helps.
Thank you very much, this will certainly be useful in debugging it.
 The allocations may be indirect. For example, formatting a 
 string to log a message may allocate. Try marking your 
 destructors as  nogc to see whether the compiler agrees.
I tried what you said and I think I see the problem. I managed to create an example program that duplicates the problem: import std.stdio; class A { string path; this(string p) { path = p; } ~this() { } void a(){} void b(){} } class B { A a; this() { this.a = new A("laladiv"); } ~this() { delete a; } } void main() { B a = new B(); B b = new B(); delete b; }
The solution was just to remove the "delete a" from the destructor if someone comes across this later. Could someone tell me why though?
Jan 25 2015
next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sunday, 25 January 2015 at 08:41:25 UTC, Bayan Rafeh wrote:
 The solution was just to remove the "delete a" from the 
 destructor if someone comes across this later. Could someone 
 tell me why though?
The "non-reentrant" bit applies to all GC operations, really - not just allocations. Explicit deletions are forbidden as well. Use of the "delete" keyword is discouraged, and in that context, it is also not used in a safe way. By the time B's destructor is called, the A instance might have been already collected by the garbage collector, causing a double free bug (one of the reasons to not use delete).
Jan 25 2015
prev sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Sun, 25 Jan 2015 08:41:24 +0000, Bayan Rafeh wrote:

 I tried what you said and I think I see the problem. I managed to
 create an example program that duplicates the problem:


 import std.stdio;

 class A {
     string path;

     this(string p) {
         path =3D p;
     }
     ~this() {
     }
     void a(){}

     void b(){}
 }

 class B {
     A a;
     this() {
         this.a =3D new A("laladiv");
     }
     ~this() {
         delete a;
     }
 }

 void main() {
     B a =3D new B(); B b =3D new B(); delete b;
 }
=20 The solution was just to remove the "delete a" from the destructor if someone comes across this later. Could someone tell me why though?
there is no guarantees on destruction order. and GC will not nullify any=20 references to collected data. so, `a` can be already collected and=20 finalized when `B.~this()` is called. yet reference is still there, so=20 `delete a;` will try to delete already dead object. this will lead to=20 crash. without precise GC collector is not able to automatically nullify all=20 "dead" references. and even if there will be such possibility, it can=20 slow down collections alot (GC will nullifying alot of references that=20 aren't used anyway), so i don't think that it do nullifying. there is a simple rule: "dtor should not touch GC-managed resources".=20 this will not give you "predictable destruction order" (that's why most=20 people try to manually delete something in dtor), and this simply will=20 not work at all. if you want predictable destruction order, don't use GC at all, use=20 manual memory management. it doesn't matter which hack you will invent to=20 force destruction order, any hack will either be very fragile, or will=20 not work. this is due to nature of GC-manged memory. so: don't use GC-managed resources in dtors. don't use in any way -- this=20 including accessing 'em. i.e. reading `a.path` in dtor is invalid too. it=20 will not necessarily crash, but it's the source of "use after free" error. and don't even think that you can trick GC using checks from=20 `core.memory`! this will not work too. sure, you can check if memory used=20 by `a` is still alive, but that memory can be used by completely=20 different object! tl;dr: 1. don't use GC-managed objects in dtors. not even try to access 'em. 2. don't try to trick GC. either don't use it, or cooperate with it.=
Jan 25 2015
parent reply "Bayan Rafeh" <bayan.rafeh92 gmail.com> writes:
On Sunday, 25 January 2015 at 19:15:54 UTC, ketmar wrote:
 On Sun, 25 Jan 2015 08:41:24 +0000, Bayan Rafeh wrote:

 I tried what you said and I think I see the problem. I 
 managed to
 create an example program that duplicates the problem:


 import std.stdio;

 class A {
     string path;

     this(string p) {
         path = p;
     }
     ~this() {
     }
     void a(){}

     void b(){}
 }

 class B {
     A a;
     this() {
         this.a = new A("laladiv");
     }
     ~this() {
         delete a;
     }
 }

 void main() {
     B a = new B(); B b = new B(); delete b;
 }
The solution was just to remove the "delete a" from the destructor if someone comes across this later. Could someone tell me why though?
there is no guarantees on destruction order. and GC will not nullify any references to collected data. so, `a` can be already collected and finalized when `B.~this()` is called. yet reference is still there, so `delete a;` will try to delete already dead object. this will lead to crash. without precise GC collector is not able to automatically nullify all "dead" references. and even if there will be such possibility, it can slow down collections alot (GC will nullifying alot of references that aren't used anyway), so i don't think that it do nullifying. there is a simple rule: "dtor should not touch GC-managed resources". this will not give you "predictable destruction order" (that's why most people try to manually delete something in dtor), and this simply will not work at all. if you want predictable destruction order, don't use GC at all, use manual memory management. it doesn't matter which hack you will invent to force destruction order, any hack will either be very fragile, or will not work. this is due to nature of GC-manged memory. so: don't use GC-managed resources in dtors. don't use in any way -- this including accessing 'em. i.e. reading `a.path` in dtor is invalid too. it will not necessarily crash, but it's the source of "use after free" error. and don't even think that you can trick GC using checks from `core.memory`! this will not work too. sure, you can check if memory used by `a` is still alive, but that memory can be used by completely different object! tl;dr: 1. don't use GC-managed objects in dtors. not even try to access 'em. 2. don't try to trick GC. either don't use it, or cooperate with it.
All right, I removed all my destructors(turns out I don't really need them), but I'm still running into this very same error. This is another problematic example program: import std.stdio; void main(){ auto a = new A("/tmp/invalid"); } class A { File f; string path; this(string path) { this.path = path; // f = File(path, "r"); } invariant() { File test = File(path, "r"); } } is invariant() called during the destruction phase?
Jan 25 2015
next sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 01/25/2015 02:06 PM, Bayan Rafeh wrote:

 is invariant() called during the destruction phase?
Something is fishy. import std.stdio; void main(){ writeln("entered main"); auto a = new A(); writeln("leaving main"); } class A { File file; this() { writeln("this"); } ~this() { writeln("~this"); } invariant() { writeln("invariant"); } } The program produces the following output: entered main this invariant leaving main invariant invariant <-- ? ~this invariant <-- ? Removing the File member changes the output. Now two of the invariant calls are missing: entered main this invariant leaving main invariant ~this However, the last invariant is still there and I think by design: The destructor should see a valid state. Ali
Jan 25 2015
parent "Bayan Rafeh" <bayan.rafeh92 gmail.com> writes:
On Sunday, 25 January 2015 at 22:46:56 UTC, Ali Çehreli wrote:
 On 01/25/2015 02:06 PM, Bayan Rafeh wrote:

 is invariant() called during the destruction phase?
Something is fishy. import std.stdio; void main(){ writeln("entered main"); auto a = new A(); writeln("leaving main"); } class A { File file; this() { writeln("this"); } ~this() { writeln("~this"); } invariant() { writeln("invariant"); } } The program produces the following output: entered main this invariant leaving main invariant invariant <-- ? ~this invariant <-- ? Removing the File member changes the output. Now two of the invariant calls are missing: entered main this invariant leaving main invariant ~this However, the last invariant is still there and I think by design: The destructor should see a valid state. Ali
Could this be a bug in D?
Jan 25 2015
prev sibling next sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Sun, 25 Jan 2015 22:06:26 +0000, Bayan Rafeh wrote:

 This is another problematic example program:
 import std.stdio;
=20
 void main(){
      auto a =3D new A("/tmp/invalid");
 }
=20
 class A {
      File f;
      string path;
=20
      this(string path) {
          this.path =3D path;
 //        f =3D File(path, "r");
      }
=20
      invariant() {
          File test =3D File(path, "r");
      }
 }
=20
 is invariant() called during the destruction phase?
the thing is that your invariant is not a correct invariant at all.=20 invariants are meant to check *internal* object consistency, not external=20 conditions. compiler is free to call invariant block at any time after=20 object is properly initialised (i.e. after ctor is complete) and is not=20 executing member method. so it's perfectly legal to call invariant before=20 dtor, as you should not check anything that is not belonging to the=20 object itself in in. in other words: you can't check any contents of any reference-typed=20 variables in your invariant block. `string` is reference-typed (it's a=20 dynamic array managed by GC in your case), so you can't check it contents=20 in invariant. you CAN, however, use `f` methods in your invariant, as `f`=20 is a struct which lives *inside* your object, and not a reference var. but note that it's a bad practice anyway, as some struct can use some GC- managed objects which can't be safely used in invariant block.=
Jan 25 2015
parent reply "Bayan Rafeh" <bayan.rafeh92 gmail.com> writes:
 the thing is that your invariant is not a correct invariant at 
 all.
 invariants are meant to check *internal* object consistency, 
 not external
 conditions. compiler is free to call invariant block at any 
 time after
 object is properly initialised (i.e. after ctor is complete) 
 and is not
 executing member method. so it's perfectly legal to call 
 invariant before
 dtor, as you should not check anything that is not belonging to 
 the
 object itself in in.
 in other words: you can't check any contents of any 
 reference-typed
 variables in your invariant block. `string` is reference-typed 
 (it's a
 dynamic array managed by GC in your case), so you can't check 
 it contents
 in invariant. you CAN, however, use `f` methods in your 
 invariant, as `f`
 is a struct which lives *inside* your object, and not a 
 reference var.

 but note that it's a bad practice anyway, as some struct can 
 use some GC-
 managed objects which can't be safely used in invariant block.
There are 2 problems here: 1. By definition, after the destructor is called the object state is destroyed. It makes no sense to check the invariant after the destructor is called because there is no state for us to check. 2. Invariants theoretically describe the legal states the object can be in which includes variables that are GC managed, therefore I should be able to check the invariant on GC managed objects as well. What's the point of having this feature if I can't even check the invariants of a simple data structure like a set, let alone classes involving files?
Jan 25 2015
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Mon, 26 Jan 2015 05:25:48 +0000, Bayan Rafeh wrote:

 There are 2 problems here:
=20
 1. By definition, after the destructor is called the object state is
 destroyed. It makes no sense to check the invariant after the destructor
 is called because there is no state for us to check.
i agree that calling invariant after destruction is invalid. i believe=20 that this is a bug.
 2. Invariants theoretically describe the legal states the object can be
 in which includes variables that are GC managed, therefore I should be
 able to check the invariant on GC managed objects as well. What's the
 point of having this feature if I can't even check the invariants of a
 simple data structure like a set, let alone classes involving files?
checking something that is not owned by an object is invalid. invariants=20 MUST NOT be dependent on external data or system state. invariants that=20 tries to check for GC objects ARE dependent of external state. let alone=20 invariants that checking for files, which can be removed while the object=20 is still alive. object can own another object, but it can't own another object *state*. if you want to check something that involves checking something like=20 that, you should use `assert()`s in function body and/or in/out function=20 parts. you are trying to use invariants for the things that invariants can't=20 (and must not) check. invariants are meant for checking *internal*=20 *object* *consistency*. NOT correctness. NOT applicability. ONLY=20 consistency. object can be in "inapplicable" state, but still consistent.=
Jan 25 2015
parent reply "Bayan Rafeh" <bayan.rafeh92 gmail.com> writes:
 you are trying to use invariants for the things that invariants 
 can't
 (and must not) check. invariants are meant for checking 
 *internal*
 *object* *consistency*. NOT correctness. NOT applicability. ONLY
 consistency. object can be in "inapplicable" state, but still 
 consistent.
Then I must have misunderstood the documentation, I apologize, I thought the point of invariant was to ensure correctness of the object's state. I'll go ahead and report the bug.
Jan 26 2015
next sibling parent reply "Bayan Rafeh" <bayan.rafeh92 gmail.com> writes:
Bug report at https://issues.dlang.org/show_bug.cgi?id=14051
Jan 26 2015
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 26 January 2015 at 15:54:12 UTC, Bayan Rafeh wrote:
 I apologize, I thought the point of invariant was to ensure 
 correctness of the
object's state It is simply matter of "internal correctness" vs "in-program correctness". Sorry, documentation should probably mention that explicitly. I have created a documentation PR : https://github.com/D-Programming-Language/dlang.org/pull/851
Jan 26 2015
prev sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Mon, 26 Jan 2015 11:37:12 +0000, Bayan Rafeh wrote:

 you are trying to use invariants for the things that invariants can't
 (and must not) check. invariants are meant for checking *internal*
 *object* *consistency*. NOT correctness. NOT applicability. ONLY
 consistency. object can be in "inapplicable" state, but still
 consistent.
=20 Then I must have misunderstood the documentation, I apologize, I thought the point of invariant was to ensure correctness of the object's state. I'll go ahead and report the bug.
but please, take my words with a grain of salt. i'm in no way a=20 representative of D devs. that is how *i* understand invariants. it seems=20 to be consistent with the cases where invariant works, but the designers=20 may have different opinion, and don't have time to properly fix invariant=20 handling.=
Jan 26 2015
parent reply "Bayan Rafeh" <bayan.rafeh92 gmail.com> writes:
 but please, take my words with a grain of salt. i'm in no way a
 representative of D devs. that is how *i* understand 
 invariants. it seems to be consistent with the cases where 
 invariant works,
This is the first serious project I do with D, so I'm kind of discovering the language as I write my code. I found the contracts page and got overly excited about it, so you're probably right.
 but the designers may have different opinion, and don't have 
 time to
 properly fix invariant handling.
That's fair. Useful as it may be for debugging, it's something I can live without.
Jan 26 2015
parent ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 27 Jan 2015 06:46:20 +0000, Bayan Rafeh wrote:

 This is the first serious project I do with D
and now you're lost to other C-like languages, methinks. ;-)=
Jan 27 2015
prev sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Sun, 25 Jan 2015 22:06:26 +0000, Bayan Rafeh wrote:

p.s. yet creating new `File` in invariant is wrong nevertheless, as it=20
changes the program state. invariant checks SHOULD NEVER CHANGE THE=20
PROGRAM STATE.=
Jan 25 2015