www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - What to do with InvalidMemoryOperationError

reply =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
My executable throws as

     core.exception.InvalidMemoryOperationError (0)

when compiled with DMD git master.

I get no stack trace in GDB.

What to do?
Jan 21 2015
next sibling parent reply =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
 My executable throws as

     core.exception.InvalidMemoryOperationError (0)
I've tracked it down to being caused by foreach (line; File(path).byLine) {} when path contains a very large text file (392 MB, 1658080 lines). Do I have any alternatives to this that doesn't trigger exception?
Jan 21 2015
next sibling parent reply =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
On Wednesday, 21 January 2015 at 12:10:20 UTC, Nordlöw wrote:
 On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
 My executable throws as

    core.exception.InvalidMemoryOperationError (0)
I've tracked it down to being caused by foreach (line; File(path).byLine) {} when path contains a very large text file (392 MB, 1658080 lines). Do I have any alternatives to this that doesn't trigger exception?
I get the exact same behaviour when I try using MmFile instead: import std.mmfile: MmFile; auto mmf = new MmFile(path, MmFile.Mode.read, 0, null, pageSize); const data = cast(char[])mmf[]; foreach (line; data.splitter('\n') {} Does byLine use splitter? If so, maybe splitter is the problem...
Jan 21 2015
parent reply "anonymous" <anonymous example.com> writes:
On Wednesday, 21 January 2015 at 13:07:11 UTC, Nordlöw wrote:
 On Wednesday, 21 January 2015 at 12:10:20 UTC, Nordlöw wrote:
 On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
 My executable throws as

   core.exception.InvalidMemoryOperationError (0)
I've tracked it down to being caused by foreach (line; File(path).byLine) {} when path contains a very large text file (392 MB, 1658080 lines). Do I have any alternatives to this that doesn't trigger exception?
I get the exact same behaviour when I try using MmFile instead: import std.mmfile: MmFile; auto mmf = new MmFile(path, MmFile.Mode.read, 0, null, pageSize); const data = cast(char[])mmf[]; foreach (line; data.splitter('\n') {} Does byLine use splitter? If so, maybe splitter is the problem...
I tested it with a generated large file, and the (rough) file size and line count are apparently not enough to trigger the bug. Could you share the text file? Maybe it can be compressed to a manageable size? Or maybe you can reduce it to a manageable size? A 'binary reduction' might work: Take the first half of the file, and test with that. Error? Proceed with that half. No error? Try the other half. When both halves produce no error, give up. Or maybe dustmite can help here? If the file contains sensitive information, and you cannot reduce it to a reasonable size, you may be able to programmatically replace classes of characters with one character. E.g. replace all alphanumeric characters with 'x'. Be cautious of replacing multibyte characters with single bytes. And newlines should probably be left intact.
Jan 21 2015
parent =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
On Wednesday, 21 January 2015 at 20:50:30 UTC, anonymous wrote:
 Or maybe dustmite can help here?

 If the file contains sensitive information, and you cannot 
 reduce it to a reasonable size, you may be able to 
 programmatically replace classes of characters with one 
 character. E.g. replace all alphanumeric characters with 'x'. 
 Be cautious of replacing multibyte characters with single 
 bytes. And newlines should probably be left intact.
Breaking out that single code calling File.byLine on the same file didn't trigger the error, unfortunately... I guess dustmite is my only friend here. I'm guessing some other code prior to the call together with the call is causing the bug. I'll do some more testing and see if I can reduce my code...
Jan 22 2015
prev sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 01/21/2015 04:10 AM, "Nordlöw" wrote:
 On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
 My executable throws as

     core.exception.InvalidMemoryOperationError (0)
I've tracked it down to being caused by foreach (line; File(path).byLine) {} when path contains a very large text file (392 MB, 1658080 lines). Do I have any alternatives to this that doesn't trigger exception?
Known bug with a pull request: https://issues.dlang.org/show_bug.cgi?id=13856 Here is the duplicate of it, which I've opened recently: https://issues.dlang.org/show_bug.cgi?id=14005 Ali
Jan 21 2015
parent reply "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
On Wednesday, 21 January 2015 at 14:50:06 UTC, Ali Çehreli wrote:
 Known bug with a pull request:

   https://issues.dlang.org/show_bug.cgi?id=13856

 Here is the duplicate of it, which I've opened recently:

   https://issues.dlang.org/show_bug.cgi?id=14005

 Ali
How can this affect Mmfile plus splitter() aswell?
Jan 21 2015
parent "anonymous" <anonymous example.com> writes:
On Wednesday, 21 January 2015 at 15:34:35 UTC, Per Nordlöw wrote:
 On Wednesday, 21 January 2015 at 14:50:06 UTC, Ali Çehreli 
 wrote:
 Known bug with a pull request:

  https://issues.dlang.org/show_bug.cgi?id=13856

 Here is the duplicate of it, which I've opened recently:

  https://issues.dlang.org/show_bug.cgi?id=14005

 Ali
How can this affect Mmfile plus splitter() aswell?
I think it can't. The root of 13856/14005 is in std.stdio.readln. Unless MmFile uses readln somehow, this looks like a different issue to me.
Jan 21 2015
prev sibling next sibling parent reply "Joakim" <dlang joakim.fea.st> writes:
On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
 My executable throws as

     core.exception.InvalidMemoryOperationError (0)

 when compiled with DMD git master.

 I get no stack trace in GDB.

 What to do?
InvalidMemoryOperationError generally means that you are performing certain disallowed memory operations during a full garbage collection, such as allocating while the gc is running. This usually happens when you call a function that allocates in a destructor, which will trigger this error as the destructor is run by the gc. It appears that the gc issues mentioned above can also trigger it. If you're running off git head, maybe you can apply that PR 2794 and see if it helps. Otherwise, maybe you've hit some other gc issue somewhere.
Jan 23 2015
parent reply =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
On Friday, 23 January 2015 at 10:13:43 UTC, Joakim wrote:
 InvalidMemoryOperationError generally means that you are 
 performing certain disallowed memory operations during a full 
 garbage collection, such as allocating while the gc is running.
If my app is single-threaded this cannot happen then.
  This usually happens when you call a function that allocates 
 in a destructor, which will trigger this error as the 
 destructor is run by the gc.

 It appears that the gc issues mentioned above can also trigger 
 it.  If you're running off git head, maybe you can apply that 
 PR 2794 and see if it helps.  Otherwise, maybe you've hit some 
 other gc issue somewhere.
So if GC.disable prevents the error from occurring I will have a clue, right?
Jan 23 2015
parent "anonymous" <anonymous example.com> writes:
On Friday, 23 January 2015 at 21:25:01 UTC, Nordlöw wrote:
 On Friday, 23 January 2015 at 10:13:43 UTC, Joakim wrote:
 InvalidMemoryOperationError generally means that you are 
 performing certain disallowed memory operations during a full 
 garbage collection, such as allocating while the gc is running.
If my app is single-threaded this cannot happen then.
Single-/multi-threaded doesn't make a difference. This happens with a single thread, too. The GC calls destructors. When a destructor allocates then, it allocates during a GC run.
Jan 23 2015
prev sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
 My executable throws as

     core.exception.InvalidMemoryOperationError (0)

 when compiled with DMD git master.

 I get no stack trace in GDB.

 What to do?
Hi, I created a wiki page which I hope will help you solve this problem: http://wiki.dlang.org/InvalidMemoryOperationError Hope this helps.
Jan 24 2015
parent =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
On Sunday, 25 January 2015 at 00:44:07 UTC, Vladimir Panteleev 
wrote:
 I created a wiki page which I hope will help you solve this
 problem:

 http://wiki.dlang.org/InvalidMemoryOperationError

 Hope this helps.
Great! Thanks!
Jan 25 2015