• =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= (5/5) Jan 21 2015 My executable throws as
• =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= (5/7) Jan 21 2015 I've tracked it down to being caused by
• =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= (8/18) Jan 21 2015 I get the exact same behaviour when I try using MmFile instead:
• anonymous (15/38) Jan 21 2015 I tested it with a generated large file, and the (rough) file
• =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= (7/14) Jan 22 2015 Breaking out that single code calling File.byLine on the same
• =?UTF-8?B?QWxpIMOHZWhyZWxp?= (6/14) Jan 21 2015 Known bug with a pull request:
• "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> (2/7) Jan 21 2015 How can this affect Mmfile plus splitter() aswell?
• anonymous (4/16) Jan 21 2015 I think it can't. The root of 13856/14005 is in std.stdio.readln.
• Joakim (11/16) Jan 23 2015 InvalidMemoryOperationError generally means that you are
• =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= (4/14) Jan 23 2015 So if GC.disable prevents the error from occurring I will have a
• anonymous (4/9) Jan 23 2015 Single-/multi-threaded doesn't make a difference. This happens
• Vladimir Panteleev (6/11) Jan 24 2015 Hi,
• =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= (3/7) Jan 25 2015 Great! Thanks!

=?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:

My executable throws as

     core.exception.InvalidMemoryOperationError (0)

when compiled with DMD git master.

I get no stack trace in GDB.

What to do?

=?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:

On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
 My executable throws as

     core.exception.InvalidMemoryOperationError (0)

I've tracked it down to being caused by

     foreach (line; File(path).byLine) {}

when path contains a very large text file (392 MB, 1658080 lines).

Do I have any alternatives to this that doesn't trigger exception?

=?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:

On Wednesday, 21 January 2015 at 12:10:20 UTC, Nordlöw wrote:
 On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
 My executable throws as

    core.exception.InvalidMemoryOperationError (0)

 I've tracked it down to being caused by

     foreach (line; File(path).byLine) {}

 when path contains a very large text file (392 MB, 1658080 
 lines).

 Do I have any alternatives to this that doesn't trigger 
 exception?

I get the exact same behaviour when I try using MmFile instead:

     import std.mmfile: MmFile;
     auto mmf = new MmFile(path, MmFile.Mode.read, 0, null, 
pageSize);
     const data = cast(char[])mmf[];
     foreach (line; data.splitter('\n') {}

Does byLine use splitter? If so, maybe splitter is the problem...

"anonymous" <anonymous example.com> writes:

On Wednesday, 21 January 2015 at 13:07:11 UTC, Nordlöw wrote:
 On Wednesday, 21 January 2015 at 12:10:20 UTC, Nordlöw wrote:
 On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
 My executable throws as

   core.exception.InvalidMemoryOperationError (0)

 I've tracked it down to being caused by

    foreach (line; File(path).byLine) {}

 when path contains a very large text file (392 MB, 1658080 
 lines).

 Do I have any alternatives to this that doesn't trigger 
 exception?

 I get the exact same behaviour when I try using MmFile instead:

     import std.mmfile: MmFile;
     auto mmf = new MmFile(path, MmFile.Mode.read, 0, null, 
 pageSize);
     const data = cast(char[])mmf[];
     foreach (line; data.splitter('\n') {}

 Does byLine use splitter? If so, maybe splitter is the 
 problem...

I tested it with a generated large file, and the (rough) file 
size and line count are apparently not enough to trigger the bug.

Could you share the text file? Maybe it can be compressed to a 
manageable size? Or maybe you can reduce it to a manageable size?

A 'binary reduction' might work: Take the first half of the file, 
and test with that. Error? Proceed with that half. No error? Try 
the other half. When both halves produce no error, give up.

Or maybe dustmite can help here?

If the file contains sensitive information, and you cannot reduce 
it to a reasonable size, you may be able to programmatically 
replace classes of characters with one character. E.g. replace 
all alphanumeric characters with 'x'. Be cautious of replacing 
multibyte characters with single bytes. And newlines should 
probably be left intact.

=?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:

On Wednesday, 21 January 2015 at 20:50:30 UTC, anonymous wrote:
 Or maybe dustmite can help here?

 If the file contains sensitive information, and you cannot 
 reduce it to a reasonable size, you may be able to 
 programmatically replace classes of characters with one 
 character. E.g. replace all alphanumeric characters with 'x'. 
 Be cautious of replacing multibyte characters with single 
 bytes. And newlines should probably be left intact.

Breaking out that single code calling File.byLine on the same 
file didn't trigger the error, unfortunately...

I guess dustmite is my only friend here.

I'm guessing some other code prior to the call together with the 
call is causing the bug.

I'll do some more testing and see if I can reduce my code...

=?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:

On 01/21/2015 04:10 AM, "Nordlöw" wrote:
 On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
 My executable throws as

     core.exception.InvalidMemoryOperationError (0)

 I've tracked it down to being caused by

      foreach (line; File(path).byLine) {}

 when path contains a very large text file (392 MB, 1658080 lines).

 Do I have any alternatives to this that doesn't trigger exception?

Known bug with a pull request:

   https://issues.dlang.org/show_bug.cgi?id=13856

Here is the duplicate of it, which I've opened recently:

   https://issues.dlang.org/show_bug.cgi?id=14005

Ali

"Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:

On Wednesday, 21 January 2015 at 14:50:06 UTC, Ali Çehreli wrote:
 Known bug with a pull request:

   https://issues.dlang.org/show_bug.cgi?id=13856

 Here is the duplicate of it, which I've opened recently:

   https://issues.dlang.org/show_bug.cgi?id=14005

 Ali

How can this affect Mmfile plus splitter() aswell?

"anonymous" <anonymous example.com> writes:

On Wednesday, 21 January 2015 at 15:34:35 UTC, Per Nordlöw wrote:
 On Wednesday, 21 January 2015 at 14:50:06 UTC, Ali Çehreli 
 wrote:
 Known bug with a pull request:

  https://issues.dlang.org/show_bug.cgi?id=13856

 Here is the duplicate of it, which I've opened recently:

  https://issues.dlang.org/show_bug.cgi?id=14005

 Ali

 How can this affect Mmfile plus splitter() aswell?

I think it can't. The root of 13856/14005 is in std.stdio.readln. 
Unless MmFile uses readln somehow, this looks like a different 
issue to me.

"Joakim" <dlang joakim.fea.st> writes:

On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
 My executable throws as

     core.exception.InvalidMemoryOperationError (0)

 when compiled with DMD git master.

 I get no stack trace in GDB.

 What to do?

InvalidMemoryOperationError generally means that you are 
performing certain disallowed memory operations during a full 
garbage collection, such as allocating while the gc is running.  
This usually happens when you call a function that allocates in a 
destructor, which will trigger this error as the destructor is 
run by the gc.

It appears that the gc issues mentioned above can also trigger 
it.  If you're running off git head, maybe you can apply that PR 
2794 and see if it helps.  Otherwise, maybe you've hit some other 
gc issue somewhere.

=?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:

On Friday, 23 January 2015 at 10:13:43 UTC, Joakim wrote:
 InvalidMemoryOperationError generally means that you are 
 performing certain disallowed memory operations during a full 
 garbage collection, such as allocating while the gc is running.

If my app is single-threaded this cannot happen then.

  This usually happens when you call a function that allocates 
 in a destructor, which will trigger this error as the 
 destructor is run by the gc.

 It appears that the gc issues mentioned above can also trigger 
 it.  If you're running off git head, maybe you can apply that 
 PR 2794 and see if it helps.  Otherwise, maybe you've hit some 
 other gc issue somewhere.

So if GC.disable prevents the error from occurring I will have a 
clue, right?

"anonymous" <anonymous example.com> writes:

On Friday, 23 January 2015 at 21:25:01 UTC, Nordlöw wrote:
 On Friday, 23 January 2015 at 10:13:43 UTC, Joakim wrote:
 InvalidMemoryOperationError generally means that you are 
 performing certain disallowed memory operations during a full 
 garbage collection, such as allocating while the gc is running.

 If my app is single-threaded this cannot happen then.

Single-/multi-threaded doesn't make a difference. This happens 
with a single thread, too. The GC calls destructors. When a 
destructor allocates then, it allocates during a GC run.

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
 My executable throws as

     core.exception.InvalidMemoryOperationError (0)

 when compiled with DMD git master.

 I get no stack trace in GDB.

 What to do?

Hi,

I created a wiki page which I hope will help you solve this
problem:

http://wiki.dlang.org/InvalidMemoryOperationError

Hope this helps.

=?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:

On Sunday, 25 January 2015 at 00:44:07 UTC, Vladimir Panteleev 
wrote:
 I created a wiki page which I hope will help you solve this
 problem:

 http://wiki.dlang.org/InvalidMemoryOperationError

 Hope this helps.

Great! Thanks!