www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - read() performance - Linux.too?

reply "Bob W" <nospam aol.com> writes:
/*
The std.file.read() function in dmd causes a performance
issue after reading large files from 100MB upwards.
Reading the file seems to be no problem, but cleanup
afterwards takes forever.

I am therefore using std.mmfile which works fine in the
Windows version of D, but using read() would be more
convenient in several cases.

Now a few questions:

1) Does anyone know if the read() performance problem
occurs in the Linux version of D as well?

2) Is there any info available where the real problem
sits? Allocating a few 100MB does not show the same
phenomenon and dmc's fread() function is also painless.

3) I did not find anything about this issue in Bugzilla.
Did I overlook the respective entry?

*/


// Try reading a 100MB+ file with the following
// program (some patience required):

import std.stdio, std.file;

alias writefln wrl;

void main(char[][] av) {
  wrl();
  if (av.length<2)  {
    wrl("Need file name to test read() !");
    return;
  }
  char[] fn=av[1];
  wrl("Reading '%s' ...", fn);
  char[] bf=cast(char[])read(fn);
  wrl("%d bytes read.",bf.length);
  wrl("Doing something ...");
  int n=0;
  foreach(c;bf)  n+=c;
  wrl("Result: %s, done.",n);
  wrl("Expect a delay here after reading a huge file ...");
  wrl();
}
Jul 23 2006
next sibling parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
Why do you need to read the entire file into memory at once?

Anyway, this may be a Phobos problem, but most likely it's the garbage 
collector.  What if you manually delete the buffer returned by read()?

-[Unknown]



 /*
 The std.file.read() function in dmd causes a performance
 issue after reading large files from 100MB upwards.
 Reading the file seems to be no problem, but cleanup
 afterwards takes forever.
 
 I am therefore using std.mmfile which works fine in the
 Windows version of D, but using read() would be more
 convenient in several cases.
 
 Now a few questions:
 
 1) Does anyone know if the read() performance problem
 occurs in the Linux version of D as well?
 
 2) Is there any info available where the real problem
 sits? Allocating a few 100MB does not show the same
 phenomenon and dmc's fread() function is also painless.
 
 3) I did not find anything about this issue in Bugzilla.
 Did I overlook the respective entry?
 
 */
 
 
 // Try reading a 100MB+ file with the following
 // program (some patience required):
 
 import std.stdio, std.file;
 
 alias writefln wrl;
 
 void main(char[][] av) {
   wrl();
   if (av.length<2)  {
     wrl("Need file name to test read() !");
     return;
   }
   char[] fn=av[1];
   wrl("Reading '%s' ...", fn);
   char[] bf=cast(char[])read(fn);
   wrl("%d bytes read.",bf.length);
   wrl("Doing something ...");
   int n=0;
   foreach(c;bf)  n+=c;
   wrl("Result: %s, done.",n);
   wrl("Expect a delay here after reading a huge file ...");
   wrl();
 }
Jul 23 2006
parent "Bob W" <nospam aol.com> writes:
"Unknown W. Brackets" <unknown simplemachines.org> wrote in message 
news:ea1dd2$l4p$1 digitaldaemon.com...

 Why do you need to read the entire file into memory at once?


Funny question - I have a minimum of 1GB of main memory in
my computers and I intend to use it in order to get the best
performance possible.

You are probably aware of the fact that "read()" will read
the entire file at once, it also takes care of opening and
closing the file.



 Anyway, this may be a Phobos problem, but most likely it's the garbage 
 collector.  What if you manually delete the buffer returned by read()?


That'll work. But I am pretty reluctant to accept my
obligation to assist the GC before it falls into coma.
Jul 24 2006
prev sibling next sibling parent reply Dave <Dave_member pathlink.com> writes:
Bob W wrote:

 /*
 The std.file.read() function in dmd causes a performance
 issue after reading large files from 100MB upwards.
 Reading the file seems to be no problem, but cleanup
 afterwards takes forever.
 
 I am therefore using std.mmfile which works fine in the
 Windows version of D, but using read() would be more
 convenient in several cases.
 
 Now a few questions:
 
 1) Does anyone know if the read() performance problem
 occurs in the Linux version of D as well?
 
 2) Is there any info available where the real problem
 sits? Allocating a few 100MB does not show the same
 phenomenon and dmc's fread() function is also painless.
 
 3) I did not find anything about this issue in Bugzilla.
 Did I overlook the respective entry?
 
 */
 
 
 // Try reading a 100MB+ file with the following
 // program (some patience required):
 
 import std.stdio, std.file;
 
 alias writefln wrl;
 
 void main(char[][] av) {
   wrl();
   if (av.length<2)  {
     wrl("Need file name to test read() !");
     return;
   }
   char[] fn=av[1];
   wrl("Reading '%s' ...", fn);
   char[] bf=cast(char[])read(fn);
   wrl("%d bytes read.",bf.length);
   wrl("Doing something ...");
   int n=0;
   foreach(c;bf)  n+=c;
   wrl("Result: %s, done.",n);
   wrl("Expect a delay here after reading a huge file ...");
   wrl();
 }
 
 


It's more than likely the GC, the same happens w/ a program like this:

import std.outbuffer;
import std.string : atoi;
import std.stdio  : wrl = writefln;

void main(char[][] args)
{
     int n = args.length > 1 ? atoi(args[1]) : 10_000_000;
     OutBuffer b = new OutBuffer;
     for(int i = 0; i < n; i++)
     {
         b.write("hello\n");
     }
     wrl(b.toString.length);
}

Run w/o an argument (n = 10_000_000), on Windows it takes forever 
(starts swapping), on Linux it takes about a second.
Jul 23 2006
next sibling parent Karen Lanrap <karen digitaldaemon.com> writes:
Dave wrote:


          b.write("hello\n");


Funny. Changing that to


          b.write("helloo");


lets it run fast.
Jul 24 2006
prev sibling parent reply "Bob W" <nospam aol.com> writes:
"Dave" <Dave_member pathlink.com> wrote in message 
news:ea1g68$nhl$1 digitaldaemon.com...

 It's more than likely the GC, the same happens w/ a program like this:

 import std.outbuffer;
 import std.string : atoi;
 import std.stdio  : wrl = writefln;

 void main(char[][] args)
 {
     int n = args.length > 1 ? atoi(args[1]) : 10_000_000;
     OutBuffer b = new OutBuffer;
     for(int i = 0; i < n; i++)
     {
         b.write("hello\n");
     }
     wrl(b.toString.length);
 }

 Run w/o an argument (n = 10_000_000), on Windows it takes forever (starts 
 swapping), on Linux it takes about a second.



Thanks for your info - I'll remember it as a warning.

But this is probably a different case. Your program is
dynamically resizing b's buffer. This requires more
overhead than just releasing a piece of memory which
was allocated in one single step.
Jul 24 2006
parent Dave <Dave_member pathlink.com> writes:
Bob W wrote:

 "Dave" <Dave_member pathlink.com> wrote in message 
 news:ea1g68$nhl$1 digitaldaemon.com...

 It's more than likely the GC, the same happens w/ a program like this:

 import std.outbuffer;
 import std.string : atoi;
 import std.stdio  : wrl = writefln;

 void main(char[][] args)
 {
     int n = args.length > 1 ? atoi(args[1]) : 10_000_000;
     OutBuffer b = new OutBuffer;
     for(int i = 0; i < n; i++)
     {
         b.write("hello\n");
     }
     wrl(b.toString.length);
 }

 Run w/o an argument (n = 10_000_000), on Windows it takes forever (starts 
 swapping), on Linux it takes about a second.


 
 
 Thanks for your info - I'll remember it as a warning.
 
 But this is probably a different case. Your program is
 dynamically resizing b's buffer. This requires more
 overhead than just releasing a piece of memory which
 was allocated in one single step.
 


It seems to be "thrashing" during the full collection at program exit, 
but I haven't looked into it fully. I think that is probably what is 
happening in your case as well (that's why I mentioned it, but I should 
have explained that better).
Jul 24 2006
prev sibling parent reply Derek Parnell <derek nomail.afraid.org> writes:
On Mon, 24 Jul 2006 04:55:17 +0200, Bob W wrote:


 /*
 The std.file.read() function in dmd causes a performance
 issue after reading large files from 100MB upwards.
 Reading the file seems to be no problem, but cleanup
 afterwards takes forever.


Its a GC effect. The GC is scanning through the buffer looking for
addresses to clean up.

A simple delete of the buffer will prevent the GC from trying so hard.

 // Try reading a 100MB+ file with the following
 // program (some patience required):
 
 import std.stdio, std.file;
 
 alias writefln wrl;
 
 void main(char[][] av) {
   wrl();
   if (av.length<2)  {
     wrl("Need file name to test read() !");
     return;
   }
   char[] fn=av[1];
   wrl("Reading '%s' ...", fn);
   char[] bf=cast(char[])read(fn);
   wrl("%d bytes read.",bf.length);
   wrl("Doing something ...");
   int n=0;
   foreach(c;bf)  n+=c;
   wrl("Result: %s, done.",n);

   delete bf;

   wrl("No delay here now after reading a huge file ...");
   wrl();
 }

-- 
Derek
(skype: derek.j.parnell)
Melbourne, Australia
"Down with mediocrity!"
24/07/2006 2:11:34 PM
Jul 23 2006
next sibling parent reply "Lionello Lunesu" <lionello lunesu.remove.com> writes:
"Derek Parnell" <derek nomail.afraid.org> wrote in message 
news:dg25ykpt8kxw$.1r2mhu0u851l0.dlg 40tude.net...

 On Mon, 24 Jul 2006 04:55:17 +0200, Bob W wrote:


 /*
 The std.file.read() function in dmd causes a performance
 issue after reading large files from 100MB upwards.
 Reading the file seems to be no problem, but cleanup
 afterwards takes forever.


 Its a GC effect. The GC is scanning through the buffer looking for
 addresses to clean up.


Wouldn't it be possible to add some way of telling the GC not to scan 
something? Perhaps there's already something in std.gc, I didn't check, but 
I actually think the compiler could be doing this by checking the TypeInfo. 
I wouldn't go so far as to expect it to only scan the pointer fields of a 
struct, but at least it could ignore char[] and float[] (and other arrays 
containing non-pointer types).

I've made that Universal Machine of the programming contest (see thread 
below) and am running into memory problems. I have the feeling that a lot of 
the opcodes in the machine code are considered as pointers. Memory just 
keeps growing and the GC cycles take longer and longer.

It was great to write the UM without having to worry about memory, but now 
I'll have to worry about it and in a totally new way: trying to outsmart the 
GC. Either that, or malloc/memset/free : (

L.
Jul 24 2006
parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
Actually, I believe it's just:

import std.gc;

// ...

ubyte[] data = new ubyte[1024 * 1024];
std.gc.removeRange(data);

This tells it, afaik, not to scan the described range for pointers.  It 
seems to me entirely possible that the compiler could automatically 
generate this code for new ubyte[] and such calls.

-[Unknown]



 "Derek Parnell" <derek nomail.afraid.org> wrote in message 
 news:dg25ykpt8kxw$.1r2mhu0u851l0.dlg 40tude.net...

 On Mon, 24 Jul 2006 04:55:17 +0200, Bob W wrote:


 /*
 The std.file.read() function in dmd causes a performance
 issue after reading large files from 100MB upwards.
 Reading the file seems to be no problem, but cleanup
 afterwards takes forever.


 Its a GC effect. The GC is scanning through the buffer looking for
 addresses to clean up.


 
 Wouldn't it be possible to add some way of telling the GC not to scan 
 something? Perhaps there's already something in std.gc, I didn't check, but 
 I actually think the compiler could be doing this by checking the TypeInfo. 
 I wouldn't go so far as to expect it to only scan the pointer fields of a 
 struct, but at least it could ignore char[] and float[] (and other arrays 
 containing non-pointer types).
 
 I've made that Universal Machine of the programming contest (see thread 
 below) and am running into memory problems. I have the feeling that a lot of 
 the opcodes in the machine code are considered as pointers. Memory just 
 keeps growing and the GC cycles take longer and longer.
 
 It was great to write the UM without having to worry about memory, but now 
 I'll have to worry about it and in a totally new way: trying to outsmart the 
 GC. Either that, or malloc/memset/free : (
 
 L.
Jul 24 2006
parent reply Derek Parnell <derek nomail.afraid.org> writes:
On Mon, 24 Jul 2006 22:02:31 -0700, Unknown W. Brackets wrote:


 Actually, I believe it's just:
 
 import std.gc;
 
 // ...
 
 ubyte[] data = new ubyte[1024 * 1024];
 std.gc.removeRange(data);
 
 This tells it, afaik, not to scan the described range for pointers.  It 
 seems to me entirely possible that the compiler could automatically 
 generate this code for new ubyte[] and such calls.


Yes, but wouldn't that RAM be deallocated only at program end? If you
wanted it deallocated earlier you would still have to delete it.

-- 
Derek
(skype: derek.j.parnell)
Melbourne, Australia
"Down with mediocrity!"
25/07/2006 3:09:46 PM
Jul 24 2006
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Tue, 25 Jul 2006 15:15:24 +1000, Derek Parnell  
<derek nomail.afraid.org> wrote:

 On Mon, 24 Jul 2006 22:02:31 -0700, Unknown W. Brackets wrote:


 Actually, I believe it's just:

 import std.gc;

 // ...

 ubyte[] data = new ubyte[1024 * 1024];
 std.gc.removeRange(data);

 This tells it, afaik, not to scan the described range for pointers.  It
 seems to me entirely possible that the compiler could automatically
 generate this code for new ubyte[] and such calls.


 Yes, but wouldn't that RAM be deallocated only at program end? If you
 wanted it deallocated earlier you would still have to delete it.


The range pointed at by the array 'data' shouldn't be scanned, but there  
is no reason the array reference itself cannot be scanned and therefore  
collected, right? And if the array reference is collected, the data will  
be freed, just not scanned for other pointers, right?

Regan
Jul 24 2006
next sibling parent "Unknown W. Brackets" <unknown simplemachines.org> writes:
Yes, that's what I meant.  You'd remove the range of memory from 
scanning, but keep the root.

Please correct me if I'm wrong.

Thanks,
-[Unknown]



 On Tue, 25 Jul 2006 15:15:24 +1000, Derek Parnell 
 <derek nomail.afraid.org> wrote:

 On Mon, 24 Jul 2006 22:02:31 -0700, Unknown W. Brackets wrote:


 Actually, I believe it's just:

 import std.gc;

 // ...

 ubyte[] data = new ubyte[1024 * 1024];
 std.gc.removeRange(data);

 This tells it, afaik, not to scan the described range for pointers.  It
 seems to me entirely possible that the compiler could automatically
 generate this code for new ubyte[] and such calls.


 Yes, but wouldn't that RAM be deallocated only at program end? If you
 wanted it deallocated earlier you would still have to delete it.


 
 The range pointed at by the array 'data' shouldn't be scanned, but there 
 is no reason the array reference itself cannot be scanned and therefore 
 collected, right? And if the array reference is collected, the data will 
 be freed, just not scanned for other pointers, right?
 
 Regan
Jul 24 2006
prev sibling parent reply Derek Parnell <derek nomail.afraid.org> writes:
On Tue, 25 Jul 2006 17:18:24 +1200, Regan Heath wrote:


 On Tue, 25 Jul 2006 15:15:24 +1000, Derek Parnell  
 <derek nomail.afraid.org> wrote:

 On Mon, 24 Jul 2006 22:02:31 -0700, Unknown W. Brackets wrote:


 Actually, I believe it's just:

 import std.gc;

 // ...

 ubyte[] data = new ubyte[1024 * 1024];
 std.gc.removeRange(data);

 This tells it, afaik, not to scan the described range for pointers.  It
 seems to me entirely possible that the compiler could automatically
 generate this code for new ubyte[] and such calls.


 Yes, but wouldn't that RAM be deallocated only at program end? If you
 wanted it deallocated earlier you would still have to delete it.


 
 The range pointed at by the array 'data' shouldn't be scanned, but there  
 is no reason the array reference itself cannot be scanned and therefore  
 collected, right? And if the array reference is collected, the data will  
 be freed, just not scanned for other pointers, right?


Yes that sort of makes sense. So does the parameter stack get scanned after
a function returns but before the calling function takes control again,
because that's where the array reference resides usually. Or is it that
when a 'new' is done, the returned address is added to a list that the GC
uses to free up RAM from? 

-- 
Derek
(skype: derek.j.parnell)
Melbourne, Australia
"Down with mediocrity!"
25/07/2006 3:29:21 PM
Jul 24 2006
parent "Unknown W. Brackets" <unknown simplemachines.org> writes:
Derek,

Your question doesn't make complete sense to me, so I'm going to back up 
a bit.  Please forgive me if I patronize you, or fail to answer your 
question.

The garbage collector has "ranges" of memory it scans (as I'm completely 
sure you already know.)  For example, you could add an arbitrary range. 
  Consider:

void* p = malloc(100);
std.gc.addRange(p, p + cast(ptrdiff_t) 100);

This will cause it to scan the space between (p) and (p + 100) or 
pointers (roots.)  Removing a range does not mean, as far as I can see, 
that the memory it points to will never be freed; just that it will not 
be scanned.

An addRange() happens automatically when you new with the garbage collector.

-[Unknown]



 On Tue, 25 Jul 2006 17:18:24 +1200, Regan Heath wrote:
 

 On Tue, 25 Jul 2006 15:15:24 +1000, Derek Parnell  
 <derek nomail.afraid.org> wrote:

 On Mon, 24 Jul 2006 22:02:31 -0700, Unknown W. Brackets wrote:


 Actually, I believe it's just:

 import std.gc;

 // ...

 ubyte[] data = new ubyte[1024 * 1024];
 std.gc.removeRange(data);

 This tells it, afaik, not to scan the described range for pointers.  It
 seems to me entirely possible that the compiler could automatically
 generate this code for new ubyte[] and such calls.


 Yes, but wouldn't that RAM be deallocated only at program end? If you
 wanted it deallocated earlier you would still have to delete it.


 The range pointed at by the array 'data' shouldn't be scanned, but there  
 is no reason the array reference itself cannot be scanned and therefore  
 collected, right? And if the array reference is collected, the data will  
 be freed, just not scanned for other pointers, right?


 
 Yes that sort of makes sense. So does the parameter stack get scanned after
 a function returns but before the calling function takes control again,
 because that's where the array reference resides usually. Or is it that
 when a 'new' is done, the returned address is added to a list that the GC
 uses to free up RAM from?
Jul 24 2006
prev sibling next sibling parent "Bob W" <nospam aol.com> writes:
"Derek Parnell" <derek nomail.afraid.org> wrote in message 
news:dg25ykpt8kxw$.1r2mhu0u851l0.dlg 40tude.net...

 Its a GC effect. The GC is scanning through the buffer looking for
 addresses to clean up.


Sounds like the GC isn't overly smart. It would be nice to
have this fixed before the dmd 0.163 to dmd 1.0 transition.



 A simple delete of the buffer will prevent the GC from trying so hard.


Yes, I know. But as already mentioned in another post,
I am pretty reluctant to accept my responsibility to assist
the GC in performing an elegant exit.
Jul 24 2006
prev sibling parent reply Lionello Lunesu <lio lunesu.remove.com> writes:
Derek Parnell wrote:

 On Mon, 24 Jul 2006 04:55:17 +0200, Bob W wrote:
 

 /*
 The std.file.read() function in dmd causes a performance
 issue after reading large files from 100MB upwards.
 Reading the file seems to be no problem, but cleanup
 afterwards takes forever.


 
 Its a GC effect. The GC is scanning through the buffer looking for
 addresses to clean up.


I just read the response to my post and it seems the read() function 
should do the std.gc.removeRange on the memory containing the read file, 
no? There's no way that memory could contain pointers.

Of course, somebody could change the memory afterwards, replacing 
internal file references with memory pointers and it'll get f**** up.

L.
Jul 24 2006
parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
Yet, someone could also do this:

ubyte[] buffer = std.file.read(filename);
*(buffer.ptr + 9000) = 0;

That would probably be a Bad Thing as well, doesn't mean Phobos should 
worry about it...

I completely agree that Phobos' read() should have a removeRange() call 
there, unless it is decided to add such a thing to the standard library.

-[Unknown]



 Derek Parnell wrote:

 On Mon, 24 Jul 2006 04:55:17 +0200, Bob W wrote:


 /*
 The std.file.read() function in dmd causes a performance
 issue after reading large files from 100MB upwards.
 Reading the file seems to be no problem, but cleanup
 afterwards takes forever.


 Its a GC effect. The GC is scanning through the buffer looking for
 addresses to clean up.


 
 I just read the response to my post and it seems the read() function 
 should do the std.gc.removeRange on the memory containing the read file, 
 no? There's no way that memory could contain pointers.
 
 Of course, somebody could change the memory afterwards, replacing 
 internal file references with memory pointers and it'll get f**** up.
 
 L.
Jul 24 2006
parent reply Derek Parnell <derek nomail.afraid.org> writes:
On Mon, 24 Jul 2006 23:04:09 -0700, Unknown W. Brackets wrote:


 Derek,
 
 Your question doesn't make complete sense to me, so I'm going to back up 
 a bit.  Please forgive me if I patronize you, or fail to answer your 
 question.


Not a problem.
 

 The garbage collector has "ranges" of memory it scans (as I'm completely 
 sure you already know.)  For example, you could add an arbitrary range. 
   Consider:
 
 void* p = malloc(100);
 std.gc.addRange(p, p + cast(ptrdiff_t) 100);
 
 This will cause it to scan the space between (p) and (p + 100) or 
 pointers (roots.)  Removing a range does not mean, as far as I can see, 
 that the memory it points to will never be freed; just that it will not 
 be scanned.


So long as the root itself is stored somewhere that the GC can find it. I
guess this is done via the addRoot() call and I assume that 'new'
automatically calls this.


 I completely agree that Phobos' read() should have a removeRange() call 
 there, unless it is decided to add such a thing to the standard library.


I just tried something like that out. In file.d I changed 

    buf = new byte[size];

to ...

    void *p = malloc(size);
    if (p == null)
        throw new OutOfMemoryException;
    buf = cast(byte[])p[0..size];


I assume that the " = p[x..y]" construct adds the root to the GC.

-- 
Derek
(skype: derek.j.parnell)
Melbourne, Australia
"Down with mediocrity!"
25/07/2006 4:17:40 PM
Jul 24 2006
next sibling parent Lionello Lunesu <lio lunesu.remove.com> writes:
Derek Parnell wrote:

.....
     void *p = malloc(size);
     if (p == null)
         throw new OutOfMemoryException;
     buf = cast(byte[])p[0..size];
 
 I assume that the " = p[x..y]" construct adds the root to the GC.


I doubt that.

L.
Jul 24 2006
prev sibling parent "Unknown W. Brackets" <unknown simplemachines.org> writes:
I don't think it does.  I think it has to be in one of the GC's pools 
and buckets.  I don't think you can add malloc'd memory to the GC.

I may be wrong.

Why not...

buf = new byte[size];
std.gc.removeRange(buf);

Yes, it's a bit silly removing the range immediately after adding it, 
but the point is to get it not to try to scan the memory, right?

-[Unknown]



 On Mon, 24 Jul 2006 23:04:09 -0700, Unknown W. Brackets wrote:
 

 Derek,

 Your question doesn't make complete sense to me, so I'm going to back up 
 a bit.  Please forgive me if I patronize you, or fail to answer your 
 question.


 
 Not a problem.
  

 The garbage collector has "ranges" of memory it scans (as I'm completely 
 sure you already know.)  For example, you could add an arbitrary range. 
   Consider:

 void* p = malloc(100);
 std.gc.addRange(p, p + cast(ptrdiff_t) 100);

 This will cause it to scan the space between (p) and (p + 100) or 
 pointers (roots.)  Removing a range does not mean, as far as I can see, 
 that the memory it points to will never be freed; just that it will not 
 be scanned.


 
 So long as the root itself is stored somewhere that the GC can find it. I
 guess this is done via the addRoot() call and I assume that 'new'
 automatically calls this.
 

 I completely agree that Phobos' read() should have a removeRange() call 
 there, unless it is decided to add such a thing to the standard library.


 
 I just tried something like that out. In file.d I changed 
 
     buf = new byte[size];
 
 to ...
 
     void *p = malloc(size);
     if (p == null)
         throw new OutOfMemoryException;
     buf = cast(byte[])p[0..size];
 
 
 I assume that the " = p[x..y]" construct adds the root to the GC.
Jul 25 2006