www.digitalmars.com         C & C++   DMDScript  

D - Read a file

reply manfred toppoint.de writes:
Hello,

i have write a test programm that takes 65 seconds.
File tralala has 213753 lines.   

import std.stream;
int main() {
File fd = new File("tralala");
char[][] werte;
while (!fd.eof()) {
fd.readLine();
}
return 0;
}

Give it a better way to read a File line by line? 
The same perl programm takes 11 seconds.
Jan 29 2004
next sibling parent "Walter" <walter digitalmars.com> writes:
<manfred toppoint.de> wrote in message
news:bvb54a$1qk5$1 digitaldaemon.com...
 Hello,

 i have write a test programm that takes 65 seconds.
 File tralala has 213753 lines.

 import std.stream;
 int main() {
 File fd = new File("tralala");
 char[][] werte;
 while (!fd.eof()) {
 fd.readLine();
 }
 return 0;
 }

 Give it a better way to read a File line by line?
 The same perl programm takes 11 seconds.
Try std.file.read().
Jan 29 2004
prev sibling parent reply "C" <dont respond.com> writes:
Hmm thats interesting.

std.file.read took about 1 second on a 250K line file , the while loop took
about 12 seconds for me , which is a very long time.

is it the readLine call thats making this so slow ?

C

<manfred toppoint.de> wrote in message
news:bvb54a$1qk5$1 digitaldaemon.com...
 Hello,

 i have write a test programm that takes 65 seconds.
 File tralala has 213753 lines.

 import std.stream;
 int main() {
 File fd = new File("tralala");
 char[][] werte;
 while (!fd.eof()) {
 fd.readLine();
 }
 return 0;
 }

 Give it a better way to read a File line by line?
 The same perl programm takes 11 seconds.
Jan 29 2004
next sibling parent reply "C" <dont respond.com> writes:
This C++ takes 3 seconds to read line by line.

#include <fstream>
#include <string>

using namespace std;

int main () {

 ifstream in("tralala");
 string line;
 while ( !in.eof ()  ) {

  std::getline(in,line);

 }
 return 1;
}

Ill try to use Mattys libs to find the bottleneck , as reading line by line
is a common operation.

C

"C" <dont respond.com> wrote in message
news:bvbolu$2qpf$1 digitaldaemon.com...
 Hmm thats interesting.

 std.file.read took about 1 second on a 250K line file , the while loop
took
 about 12 seconds for me , which is a very long time.

 is it the readLine call thats making this so slow ?

 C

 <manfred toppoint.de> wrote in message
 news:bvb54a$1qk5$1 digitaldaemon.com...
 Hello,

 i have write a test programm that takes 65 seconds.
 File tralala has 213753 lines.

 import std.stream;
 int main() {
 File fd = new File("tralala");
 char[][] werte;
 while (!fd.eof()) {
 fd.readLine();
 }
 return 0;
 }

 Give it a better way to read a File line by line?
 The same perl programm takes 11 seconds.
Jan 29 2004
next sibling parent "Matthew" <matthew.hat stlsoft.dot.org> writes:
"C" <dont respond.com> wrote in message
news:bvbpjm$2sjj$1 digitaldaemon.com...
 This C++ takes 3 seconds to read line by line.

 #include <fstream>
 #include <string>

 using namespace std;

 int main () {

  ifstream in("tralala");
  string line;
  while ( !in.eof ()  ) {

   std::getline(in,line);

  }
  return 1;
 }

 Ill try to use Mattys libs to find the bottleneck , as reading line by
line
 is a common operation.
Cool. Then you can "suggest" to Walter that we include it in Phobos. I've had no luck in that so far. :)
 C

 "C" <dont respond.com> wrote in message
 news:bvbolu$2qpf$1 digitaldaemon.com...
 Hmm thats interesting.

 std.file.read took about 1 second on a 250K line file , the while loop
took
 about 12 seconds for me , which is a very long time.

 is it the readLine call thats making this so slow ?

 C

 <manfred toppoint.de> wrote in message
 news:bvb54a$1qk5$1 digitaldaemon.com...
 Hello,

 i have write a test programm that takes 65 seconds.
 File tralala has 213753 lines.

 import std.stream;
 int main() {
 File fd = new File("tralala");
 char[][] werte;
 while (!fd.eof()) {
 fd.readLine();
 }
 return 0;
 }

 Give it a better way to read a File line by line?
 The same perl programm takes 11 seconds.
Jan 29 2004
prev sibling parent "Walter" <walter digitalmars.com> writes:
"C" <dont respond.com> wrote in message
news:bvbpjm$2sjj$1 digitaldaemon.com...
 Ill try to use Mattys libs to find the bottleneck , as reading line by
line
 is a common operation.
I tend to use std.file.read() followed by std.string.splitlines().
Jan 30 2004
prev sibling next sibling parent reply Damon Gray <dontbotherasking go.away.mr.bad.spammer.net> writes:
I believe it is readLine() that is may be making this slow. readLine 
seems to read a line one byte at a time. See the code for readLine in 
stream.d.

That is why I've been spending some free time lately creating a 
BufferedStream class that actually buffers read()s and uses that to get 
line oriented input. I'll post it when I've had a chance to get away 
from work enough to finish it. :)

-Damon-


C wrote:
 Hmm thats interesting.
 
 std.file.read took about 1 second on a 250K line file , the while loop took
 about 12 seconds for me , which is a very long time.
 
 is it the readLine call thats making this so slow ?
 
 C
 
 <manfred toppoint.de> wrote in message
 news:bvb54a$1qk5$1 digitaldaemon.com...
 
Hello,

i have write a test programm that takes 65 seconds.
File tralala has 213753 lines.

import std.stream;
int main() {
File fd = new File("tralala");
char[][] werte;
while (!fd.eof()) {
fd.readLine();
}
return 0;
}

Give it a better way to read a File line by line?
The same perl programm takes 11 seconds.
Jan 29 2004
parent "C" <dont respond.com> writes:
Ok cool I look forward to it.   I can't believe we don't have a phobos
library group yet.  So frustrurating...

C

"Damon Gray" <dontbotherasking go.away.mr.bad.spammer.net> wrote in message
news:40196AB3.5040107 go.away.mr.bad.spammer.net...
 I believe it is readLine() that is may be making this slow. readLine
 seems to read a line one byte at a time. See the code for readLine in
 stream.d.

 That is why I've been spending some free time lately creating a
 BufferedStream class that actually buffers read()s and uses that to get
 line oriented input. I'll post it when I've had a chance to get away
 from work enough to finish it. :)

 -Damon-


 C wrote:
 Hmm thats interesting.

 std.file.read took about 1 second on a 250K line file , the while loop
took
 about 12 seconds for me , which is a very long time.

 is it the readLine call thats making this so slow ?

 C

 <manfred toppoint.de> wrote in message
 news:bvb54a$1qk5$1 digitaldaemon.com...

Hello,

i have write a test programm that takes 65 seconds.
File tralala has 213753 lines.

import std.stream;
int main() {
File fd = new File("tralala");
char[][] werte;
while (!fd.eof()) {
fd.readLine();
}
return 0;
}

Give it a better way to read a File line by line?
The same perl programm takes 11 seconds.
Jan 29 2004
prev sibling parent reply Patrick Down <Patrick_member pathlink.com> writes:
I don't have the source in front of me right now but I believe it's the 
eof().  It is implemented in an atrocious fashion if I remember correctly.


In article <bvbolu$2qpf$1 digitaldaemon.com>, C says...
Hmm thats interesting.

std.file.read took about 1 second on a 250K line file , the while loop took
about 12 seconds for me , which is a very long time.

is it the readLine call thats making this so slow ?

C

<manfred toppoint.de> wrote in message
news:bvb54a$1qk5$1 digitaldaemon.com...
 Hello,

 i have write a test programm that takes 65 seconds.
 File tralala has 213753 lines.

 import std.stream;
 int main() {
 File fd = new File("tralala");
 char[][] werte;
 while (!fd.eof()) {
 fd.readLine();
 }
 return 0;
 }

 Give it a better way to read a File line by line?
 The same perl programm takes 11 seconds.
Jan 29 2004
parent reply "Matthew" <matthew.hat stlsoft.dot.org> writes:
 I don't have the source in front of me right now but I believe it's the
 eof().  It is implemented in an atrocious fashion if I remember correctly.
You weren't kidding: bit eof() { return position() == size(); } It w*nks. Unbelievable stuff! ulong position() { return seek(0, SeekPos.Current); } // returns size of stream ulong size() { ulong pos = position(), result = seek(0, SeekPos.End); position(pos); return result; } That's three kernel calls to check whether we've hit EOF! My confidence in stream.d has just fallen to about 5%. Walter, did anyone do a review on this before accepting it into Phobos? This needs a serious rewrite, if not a redesign.
 In article <bvbolu$2qpf$1 digitaldaemon.com>, C says...
Hmm thats interesting.

std.file.read took about 1 second on a 250K line file , the while loop
took
about 12 seconds for me , which is a very long time.

is it the readLine call thats making this so slow ?

C

<manfred toppoint.de> wrote in message
news:bvb54a$1qk5$1 digitaldaemon.com...
 Hello,

 i have write a test programm that takes 65 seconds.
 File tralala has 213753 lines.

 import std.stream;
 int main() {
 File fd = new File("tralala");
 char[][] werte;
 while (!fd.eof()) {
 fd.readLine();
 }
 return 0;
 }

 Give it a better way to read a File line by line?
 The same perl programm takes 11 seconds.
Jan 29 2004
parent reply "Walter" <walter digitalmars.com> writes:
"Matthew" <matthew.hat stlsoft.dot.org> wrote in message
news:bvc0vu$82o$2 digitaldaemon.com...
 Walter, did anyone do a review on this before accepting it into Phobos?
This
 needs a serious rewrite, if not a redesign.
No. It was very early on, done just to get things off the ground. It could use a rewrite. Anyone up for it?
Jan 30 2004
next sibling parent "Matthew" <matthew.hat stlsoft.dot.org> writes:
No, but I'll certainly volunteer reviewing services

"Walter" <walter digitalmars.com> wrote in message
news:bvfhj7$31e4$1 digitaldaemon.com...
 "Matthew" <matthew.hat stlsoft.dot.org> wrote in message
 news:bvc0vu$82o$2 digitaldaemon.com...
 Walter, did anyone do a review on this before accepting it into Phobos?
This
 needs a serious rewrite, if not a redesign.
No. It was very early on, done just to get things off the ground. It could use a rewrite. Anyone up for it?
Jan 30 2004
prev sibling parent reply Andy Friesen <andy ikagames.com> writes:
Walter wrote:
 "Matthew" <matthew.hat stlsoft.dot.org> wrote in message
 news:bvc0vu$82o$2 digitaldaemon.com...
 
Walter, did anyone do a review on this before accepting it into Phobos?
This
needs a serious rewrite, if not a redesign.
No. It was very early on, done just to get things off the ground. It could use a rewrite. Anyone up for it?
Here's an attempt: http://ikagames.com/andy/d/stream.d It's basically a bunch of classes that each do some small thing, and accepts a parent class as a template argument. The stream aliases at the very bottom mash them into a cohesive class. If nothing else, it'll get some ideas on the table. :) -- andy
Jan 30 2004
next sibling parent reply Manfred Nowak <svv1999 hotmail.com> writes:
Andy Friesen wrote:

 Here's an attempt:
 http://ikagames.com/andy/d/stream.d
[...] Tested it like this: import stream; void main(){ ReadableFile src=new ReadableFile(""); while(!src.eof()){ printf("."); Console.writeLine= src.readLine; } Console.writeLine="Hello world\n"; } But it did not throw an error. So long.
Jan 31 2004
next sibling parent reply Andy Friesen <andy ikagames.com> writes:
Manfred Nowak wrote:

 Andy Friesen wrote:
 
Here's an attempt:
http://ikagames.com/andy/d/stream.d
[...] Tested it like this: import stream; void main(){ ReadableFile src=new ReadableFile(""); while(!src.eof()){ printf("."); Console.writeLine= src.readLine; } Console.writeLine="Hello world\n"; } But it did not throw an error.
hm. Works for me. If it's not throwing an error, then fopen("") is returning a non-null value when "" is passed as a filename. This is puzzling. -- andy
Jan 31 2004
parent Manfred Nowak <svv1999 hotmail.com> writes:
Andy Friesen wrote:

 This is puzzling.
Shame on me. I started the wrong program. The compiled `stream.exe' throws the exception. Accidently I started `test.exe', which only showed, `Hello world'. Sorry for the inconvenience. So long.
Feb 01 2004
prev sibling parent Manfred Hansen <manfred toppoint.de> writes:
How are you install stream.d and compile this?
I have tried following:

dmd stream_tes1.d stream.d

i get the following errormessage from the linker:
hansen hansen-lx:~/d$ dmd stream_tes1.d stream1.d
gcc stream_tes1.o stream1.o -o stream_tes1 -lphobos -lpthread -lm
stream_tes1.o(.gnu.linkonce.t_Dmain+0x12): In function `_Dmain':
: undefined reference to
`_Class_7stream111__anonymous116ReadableFileStream_C7stream111__anonymous59BinaryReader_C7stream111__anonymous15Seeker_C6Object6Seeker12BinaryReader18ReadableFileStream'
collect2: ld returned 1 exit status
--- errorlevel 256

I have change the first line from stream.d to:
module stream;

The file's stream_tes1.d and stream.d are in the same directory. 

dmd -c stream_tes1.d stream.d run without any error.

Manfred

Manfred Nowak wrote:

 Andy Friesen wrote:
 
 Here's an attempt:
 http://ikagames.com/andy/d/stream.d
[...] Tested it like this: import stream; void main(){ ReadableFile src=new ReadableFile(""); while(!src.eof()){ printf("."); Console.writeLine= src.readLine; } Console.writeLine="Hello world\n"; } But it did not throw an error. So long.
Feb 01 2004
prev sibling parent reply "C" <dont respond.com> writes:
Quick note ( 859 )
        return handle != null;

should read !== null ( i know u know this ).

I like this template stacking , 'bolt-ins' you call them ?  Looks like fun ,
Ill play around with them :D.  Is Modern C++ the only book that does things
like this ?  Another source of documentation , examples , tutorial would be
cool too.

Also some unittests and examples ?  I tested the streams on the 250K file
with


void main ()
{
 try
 {
  ReadableFile x = new ReadableFile("tralala");

  while ( x.eof() )
  {
   Console.writeLine(x.readLine() );

  }
 }

 catch (IOError x)
 {
  Console.writeLine( x.toString() ) ;
 }
}

and I get no printing to stdout , it ends within .5 second , am I using it
wrong ?

Thanks,
C
"Andy Friesen" <andy ikagames.com> wrote in message
news:bvfk4b$3tl$1 digitaldaemon.com...
 Walter wrote:
 "Matthew" <matthew.hat stlsoft.dot.org> wrote in message
 news:bvc0vu$82o$2 digitaldaemon.com...

Walter, did anyone do a review on this before accepting it into Phobos?
This
needs a serious rewrite, if not a redesign.
No. It was very early on, done just to get things off the ground. It
could
 use a rewrite. Anyone up for it?
Here's an attempt: http://ikagames.com/andy/d/stream.d It's basically a bunch of classes that each do some small thing, and accepts a parent class as a template argument. The stream aliases at the very bottom mash them into a cohesive class. If nothing else, it'll get some ideas on the table. :) -- andy
Jan 31 2004
next sibling parent reply larry cowan <larry_member pathlink.com> writes:
In article <bvh0i9$2cqa$1 digitaldaemon.com>, C says...
Quick note ( 859 )
        return handle != null;

should read !== null ( i know u know this ).

I like this template stacking , 'bolt-ins' you call them ?  Looks like fun ,
Ill play around with them :D.  Is Modern C++ the only book that does things
like this ?  Another source of documentation , examples , tutorial would be
cool too.

Also some unittests and examples ?  I tested the streams on the 250K file
with


void main ()
{
 try
 {
  ReadableFile x = new ReadableFile("tralala");

  while ( x.eof() )
---------------------- while ( !x.eof() ) -------------------------- ?
  {
   Console.writeLine(x.readLine() );

  }
 }

 catch (IOError x)
 {
  Console.writeLine( x.toString() ) ;
 }
}

and I get no printing to stdout , it ends within .5 second , am I using it
wrong ?

Thanks,
C
"Andy Friesen" <andy ikagames.com> wrote in message
news:bvfk4b$3tl$1 digitaldaemon.com...
 Walter wrote:
 "Matthew" <matthew.hat stlsoft.dot.org> wrote in message
 news:bvc0vu$82o$2 digitaldaemon.com...

Walter, did anyone do a review on this before accepting it into Phobos?
This
needs a serious rewrite, if not a redesign.
No. It was very early on, done just to get things off the ground. It
could
 use a rewrite. Anyone up for it?
Here's an attempt: http://ikagames.com/andy/d/stream.d It's basically a bunch of classes that each do some small thing, and accepts a parent class as a template argument. The stream aliases at the very bottom mash them into a cohesive class. If nothing else, it'll get some ideas on the table. :) -- andy
Jan 31 2004
parent "C" <dont respond.com> writes:
Doh!  Im a jackass :S.

Took 3 seconds ( aproximate ), considerably faster !

C

"larry cowan" <larry_member pathlink.com> wrote in message
news:bvh1cj$2e8i$1 digitaldaemon.com...
 In article <bvh0i9$2cqa$1 digitaldaemon.com>, C says...
Quick note ( 859 )
        return handle != null;

should read !== null ( i know u know this ).

I like this template stacking , 'bolt-ins' you call them ?  Looks like
fun ,
Ill play around with them :D.  Is Modern C++ the only book that does
things
like this ?  Another source of documentation , examples , tutorial would
be
cool too.

Also some unittests and examples ?  I tested the streams on the 250K file
with


void main ()
{
 try
 {
  ReadableFile x = new ReadableFile("tralala");

  while ( x.eof() )
---------------------- while ( !x.eof() ) -------------------------- ?
  {
   Console.writeLine(x.readLine() );

  }
 }

 catch (IOError x)
 {
  Console.writeLine( x.toString() ) ;
 }
}

and I get no printing to stdout , it ends within .5 second , am I using
it
wrong ?

Thanks,
C
"Andy Friesen" <andy ikagames.com> wrote in message
news:bvfk4b$3tl$1 digitaldaemon.com...
 Walter wrote:
 "Matthew" <matthew.hat stlsoft.dot.org> wrote in message
 news:bvc0vu$82o$2 digitaldaemon.com...

Walter, did anyone do a review on this before accepting it into
Phobos?
 This

needs a serious rewrite, if not a redesign.
No. It was very early on, done just to get things off the ground. It
could
 use a rewrite. Anyone up for it?
Here's an attempt: http://ikagames.com/andy/d/stream.d It's basically a bunch of classes that each do some small thing, and accepts a parent class as a template argument. The stream aliases at the very bottom mash them into a cohesive class. If nothing else, it'll get some ideas on the table. :) -- andy
Jan 31 2004
prev sibling next sibling parent reply Andy Friesen <andy ikagames.com> writes:
C wrote:
 Quick note ( 859 )
         return handle != null;
 
 should read !== null ( i know u know this ).
Since handle is a pointer, and not a reference, it doesn't matter. I suppose it's a good idea, though, as it's better self-documentation if it consistently behaves like a reference.
 
 I like this template stacking , 'bolt-ins' you call them ?  Looks like fun ,
 Ill play around with them :D.  Is Modern C++ the only book that does things
 like this ?  Another source of documentation , examples , tutorial would be
 cool too.
Modern C++ goes way, way further than this. Most compilers tremble at the mere mention of Loki. (a library made by the same author, for demonstrating the techniques implemented in the book)
 
 Also some unittests and examples ?
 
Good idea. -- andy
Jan 31 2004
parent "C" <dont respond.com> writes:
 Since handle is a pointer, and not a reference, it doesn't matter.
Doh , your right of course.
 Modern C++ goes way, way further than this.  Most compilers tremble at
 the mere mention of Loki. (a library made by the same author, for
 demonstrating the techniques implemented in the book)
Hmm ok. You've done alot with D templates, anyway to convinvce you to write a tutorial on C++ and D templates ? I almost never write generic code in C++ but would like to in D. A tutorial would help alot of people, myself included. Also im a little confused on the use of interface , in C++ this would be a pure virtual abstract class ? How does interface work with templates , are there some advantages ? And looking over the code its not clear to me why you instantiate with Object , ill keep reading it. Thanks, C "Andy Friesen" <andy ikagames.com> wrote in message news:bvh82p$2p2c$1 digitaldaemon.com...
 C wrote:
 Quick note ( 859 )
         return handle != null;

 should read !== null ( i know u know this ).
Since handle is a pointer, and not a reference, it doesn't matter. I suppose it's a good idea, though, as it's better self-documentation if it consistently behaves like a reference.
 I like this template stacking , 'bolt-ins' you call them ?  Looks like
fun ,
 Ill play around with them :D.  Is Modern C++ the only book that does
things
 like this ?  Another source of documentation , examples , tutorial would
be
 cool too.
Modern C++ goes way, way further than this. Most compilers tremble at the mere mention of Loki. (a library made by the same author, for demonstrating the techniques implemented in the book)
 Also some unittests and examples ?
Good idea. -- andy
Feb 01 2004
prev sibling parent "Matthew" <matthew.hat stlsoft.dot.org> writes:
"C" <dont respond.com> wrote in message
news:bvh0i9$2cqa$1 digitaldaemon.com...
 Quick note ( 859 )
         return handle != null;

 should read !== null ( i know u know this ).

 I like this template stacking , 'bolt-ins' you call them ?  Looks like fun
,
 Ill play around with them :D.  Is Modern C++ the only book that does
things
 like this ?  Another source of documentation , examples , tutorial would
be
 cool too.
ATL uses this a lot, although they don't name the technique, so ATL Internals is also a good text.
 Also some unittests and examples ?  I tested the streams on the 250K file
 with


 void main ()
 {
  try
  {
   ReadableFile x = new ReadableFile("tralala");

   while ( x.eof() )
   {
    Console.writeLine(x.readLine() );

   }
  }

  catch (IOError x)
  {
   Console.writeLine( x.toString() ) ;
  }
 }

 and I get no printing to stdout , it ends within .5 second , am I using it
 wrong ?

 Thanks,
 C
 "Andy Friesen" <andy ikagames.com> wrote in message
 news:bvfk4b$3tl$1 digitaldaemon.com...
 Walter wrote:
 "Matthew" <matthew.hat stlsoft.dot.org> wrote in message
 news:bvc0vu$82o$2 digitaldaemon.com...

Walter, did anyone do a review on this before accepting it into
Phobos?
 This

needs a serious rewrite, if not a redesign.
No. It was very early on, done just to get things off the ground. It
could
 use a rewrite. Anyone up for it?
Here's an attempt: http://ikagames.com/andy/d/stream.d It's basically a bunch of classes that each do some small thing, and accepts a parent class as a template argument. The stream aliases at the very bottom mash them into a cohesive class. If nothing else, it'll get some ideas on the table. :) -- andy
Jan 31 2004