www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Performant method for reading huge text files

reply "Rene Zwanenburg" <renezwanenburg gmail.com> writes:
I'm running into a problem I've come across before but never 
found a satisfactory solution for.

There's a pretty large ascii file I need to process, currently 
about 3GB but size will increase in the future. D's ranges in 
combination with std.algorithm are simply perfect for what I'm 
doing, and it's trivial to write nice code which doesn't load the 
entire file into memory.

The problem is speed. I'm using LockingTextReader in std.stdio, 
but it't not nearly fast enough. On my system it only reads about 
3 MB/s with one core spending all it's time in IO calls.

Sadly I need to support 32 bit, so memory mapped files aren't an 
option. Does someone know of a way to increase throughput while 
still allowing me to use a range API?
Feb 03 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Rene Zwanenburg:

 The problem is speed. I'm using LockingTextReader in std.stdio, 
 but it't not nearly fast enough. On my system it only reads 
 about 3 MB/s with one core spending all it's time in IO calls.
Are you reading the text by lines? In Bugzilla there is a byLineFast: https://d.puremagic.com/issues/show_bug.cgi?id=11810 Bye, bearophile
Feb 03 2014
parent reply "Rene Zwanenburg" <renezwanenburg gmail.com> writes:
On Monday, 3 February 2014 at 23:50:54 UTC, bearophile wrote:
 Rene Zwanenburg:

 The problem is speed. I'm using LockingTextReader in 
 std.stdio, but it't not nearly fast enough. On my system it 
 only reads about 3 MB/s with one core spending all it's time 
 in IO calls.
Are you reading the text by lines? In Bugzilla there is a byLineFast: https://d.puremagic.com/issues/show_bug.cgi?id=11810 Bye, bearophile
Nope, I'm feeding it to csvReader which uses an input range of characters. Come to think of it.. Well this is embarassing, I've been sloppy with my profiling :). It appears the time is actually spent converting strings to doubles, done by csvReader to read a row into my Record struct. No way to speed that up I suppose. Still I find it surprising that parsing doubles is so slow.
Feb 03 2014
next sibling parent reply "Chris Williams" <yoreanon-chrisw yahoo.co.jp> writes:
On Tuesday, 4 February 2014 at 00:04:23 UTC, Rene Zwanenburg 
wrote:
 On Monday, 3 February 2014 at 23:50:54 UTC, bearophile wrote:
 Rene Zwanenburg:

 The problem is speed. I'm using LockingTextReader in 
 std.stdio, but it't not nearly fast enough. On my system it 
 only reads about 3 MB/s with one core spending all it's time 
 in IO calls.
Are you reading the text by lines? In Bugzilla there is a byLineFast: https://d.puremagic.com/issues/show_bug.cgi?id=11810 Bye, bearophile
Nope, I'm feeding it to csvReader which uses an input range of characters. Come to think of it.. Well this is embarassing, I've been sloppy with my profiling :). It appears the time is actually spent converting strings to doubles, done by csvReader to read a row into my Record struct. No way to speed that up I suppose. Still I find it surprising that parsing doubles is so slow.
Parsing should be faster than I/O. Set up two buffers and have one thread reading into buffer A while you parse buffer B with a second thread.
Feb 04 2014
parent "Chris Williams" <yoreanon-chrisw yahoo.co.jp> writes:
 Parsing should be faster than I/O. Set up two buffers and have 
 one thread reading into buffer A while you parse buffer B with 
 a second thread.
...and then flip buffers whenever the slower of the two has completed.
Feb 04 2014
prev sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 04 Feb 2014 00:04:22 +0000
schrieb "Rene Zwanenburg" <renezwanenburg gmail.com>:

 On Monday, 3 February 2014 at 23:50:54 UTC, bearophile wrote:
 Rene Zwanenburg:

 The problem is speed. I'm using LockingTextReader in 
 std.stdio, but it't not nearly fast enough. On my system it 
 only reads about 3 MB/s with one core spending all it's time 
 in IO calls.
Are you reading the text by lines? In Bugzilla there is a byLineFast: https://d.puremagic.com/issues/show_bug.cgi?id=11810 Bye, bearophile
Nope, I'm feeding it to csvReader which uses an input range of characters. Come to think of it.. Well this is embarassing, I've been sloppy with my profiling :). It appears the time is actually spent converting strings to doubles, done by csvReader to read a row into my Record struct. No way to speed that up I suppose. Still I find it surprising that parsing doubles is so slow.
Parsing textual representations of numbers is slow. The other way around is faster. You have to check all kinds of stuff, like preceding +/-, starts with a dot, are all characters '0' to '9', is there an exponent? Is it "NaN" or "nan"? Floating point math is slow, but when you store the intermediate results while parsing inside an integer, you may run out of digits if the number string is long. On the other hand repeated floating point math will introduce some error as you append digits. Here is the ~400 lines version in Phobos: https://github.com/D-Programming-Language/phobos/blob/master/std/conv.d#L2250 -- Marco
Feb 06 2014
prev sibling parent "Kagamin" <spam here.lot> writes:
You can also try a BufferedRange.
http://forum.dlang.org/thread/l9q66g$2he3$1 digitalmars.com
Feb 05 2014