digitalmars.D.learn - Performant method for reading huge text files
- Rene Zwanenburg (13/13) Feb 03 2014 I'm running into a problem I've come across before but never
- bearophile (6/9) Feb 03 2014 Are you reading the text by lines? In Bugzilla there is a
- Rene Zwanenburg (8/18) Feb 03 2014 Nope, I'm feeding it to csvReader which uses an input range of
- Chris Williams (5/26) Feb 04 2014 Parsing should be faster than I/O. Set up two buffers and have
- Chris Williams (2/5) Feb 04 2014 ...and then flip buffers whenever the slower of the two has
- Marco Leise (15/38) Feb 06 2014 Parsing textual representations of numbers is slow. The other
- Kagamin (2/2) Feb 05 2014 You can also try a BufferedRange.
I'm running into a problem I've come across before but never found a satisfactory solution for. There's a pretty large ascii file I need to process, currently about 3GB but size will increase in the future. D's ranges in combination with std.algorithm are simply perfect for what I'm doing, and it's trivial to write nice code which doesn't load the entire file into memory. The problem is speed. I'm using LockingTextReader in std.stdio, but it't not nearly fast enough. On my system it only reads about 3 MB/s with one core spending all it's time in IO calls. Sadly I need to support 32 bit, so memory mapped files aren't an option. Does someone know of a way to increase throughput while still allowing me to use a range API?
Feb 03 2014
Rene Zwanenburg:The problem is speed. I'm using LockingTextReader in std.stdio, but it't not nearly fast enough. On my system it only reads about 3 MB/s with one core spending all it's time in IO calls.Are you reading the text by lines? In Bugzilla there is a byLineFast: https://d.puremagic.com/issues/show_bug.cgi?id=11810 Bye, bearophile
Feb 03 2014
On Monday, 3 February 2014 at 23:50:54 UTC, bearophile wrote:Rene Zwanenburg:Nope, I'm feeding it to csvReader which uses an input range of characters. Come to think of it.. Well this is embarassing, I've been sloppy with my profiling :). It appears the time is actually spent converting strings to doubles, done by csvReader to read a row into my Record struct. No way to speed that up I suppose. Still I find it surprising that parsing doubles is so slow.The problem is speed. I'm using LockingTextReader in std.stdio, but it't not nearly fast enough. On my system it only reads about 3 MB/s with one core spending all it's time in IO calls.Are you reading the text by lines? In Bugzilla there is a byLineFast: https://d.puremagic.com/issues/show_bug.cgi?id=11810 Bye, bearophile
Feb 03 2014
On Tuesday, 4 February 2014 at 00:04:23 UTC, Rene Zwanenburg wrote:On Monday, 3 February 2014 at 23:50:54 UTC, bearophile wrote:Parsing should be faster than I/O. Set up two buffers and have one thread reading into buffer A while you parse buffer B with a second thread.Rene Zwanenburg:Nope, I'm feeding it to csvReader which uses an input range of characters. Come to think of it.. Well this is embarassing, I've been sloppy with my profiling :). It appears the time is actually spent converting strings to doubles, done by csvReader to read a row into my Record struct. No way to speed that up I suppose. Still I find it surprising that parsing doubles is so slow.The problem is speed. I'm using LockingTextReader in std.stdio, but it't not nearly fast enough. On my system it only reads about 3 MB/s with one core spending all it's time in IO calls.Are you reading the text by lines? In Bugzilla there is a byLineFast: https://d.puremagic.com/issues/show_bug.cgi?id=11810 Bye, bearophile
Feb 04 2014
Parsing should be faster than I/O. Set up two buffers and have one thread reading into buffer A while you parse buffer B with a second thread....and then flip buffers whenever the slower of the two has completed.
Feb 04 2014
Am Tue, 04 Feb 2014 00:04:22 +0000 schrieb "Rene Zwanenburg" <renezwanenburg gmail.com>:On Monday, 3 February 2014 at 23:50:54 UTC, bearophile wrote:Parsing textual representations of numbers is slow. The other way around is faster. You have to check all kinds of stuff, like preceding +/-, starts with a dot, are all characters '0' to '9', is there an exponent? Is it "NaN" or "nan"? Floating point math is slow, but when you store the intermediate results while parsing inside an integer, you may run out of digits if the number string is long. On the other hand repeated floating point math will introduce some error as you append digits. Here is the ~400 lines version in Phobos: https://github.com/D-Programming-Language/phobos/blob/master/std/conv.d#L2250 -- MarcoRene Zwanenburg:Nope, I'm feeding it to csvReader which uses an input range of characters. Come to think of it.. Well this is embarassing, I've been sloppy with my profiling :). It appears the time is actually spent converting strings to doubles, done by csvReader to read a row into my Record struct. No way to speed that up I suppose. Still I find it surprising that parsing doubles is so slow.The problem is speed. I'm using LockingTextReader in std.stdio, but it't not nearly fast enough. On my system it only reads about 3 MB/s with one core spending all it's time in IO calls.Are you reading the text by lines? In Bugzilla there is a byLineFast: https://d.puremagic.com/issues/show_bug.cgi?id=11810 Bye, bearophile
Feb 06 2014
You can also try a BufferedRange. http://forum.dlang.org/thread/l9q66g$2he3$1 digitalmars.com
Feb 05 2014