www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Using lazy code to process large files

reply =?UTF-8?Q?Martin_Dra=c5=a1ar?= via Digitalmars-d-learn writes:
Hi,

I am struggling to use a lazy range-based code to process large text
files. My task is simple, i.e., I can write a non-range-based code in a
really short time, but I wanted to try different approach and I am
hitting a wall after wall.

Task: read a csv-like input, take only lines starting with some string,
split by a comma, remove leading and trailing whitespaces from splitted
elements, join by comma again and write to an output.

My attempt so far:

alias stringStripLeft = std.string.stripLeft;

auto input  = File("input.csv");
auto output = File("output.csv");

auto result = input.byLine()
                   .filter!(a => a.startsWith("..."))
                   .map!(a => a.splitter(","))
                   .stringStripleft // <-- errors start here
                   .join(",");

output.write(result);

Needless to say, this does not compile. Basically, I don't know how to
feed MapResults to splitter and then sensibly join it.

Thank you for any hint.
Martin
Aug 02 2017
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/2/17 7:44 AM, Martin Dra¹ar via Digitalmars-d-learn wrote:

 Hi,
 
 I am struggling to use a lazy range-based code to process large text
 files. My task is simple, i.e., I can write a non-range-based code in a
 really short time, but I wanted to try different approach and I am
 hitting a wall after wall.
 
 Task: read a csv-like input, take only lines starting with some string,
 split by a comma, remove leading and trailing whitespaces from splitted
 elements, join by comma again and write to an output.
 
 My attempt so far:
 
 alias stringStripLeft = std.string.stripLeft;
 
 auto input  = File("input.csv");
 auto output = File("output.csv");
 
 auto result = input.byLine()
                     .filter!(a => a.startsWith("..."))
                     .map!(a => a.splitter(","))
                     .stringStripleft // <-- errors start here
                     .join(",");
 
 output.write(result);
 
 Needless to say, this does not compile. Basically, I don't know how to
 feed MapResults to splitter and then sensibly join it.


The problem is that you are 2 ranges deep when you apply splitter. The 
result of the map is a range of ranges.

Then when you apply stringStripleft, you are applying to the map result, 
not the splitter result.

What you need is to bury the action on each string into the map:

.map!(a => a.splitter(",").map!(stringStripLeft).join(","))

The internal map is because stripLeft doesn't take a range of strings 
(the result of splitter), it takes a range of dchar (which is each 
element of splitter). So you use map to apply the function to every element.

Disclaimer: I haven't tested to see this works, but I think it should.

Note that I have forwarded your call to join, even though this actually 
is not lazy, it builds a string out of it (and actually probably a 
dstring). Use joiner to do it truly lazily.

I will also note that the result is not going to look like what you 
think, as outputting a range looks like this: [element, element, 
element, ...]

You could potentially output like this:

output.write(result.joiner("\n"));

Which I think will work. Again, no testing.

I wouldn't expect good performance from this, as there is auto-decoding 
all over the place.

-Steve
Aug 02 2017
next sibling parent reply =?UTF-8?Q?Martin_Dra=c5=a1ar?= via Digitalmars-d-learn writes:
Dne 2.8.2017 v 14:45 Steven Schveighoffer via Digitalmars-d-learn napsal(a):


 The problem is that you are 2 ranges deep when you apply splitter. The
 result of the map is a range of ranges.
 
 Then when you apply stringStripleft, you are applying to the map result,
 not the splitter result.
 
 What you need is to bury the action on each string into the map:
 
 .map!(a => a.splitter(",").map!(stringStripLeft).join(","))
 
 The internal map is because stripLeft doesn't take a range of strings
 (the result of splitter), it takes a range of dchar (which is each
 element of splitter). So you use map to apply the function to every
 element.
 
 Disclaimer: I haven't tested to see this works, but I think it should.
 
 Note that I have forwarded your call to join, even though this actually
 is not lazy, it builds a string out of it (and actually probably a
 dstring). Use joiner to do it truly lazily.
 
 I will also note that the result is not going to look like what you
 think, as outputting a range looks like this: [element, element,
 element, ...]
 
 You could potentially output like this:
 
 output.write(result.joiner("\n"));
 
 Which I think will work. Again, no testing.
 
 I wouldn't expect good performance from this, as there is auto-decoding
 all over the place.
 
 -Steve


Thanks Steven for the explanation. Just to clarify - what would be
needed to avoid auto-decoding in this case? Process it all as an arrays,
using byChunk to read it, etc?

 kdevel: Thank you for your solution as well.

Martin
Aug 02 2017
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/2/17 8:59 AM, Martin Dra¹ar via Digitalmars-d-learn wrote:


 Thanks Steven for the explanation. Just to clarify - what would be
 needed to avoid auto-decoding in this case? Process it all as an arrays,
 using byChunk to read it, etc?
 


As Daniel said, using byCodeUnit will help.

I don't know what the result of this is when outputting, however. I'd be 
concerned it just integer promoted the data to dchars before outputting. 
If your file data is all ASCII it should work fine. You'd have to 
experiment to see how it works.

-Steve
Aug 02 2017
parent reply kdevel <kdevel vogtner.de> writes:
On Wednesday, 2 August 2017 at 13:45:01 UTC, Steven Schveighoffer 
wrote:

 As Daniel said, using byCodeUnit will help.


stripLeft seems to autodecode even when fed with CodeUnits. How 
do I prevent this?

       1 void main ()
       2 {
       3    import std.stdio;
       4    import std.string;
       5    import std.conv;
       6    import std.utf;
       7    import std.algorithm;
       8
       9    string [] src = [ " \xfc" ]; // blank + latin-1 
encoded u umlaut
      10    auto result = src
      11       .map!(a => a.byCodeUnit)
      12       .map!(a => a.stripLeft);
      13    result.writeln;
      14 }

Crashes with a C++-like dump.
Aug 02 2017
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/2/17 11:02 AM, kdevel wrote:

 On Wednesday, 2 August 2017 at 13:45:01 UTC, Steven Schveighoffer wrote:

 As Daniel said, using byCodeUnit will help.


 
 stripLeft seems to autodecode even when fed with CodeUnits. How do I 
 prevent this?
 
        1 void main ()
        2 {
        3    import std.stdio;
        4    import std.string;
        5    import std.conv;
        6    import std.utf;
        7    import std.algorithm;
        8
        9    string [] src = [ " \xfc" ]; // blank + latin-1 encoded u 
 umlaut
       10    auto result = src
       11       .map!(a => a.byCodeUnit)
       12       .map!(a => a.stripLeft);
       13    result.writeln;
       14 }
 
 Crashes with a C++-like dump.
 


First, as a tip, please post either a link to a paste site, or don't put 
the line numbers. It's much easier to copy-paste your code into an 
editor if you don't have the line numbers.

What has happened is that you injected a non-encoded code point. In 
UTF8, any code point above 0x7f must be encoded into a string of several 
code units. See the table on this page: https://en.wikipedia.org/wiki/%C3%9C

If we use the correct code unit sequence (0xc3 0x9c), then it works: 
https://run.dlang.io/is/4umQoo

-Steve
Aug 02 2017
parent reply kdevel <kdevel vogtner.de> writes:
On Wednesday, 2 August 2017 at 15:52:13 UTC, Steven Schveighoffer 
wrote:

[...]


 First, as a tip, please post either a link to a paste site, or 
 don't put the line numbers. It's much easier to copy-paste your 
 code into an editor if you don't have the line numbers.


With pleasure.

[...]


 If we use the correct code unit sequence (0xc3 0x9c), then [...]


If I avoid std.string.stripLeft and use std.algorithm.stripLeft(' 
') instead it works as expected:

void main ()
{
    import std.stdio;
    import std.utf;
    import std.algorithm;

    string [] src = [ " \xfc" ]; // blank + latin-1 encoded u 
umlaut
    auto result = src
       .map!(a => a.byCodeUnit)
       .map!(a => a.stripLeft(' '));
    result.writeln;
}
Aug 02 2017
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/2/17 1:16 PM, kdevel wrote:

 On Wednesday, 2 August 2017 at 15:52:13 UTC, Steven Schveighoffer wrote:
 

 If we use the correct code unit sequence (0xc3 0x9c), then [...]


 
 If I avoid std.string.stripLeft and use std.algorithm.stripLeft(' ') 
 instead it works as expected:


What is expected? What I see on the screen when I run my code is:

[Ü]

What I see when I run your "working" code is:

[?]

You are missing the point that your input string is invalid.

std.algorithm is not validating the entire string, and so it doesn't 
throw an error like string.stripLeft does. writeln doesn't do any 
decoding of individual strings. It avoids the problem and just copies 
your bad data directly.

If you fix the input, both will work correctly.

-Steve
Aug 02 2017
parent reply kdevel <kdevel vogtner.de> writes:
On Wednesday, 2 August 2017 at 17:37:09 UTC, Steven Schveighoffer 
wrote:


 What is expected? What I see on the screen when I run my code 
 is:

 [Ü]


Upper case?


 What I see when I run your "working" code is:

 [?]


Your terminal is incapable of rendering the Latin-1 encoding. The 
program prints one byte of value 0xfc. You may pipe the output 
into hexdump -C:

00000000  5b fc 5d 0a                                       |[ü].|
00000004


 You are missing the point that your input string is invalid.


It's perfectly okay to put any value a octet can take into an 
octet. I did not claim that the data in the string memory is 
syntactically valid UTF-8. Read the comment in line 9 of my post 
of 15:02:22.


 std.algorithm is not validating the entire string,


True and it should not. So this is what I want.


 and so it doesn't throw an error like string.stripLeft does.


That is the point. You wrote

| I wouldn't expect good performance from this, as there is 
auto-decoding all
| over the place.

I erroneously thought that using byCodeUnit disables the whole 
UTF-8 processing and enforces operation on (u)bytes. But this is 
not the case at least not for stripLeft and probably other string 
functions.


 writeln doesn't do any decoding of individual strings. It 
 avoids the problem and just copies your bad data directly.


That is what I expected.
Aug 02 2017
next sibling parent ag0aep6g <anonymous example.com> writes:
On 08/02/2017 08:28 PM, kdevel wrote:

 It's perfectly okay to put any value a octet can take into an octet. I 
 did not claim that the data in the string memory is syntactically valid 
 UTF-8. Read the comment in line 9 of my post of 15:02:22.


You're claiming that the data is in UTF-8 when you use `string` as the 
type. For arbitrary octets, use something like `ubyte[]`.
Aug 02 2017
prev sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/2/17 2:28 PM, kdevel wrote:

 On Wednesday, 2 August 2017 at 17:37:09 UTC, Steven Schveighoffer wrote:
 

 What is expected? What I see on the screen when I run my code is:

 [Ü]


 
 Upper case?


Sorry, should be c3 bc, not c3 9c. I misread the table on that wikipedia 
entry.


 What I see when I run your "working" code is:

 [?]


 
 Your terminal is incapable of rendering the Latin-1 encoding. The 
 program prints one byte of value 0xfc. You may pipe the output into 
 hexdump -C:
 
 00000000  5b fc 5d 0a                                       |[ü].|
 00000004


Right, I saw that. But it's still not valid utf8, which is what char and 
string are.


 You are missing the point that your input string is invalid.


 
 It's perfectly okay to put any value a octet can take into an octet. I 
 did not claim that the data in the string memory is syntactically valid 
 UTF-8. Read the comment in line 9 of my post of 15:02:22.


Except a string is utf8, period. char is a utf8 code unit, period.

If you want some other encoding, it has to be defined as a different 
type. Otherwise, you will get errors when using any D library, all of 
which should expect char to be a utf8 code-unit.


 std.algorithm is not validating the entire string,


 
 True and it should not. So this is what I want.


But it's not the same as the original. For instance, the original would 
strip tabs, yours does not.


 and so it doesn't throw an error like string.stripLeft does.


 
 That is the point. You wrote
 
 | I wouldn't expect good performance from this, as there is 
 auto-decoding all
 | over the place.
 
 I erroneously thought that using byCodeUnit disables the whole UTF-8 
 processing and enforces operation on (u)bytes. But this is not the case 
 at least not for stripLeft and probably other string functions.


std.string.stripLeft is still expecting unicode, as it's testing 
std.uni.isWhite. So it has to do decoding. std.algorithm.stripLeft (the 
way you called it anyway) is looking at char instances and doing a 
direct comparison to ONE char (' '), so it can be much much faster and 
does not have to decode. This is an optimization, not a feature. I 
wouldn't be surprised, for instance, if byCodeUnit threw an error when 
encountering an invalid sequence in debug mode or something.

If your goal is to only look for that ascii character, then using 
byCodeUnit is required to avoid auto-decoding, which is where the 
unexpected slowdown would come.

But string functions that are specifically looking for unicode sequences 
are still going to decode, even if the range isn't doing it proactively.

In any case, the input data is not valid, you should use ubyte[], or 
some other type array, not strings.

-Steve
Aug 02 2017
prev sibling next sibling parent Daniel Kozak via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
using http://dlang.org/phobos/std_utf.html#byCodeUnit could help

On Wed, Aug 2, 2017 at 2:59 PM, Martin Dra=C5=A1ar via Digitalmars-d-learn =
<
digitalmars-d-learn puremagic.com> wrote:


 Dne 2.8.2017 v 14:45 Steven Schveighoffer via Digitalmars-d-learn
 napsal(a):


 The problem is that you are 2 ranges deep when you apply splitter. The
 result of the map is a range of ranges.

 Then when you apply stringStripleft, you are applying to the map result=




,

 not the splitter result.

 What you need is to bury the action on each string into the map:

 .map!(a =3D> a.splitter(",").map!(stringStripLeft).join(","))

 The internal map is because stripLeft doesn't take a range of strings
 (the result of splitter), it takes a range of dchar (which is each
 element of splitter). So you use map to apply the function to every
 element.

 Disclaimer: I haven't tested to see this works, but I think it should.

 Note that I have forwarded your call to join, even though this actually
 is not lazy, it builds a string out of it (and actually probably a
 dstring). Use joiner to do it truly lazily.

 I will also note that the result is not going to look like what you
 think, as outputting a range looks like this: [element, element,
 element, ...]

 You could potentially output like this:

 output.write(result.joiner("\n"));

 Which I think will work. Again, no testing.

 I wouldn't expect good performance from this, as there is auto-decoding
 all over the place.

 -Steve


 Thanks Steven for the explanation. Just to clarify - what would be
 needed to avoid auto-decoding in this case? Process it all as an arrays,
 using byChunk to read it, etc?

  kdevel: Thank you for your solution as well.

 Martin
Aug 02 2017
prev sibling parent Daniel Kozak via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
something like file.byLine.map!(a=3D>a.byCodeUnit)

On Wed, Aug 2, 2017 at 3:01 PM, Daniel Kozak <kozzi11 gmail.com> wrote:


 using http://dlang.org/phobos/std_utf.html#byCodeUnit could help

 On Wed, Aug 2, 2017 at 2:59 PM, Martin Dra=C5=A1ar via Digitalmars-d-lear=


n <

 digitalmars-d-learn puremagic.com> wrote:


 Dne 2.8.2017 v 14:45 Steven Schveighoffer via Digitalmars-d-learn
 napsal(a):


 The problem is that you are 2 ranges deep when you apply splitter. The
 result of the map is a range of ranges.

 Then when you apply stringStripleft, you are applying to the map resul=






t,

 not the splitter result.

 What you need is to bury the action on each string into the map:

 .map!(a =3D> a.splitter(",").map!(stringStripLeft).join(","))

 The internal map is because stripLeft doesn't take a range of strings
 (the result of splitter), it takes a range of dchar (which is each
 element of splitter). So you use map to apply the function to every
 element.

 Disclaimer: I haven't tested to see this works, but I think it should.

 Note that I have forwarded your call to join, even though this actuall=






y

 is not lazy, it builds a string out of it (and actually probably a
 dstring). Use joiner to do it truly lazily.

 I will also note that the result is not going to look like what you
 think, as outputting a range looks like this: [element, element,
 element, ...]

 You could potentially output like this:

 output.write(result.joiner("\n"));

 Which I think will work. Again, no testing.

 I wouldn't expect good performance from this, as there is auto-decodin=






g

 all over the place.

 -Steve


 Thanks Steven for the explanation. Just to clarify - what would be
 needed to avoid auto-decoding in this case? Process it all as an arrays,
 using byChunk to read it, etc?

  kdevel: Thank you for your solution as well.

 Martin
Aug 02 2017
prev sibling parent kdevel <kdevel vogtner.de> writes:
On Wednesday, 2 August 2017 at 11:44:30 UTC, Martin Drašar wrote:

 Thank you for any hint.


       1 import std.stdio;
       2 import std.string;
       3 import std.algorithm;
       4 import std.conv;
       5
       6 void main ()
       7 {
       8    auto input  = File("input.csv");
       9
      10    auto result = input.byLine()
      11       .filter!(a => a.startsWith("..."))
      12       .map!(a => a.splitter(",")
      13          .map!(b => b.stripLeft)
      14          .join(","))
      15       .join("\n");
      16
      17    auto output = File("output.csv", "w");
      18    output.write(result);
      19 }
Aug 02 2017