www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Why is there no std.stream anymore?

reply Jordi =?UTF-8?B?R3V0acOpcnJleg==?= Hermoso <jordigh octave.org> writes:
I'd like to read from a file, one byte at a time, without loading 
the whole file in memory.

I was hoping I could do something like

    auto f = File("somefile");
    foreach(c; f.byChar) {
        process(c);
    }

but there appears to be no such way to do it anymore. Instead, 
the stdlib seems to provide several functions to do chunked reads 
from the file where I have to manually manage the buffer. I see 
that D1 had a stream, but it's no longer here and I understand 
ranges are supposed to be used instead.

What's the explanation here? Why is there no more stream and what 
am I supposed to use instead? Do I really need to be manually 
managing the read buffer myself?
Dec 11 2017
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/11/17 3:51 PM, Jordi Gutiérrez Hermoso wrote:

 I'd like to read from a file, one byte at a time, without loading the 
 whole file in memory.
 
 I was hoping I could do something like
 
     auto f = File("somefile");
     foreach(c; f.byChar) {
         process(c);
     }
 
 but there appears to be no such way to do it anymore. Instead, the 
 stdlib seems to provide several functions to do chunked reads from the 
 file where I have to manually manage the buffer. I see that D1 had a 
 stream, but it's no longer here and I understand ranges are supposed to 
 be used instead.
 
 What's the explanation here? Why is there no more stream and what am I 
 supposed to use instead? Do I really need to be manually managing the 
 read buffer myself?


Use the undead repository:

http://code.dlang.org/packages/undead

https://github.com/dlang/undeaD

https://github.com/dlang/undeaD/blob/master/src/undead/stream.d

-Steve
Dec 11 2017
parent reply Jordi =?UTF-8?B?R3V0acOpcnJleg==?= Hermoso <jordigh octave.org> writes:
On Monday, 11 December 2017 at 21:21:51 UTC, Steven Schveighoffer 
wrote:

 Use the undead repository:


Wow, really? Is the removal of stream from D some kind of error 
that hasn't been corrected yet?
Dec 11 2017
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/11/17 5:58 PM, Jordi Gutiérrez Hermoso wrote:

 On Monday, 11 December 2017 at 21:21:51 UTC, Steven Schveighoffer wrote:

 Use the undead repository:


 
 Wow, really? Is the removal of stream from D some kind of error that 
 hasn't been corrected yet?


No, it was removed because it was considered subpar/obsolete. It doesn't 
jive with the rest of Phobos.

But modules that are removed are put into the undead repository for 
those who wish to continue using it.

-Steve
Dec 11 2017
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, December 11, 2017 22:58:53 Jordi Gutiérrez Hermoso via 
Digitalmars-d-learn wrote:

 On Monday, 11 December 2017 at 21:21:51 UTC, Steven Schveighoffer

 wrote:

 Use the undead repository:


 Wow, really? Is the removal of stream from D some kind of error
 that hasn't been corrected yet?


std.stream was deemed to not be up to Phobos' current standards, and it's
not in line with Phobos' current design and implementation (most notably, it
doesn't support ranges at all). No one has cared enough to come up with an
alternative implementation and propose it for inclusion in Phobos. However,
for most needs, ranges do what you might do with a stream solution, and
std.bitmanip provides useful functions for byte-level manipulation (e.g.
taking the first elements from a range of ubytes and converting them to
int). Depending on what you're looking for,
http://code.dlang.org/packages/iopipe could also fit in quite well, though
it's very much a work in progress, and there are several serialization
libraries on code.dlang.org if that's more what you're looking for.

- Jonathan M Davis
Dec 11 2017
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Monday, 11 December 2017 at 22:58:53 UTC, Jordi Gutiérrez 
Hermoso wrote:

 On Monday, 11 December 2017 at 21:21:51 UTC, Steven 
 Schveighoffer wrote:

 Use the undead repository:


 Wow, really? Is the removal of stream from D some kind of error 
 that hasn't been corrected yet?


Well of course you can use ranges for it, see e.g. this simple 
example:

---
void main(string[] args)
{
     import std.conv, std.range, std.stdio;
     foreach (d; 
File(__FILE_FULL_PATH__).byChunk(4096).join.take(5)) {
         writefln("%s", d.to!char);
     }
}
---

Run here: https://run.dlang.io/is/Ann9e9

Though if you need superb performance, iopipe or similar will be 
faster.
Dec 11 2017
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 12/11/17 6:33 PM, Seb wrote:

 Though if you need superb performance, iopipe or similar will be faster.


Since iopipe was mentioned several times, I will say a couple things:

1. iopipe is not made for processing one element at a time, it focuses 
on buffers. The reason for this is because certain tasks (i.e. parsing) 
are much more efficient with buffered data than when using the range 
API. Even with FILE *, using fgetc for every character is going to suck 
when compared to fread, and processing the resulting array in-memory.

2. If you do want to process by element, I recommend the following chain:

// an example that uses iopipe's file stream and assumes it's UTF8 text.
// other mechanisms are available.
auto mypipe = openDev("somefile") // open a file
     .bufd                         // buffer it
     .assumeText                   // assume it's utf-8 text
     .ensureDecodeable;            // ensure there are no partial 
code-points in the window

// convert to range of "chunks", and then join into one large range
foreach(c; mypipe.asInputRange.joiner) {
     process(c);
}

Note, due to Phobos's auto-decoding, joiner is going to auto-decode all 
of the data. This means typeof(c) is going to be dchar, and not char, 
and everything needs to be proper utf-8. If you want to process the 
bytes raw, you can omit the .assumeText.ensureDecodeable part, and the 
data will be ubytes.

-Steve
Dec 12 2017
parent reply aberba <karabutaworld gmail.com> writes:
On Tuesday, 12 December 2017 at 20:51:30 UTC, Steven 
Schveighoffer wrote:

 On 12/11/17 6:33 PM, Seb wrote:

 [...]


 Since iopipe was mentioned several times, I will say a couple 
 things:

 [...]



I should really try iopipe this time round. I think I avoided 
toying with it because the making conventions put me off. Don't 
remember about the docs and available examples though.
Jun 18 2020
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/18/20 10:53 AM, aberba wrote:

 On Tuesday, 12 December 2017 at 20:51:30 UTC, Steven Schveighoffer wrote:

 On 12/11/17 6:33 PM, Seb wrote:

 [...]


 Since iopipe was mentioned several times, I will say a couple things:

 [...]


 
 
 I should really try iopipe this time round. I think I avoided toying 
 with it because the making conventions put me off. Don't remember about 
 the docs and available examples though.


I have made some updates that are probably not reflected yet in the 
docs. But let me know if there are any stumbling blocks. I intend to 
make the iopipe + std.io (from Martin) a more user-friendly library in 
the near future.

-Steve
Jun 18 2020
parent aberba <karabutaworld gmail.com> writes:
On Thursday, 18 June 2020 at 15:03:38 UTC, Steven Schveighoffer 
wrote:

 On 6/18/20 10:53 AM, aberba wrote:

 On Tuesday, 12 December 2017 at 20:51:30 UTC, Steven 
 Schveighoffer wrote:

 On 12/11/17 6:33 PM, Seb wrote:

 [...]


 Since iopipe was mentioned several times, I will say a couple 
 things:

 [...]


 
 
 I should really try iopipe this time round. I think I avoided 
 toying with it because the making conventions put me off. 
 Don't remember about the docs and available examples though.


 I have made some updates that are probably not reflected yet in 
 the docs. But let me know if there are any stumbling blocks. I 
 intend to make the iopipe + std.io (from Martin) a more 
 user-friendly library in the near future.

 -Steve


Will discuss in the GH issue
Jun 18 2020
prev sibling parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Thursday, 18 June 2020 at 14:53:58 UTC, aberba wrote:

 On Tuesday, 12 December 2017 at 20:51:30 UTC, Steven 
 Schveighoffer wrote:

 On 12/11/17 6:33 PM, Seb wrote:

 [...]


 Since iopipe was mentioned several times, I will say a couple 
 things:

 [...]



 I should really try iopipe this time round. I think I avoided 
 toying with it because the making conventions put me off. Don't 
 remember about the docs and available examples though.


I too was trying to utilize iopipe and asked questions earlier 
this year[1] and I made a file writer util[2]. I haven't really 
taken advantage of the power yet, though it does appear that be a 
really nice abstraction.

1. 
https://forum.dlang.org/thread/faawejguebluwodflevh forum.dlang.org
2. 
https://gitlab.com/jessephillips/devarticlator/-/blob/master/source/util/file.d
Jun 18 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/18/20 11:11 PM, Jesse Phillips wrote:

 On Thursday, 18 June 2020 at 14:53:58 UTC, aberba wrote:

 On Tuesday, 12 December 2017 at 20:51:30 UTC, Steven Schveighoffer wrote:

 On 12/11/17 6:33 PM, Seb wrote:

 [...]


 Since iopipe was mentioned several times, I will say a couple things:

 [...]



 I should really try iopipe this time round. I think I avoided toying 
 with it because the making conventions put me off. Don't remember 
 about the docs and available examples though.


 
 I too was trying to utilize iopipe and asked questions earlier this 
 year[1] and I made a file writer util[2]. I haven't really taken 
 advantage of the power yet, though it does appear that be a really nice 
 abstraction.
 
 1. https://forum.dlang.org/thread/faawejguebluwodflevh forum.dlang.org
 2. 
 https://gitlab.com/jessephillips/devarticlator/-/blob/maste
/source/util/file.d 
 


I reread my comment and in particular this:

"I was going to write an ascii art concept to show how the pushing 
works, but I think I'll maybe draw an actual picture. I need some time 
to accomplish this, though."

I still haven't done this. I need to make this happen, probably with a 
blog post (Mike, keep bugging me on this).

-Steve
Jun 19 2020
prev sibling parent reply Kagamin <spam here.lot> writes:
On Monday, 11 December 2017 at 23:33:44 UTC, Seb wrote:

 ---
 void main(string[] args)
 {
     import std.conv, std.range, std.stdio;
     foreach (d; 
 File(__FILE_FULL_PATH__).byChunk(4096).join.take(5)) {
         writefln("%s", d.to!char);
     }
 }
 ---


A variant: https://run.dlang.io/is/2TUQBv
Dec 14 2017
parent Kagamin <spam here.lot> writes:
Or this https://run.dlang.io/is/MO9Wiy
Dec 14 2017
prev sibling next sibling parent flamencofantasy <flamencofantasy gmail.com> writes:
On Monday, 11 December 2017 at 20:51:41 UTC, Jordi Gutiérrez 
Hermoso wrote:

 I'd like to read from a file, one byte at a time, without 
 loading the whole file in memory.

 I was hoping I could do something like

    auto f = File("somefile");
    foreach(c; f.byChar) {
        process(c);
    }

 but there appears to be no such way to do it anymore. Instead, 
 the stdlib seems to provide several functions to do chunked 
 reads from the file where I have to manually manage the buffer. 
 I see that D1 had a stream, but it's no longer here and I 
 understand ranges are supposed to be used instead.

 What's the explanation here? Why is there no more stream and 
 what am I supposed to use instead? Do I really need to be 
 manually managing the read buffer myself?


This should work;

  scope f = new MmFile("somefile");
    foreach(c; cast(string)f[]) {
        process(c);
    }
Dec 11 2017
prev sibling parent reply codephantom <me noyb.com> writes:
On Monday, 11 December 2017 at 20:51:41 UTC, Jordi Gutiérrez 
Hermoso wrote:

 I'd like to read from a file, one byte at a time, without 
 loading the whole file in memory.


just playing around with this....

// --------------------

module test;

import std.stdio, std.file, std.exception;

void main()
{
     string filename = "test.txt";
     enforce(filename.exists, "Umm..that file does not exist!");

     auto file = File(filename, "r");
     char[] charBuf;

     while (!file.eof())
     {
         charBuf = file.rawRead(new char[1]);

         if(!file.eof())
             process(cast(char)(charBuf[0]));
     }

     return;
}

void process(char someChar)
{
     import std.ascii : isPrintable;

     if( isPrintable(someChar) )
         writeln("found a printable character: ", someChar);
     else
         writeln("found a non printable character");

}
// --------------------
Dec 11 2017
parent codephantom <me noyb.com> writes:
On Tuesday, 12 December 2017 at 02:15:13 UTC, codephantom wrote:

 just playing around with this....


also...in case you only want to read n bytes..

// -----------------------

module test;

import std.stdio, std.file, std.exception;
import std.datetime.stopwatch;


void main()
{
     string filename = "test.txt";  // a text file
     //string filename = "test.exe"; // a binary file

     enforce(filename.exists, "Umm..that file does not exist!");

     auto file = File(filename, "r");
     ubyte[] buf;

     import std.datetime : MonoTime;
     auto t2 = MonoTime.currTime;

     // just read the first n bytes.
     int bytesToRead = 4; // change this n
     int bufCount;
     while ( !file.eof() && bufCount < bytesToRead )
     {
         buf = file.rawRead(new ubyte[1]);

         if(!file.eof())
         {
             process(cast(char)(buf[0]));
             bufCount++;
         }
     }


     writeln("-------------------------------------");
     writeln("this took : ", MonoTime.currTime - t2);
     writeln("-------------------------------------");
     writeln();

     return;
}

void process(char someChar)
{
     import std.ascii : isPrintable;

     if( isPrintable(someChar) )
         writeln("found a printable character: ", someChar);
     else
         writeln("found a non printable character");

}
// -----------------------
Dec 11 2017