www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - phobos: What type to use instead of File when doing I/O on streams?

reply J.Frank <joe cerberon.net> writes:
Hello again,

I'm a bit puzzled by the "File" type. Looking at the 
implementation, it seems that all I/O functions were stuffed into 
a single class^H^H^H^H^Hstruct.
What I expected to find is some kind of "Stream" class (with 
read(), write(), eof()), which is extended by a "File" class 
(with seek(), mmap(), etc.).

So, assuming my program reads from stdin and is supposed to work 
on a file as well as on a pipe (which is not seekable) - how can 
I make the compiler bark when I accidently use stdin.seek()?

Am I missing something here, too?
Nov 08 2015
next sibling parent reply Alex Parrill <initrd.gz gmail.com> writes:
On Sunday, 8 November 2015 at 20:47:08 UTC, J.Frank wrote:

 Hello again,

 I'm a bit puzzled by the "File" type. Looking at the 
 implementation, it seems that all I/O functions were stuffed 
 into a single class^H^H^H^H^Hstruct.
 What I expected to find is some kind of "Stream" class (with 
 read(), write(), eof()), which is extended by a "File" class 
 (with seek(), mmap(), etc.).

 So, assuming my program reads from stdin and is supposed to 
 work on a file as well as on a pipe (which is not seekable) - 
 how can I make the compiler bark when I accidently use 
 stdin.seek()?

 Am I missing something here, too?


Use ubyte/char input ranges. Post what you're trying to do if you 
want an example.
Nov 08 2015
parent J.Frank <joe cerberon.net> writes:
On Sunday, 8 November 2015 at 21:57:55 UTC, Alex Parrill wrote:

 Post what you're trying to do if you want an example.


$ cat test.d
import std.stdio;

//void foo(Stream in_stream, Stream out_stream) // I want 
something like this
void foo(File in_stream, File out_stream)
{
         in_stream.seek(3); // BUG

         string line;
         while ((line = in_stream.readln()) !is null)
                 out_stream.write(line);
}

void main(string[] args)
{
         foo(stdin, stdout);
}
$ cat test.txt
Line 0
Line 1
Line 2
$ gdc -o test test.d
$ ./test <test.txt
e 0
Line 1
Line 2
$ cat test.txt | ./test
std.exception.ErrnoException ../../../src/libphobos/std/stdio.d(595): Could not
seek in file `' (Illegal seek)
$
Nov 08 2015
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 8 November 2015 at 20:47:08 UTC, J.Frank wrote:

 So, assuming my program reads from stdin and is supposed to 
 work on a file as well as on a pipe (which is not seekable) - 
 how can I make the compiler bark when I accidently use 
 stdin.seek()?


You don't, in general. stdin is sometimes seekable and the 
compiler doesn't know if it is or not until you try at runtime.

You could write a wrapper struct for File though that  disables 
the seek function. It would have a File member, alias file this;, 
a constructor that forwards to it (optionally, you could also 
construct it as a File and assign it), and then a  disabled 
function with the same signature as the seek call in File. Then, 
it will no longer compile as long as you use your wrapper.
Nov 08 2015
parent reply J.Frank <joe cerberon.net> writes:
On Sunday, 8 November 2015 at 23:50:58 UTC, Adam D. Ruppe wrote:

 You don't, in general. stdin is sometimes seekable and the 
 compiler doesn't know if it is or not until you try at runtime.


Hm. "Maybe the stream is seekable, maybe it is not" is not really 
an option for a language that is supposed to be type safe.

I just found std.stream which looks very good. But it is 
deprecated??


 You could write a wrapper struct for File though that  disables 
 the seek function. It would have a File member, alias file 
 this;, a constructor that forwards to it (optionally, you could 
 also construct it as a File and assign it), and then a 
  disabled function with the same signature as the seek call in 
 File. Then, it will no longer compile as long as you use your 
 wrapper.


Thank you for your solution, but this sounds more lika a hack.
Nov 09 2015
next sibling parent reply J.Frank <joe cerberon.net> writes:
My question is now:
How can I make this work without using deprecated stuff?


import std.cstream;

void foo(InputStream in_stream, OutputStream out_stream)
{
//	in_stream.seek(3); // compile error - good :)

	char[] line;
	while ((line = in_stream.readLine()) !is null)
		out_stream.writeLine(line);
}

void main(string[] args)
{
	foo(din, dout);
}
Nov 09 2015
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 11/09/2015 01:48 AM, J.Frank wrote:

 My question is now:
 How can I make this work without using deprecated stuff?


 import std.cstream;

 void foo(InputStream in_stream, OutputStream out_stream)
 {
 //    in_stream.seek(3); // compile error - good :)

      char[] line;
      while ((line = in_stream.readLine()) !is null)
          out_stream.writeLine(line);
 }

 void main(string[] args)
 {
      foo(din, dout);
 }


You don't need the template constraints but it is good practice:

import std.stdio;
import std.range;

void foo(I, O)(I in_stream, O out_stream)
         if (isInputRange!I &&
             isOutputRange!(O, ElementType!I)) {

     // in_stream.seek(3); // compile error - good :)

     foreach (element; in_stream) {
         out_stream.put(element);
     }
}

void main(string[] args)
{
     // Also consider .byLine, which is faster and risky
     foo(stdin.byLineCopy,
         stdout.lockingTextWriter);
}

Ali
Nov 09 2015
parent reply J.Frank <joe cerberon.net> writes:
On Monday, 9 November 2015 at 14:48:35 UTC, Ali Çehreli wrote:

 import std.stdio;
 import std.range;

 void foo(I, O)(I in_stream, O out_stream)
         if (isInputRange!I &&
             isOutputRange!(O, ElementType!I)) {

     // in_stream.seek(3); // compile error - good :)

     foreach (element; in_stream) {
         out_stream.put(element);
     }
 }

 void main(string[] args)
 {
     // Also consider .byLine, which is faster and risky
     foo(stdin.byLineCopy,
         stdout.lockingTextWriter);
 }

 Ali


Uhm... no. That's not a solution to my problem. Ranges are not 
(I/O-)streams.
Nov 09 2015
next sibling parent reply Alex Parrill <initrd.gz gmail.com> writes:
On Monday, 9 November 2015 at 18:18:19 UTC, J.Frank wrote:

 On Monday, 9 November 2015 at 14:48:35 UTC, Ali Çehreli wrote:

 import std.stdio;
 import std.range;

 void foo(I, O)(I in_stream, O out_stream)
         if (isInputRange!I &&
             isOutputRange!(O, ElementType!I)) {

     // in_stream.seek(3); // compile error - good :)

     foreach (element; in_stream) {
         out_stream.put(element);
     }
 }

 void main(string[] args)
 {
     // Also consider .byLine, which is faster and risky
     foo(stdin.byLineCopy,
         stdout.lockingTextWriter);
 }

 Ali


 Uhm... no. That's not a solution to my problem. Ranges are not 
 (I/O-)streams.


Ranges are streams. file.byLine(Copy) and byChunk are effectively 
streams that are ranges.

The only problem with implementing your code in ranges is that 
`splitter` requires a forward range, which file-backed ranges 
aren't. Otherwise you could do 
`file.byChunk(4096).joiner.drop(3).splitter('\n')` to get a 
"stream" of lines.
Nov 09 2015
parent reply J.Frank <joe cerberon.net> writes:
On Monday, 9 November 2015 at 18:44:00 UTC, Alex Parrill wrote:

 Ranges are streams. file.byLine(Copy) and byChunk are 
 effectively streams that are ranges.


I might be wrong, but from what I read so far I don't think that 
"ranges are streams":

- Can you read an arbitrary length of bytes from a range? 
(Reading by line in my code was just an example.) The only way I 
see is to make it a byte range. But then reading n bytes would 
result in n calls to popFront(), which is out of the question for 
performance reasons. Is this correct?

- Can you flush() a range?

- Can you use select() on a range?

etc.

(BTW. Where do I find select()/poll() in phobos?)
Nov 09 2015
next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 9 November 2015 at 19:42:53 UTC, J.Frank wrote:

 (BTW. Where do I find select()/poll() in phobos?)


They are in `core.sys.posix.sys.select`

In general, OS headers for Posix in C like "sys/select.h" can be 
gotten by "import core.sys.posix.sys.select;" <unistd.h> is in 
"core.sys.posix.unistd"

Linux-specific ones like sys/epoll.h are in 
`core.sys.linux.sys.epoll`

and so on
Nov 09 2015
prev sibling next sibling parent Alex Parrill <initrd.gz gmail.com> writes:
On Monday, 9 November 2015 at 19:42:53 UTC, J.Frank wrote:

 On Monday, 9 November 2015 at 18:44:00 UTC, Alex Parrill wrote:

 Ranges are streams. file.byLine(Copy) and byChunk are 
 effectively streams that are ranges.


 I might be wrong, but from what I read so far I don't think 
 that "ranges are streams":

 - Can you read an arbitrary length of bytes from a range? 
 (Reading by line in my code was just an example.) The only way 
 I see is to make it a byte range. But then reading n bytes 
 would result in n calls to popFront(), which is out of the 
 question for performance reasons. Is this correct?


`myrange.take(array_size).array`

front/popFront are very good candidates for optimization; the 
compiler should be able to inline them, removing all of the 
overhead (GCD and LDC will likely be better at this than DMD, 
though).


 - Can you flush() a range?

 - Can you use select() on a range?


No, but you can't do either to anything other than file 
descriptors anyway (at least on Linux, dunno about Windows), so 
you may as well pass in a File.
Nov 09 2015
prev sibling parent Kagamin <spam here.lot> writes:
On Monday, 9 November 2015 at 19:42:53 UTC, J.Frank wrote:

 - Can you flush() a range?

 - Can you use select() on a range?


Maybe you should factor out a function that does pure data 
processing on arbitrary ranges and manage sources of the ranges - 
opening, flushing and closing files - in the caller code?
Nov 10 2015
prev sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 11/09/2015 10:18 AM, J.Frank wrote:

 On Monday, 9 November 2015 at 14:48:35 UTC, Ali Çehreli wrote:

 import std.stdio;
 import std.range;

 void foo(I, O)(I in_stream, O out_stream)
         if (isInputRange!I &&
             isOutputRange!(O, ElementType!I)) {

     // in_stream.seek(3); // compile error - good :)

     foreach (element; in_stream) {
         out_stream.put(element);
     }
 }

 void main(string[] args)
 {
     // Also consider .byLine, which is faster and risky
     foo(stdin.byLineCopy,
         stdout.lockingTextWriter);
 }

 Ali


 Uhm... no. That's not a solution to my problem. Ranges are not
 (I/O-)streams.


As far as I understand, the issue is to prevent the seek() call at 
compile time, right? If so, the range types that byLine() and 
byLineCopy() return does not have such a member function.

I think the code achieves that goal because we are not dealing with File 
objects anymore.

Ali
Nov 09 2015
parent reply J.Frank <joe cerberon.net> writes:
On Monday, 9 November 2015 at 19:38:06 UTC, Ali Çehreli wrote:

 As far as I understand, the issue is to prevent the seek() call 
 at compile time, right? If so, the range types that byLine() 
 and byLineCopy() return does not have such a member function.

 I think the code achieves that goal because we are not dealing 
 with File objects anymore.


Yes and no. Again, seek() was just an example.

I'm looking for a replacement for the deprecated inferfaces 
InputStream and OutputStream.
Nov 09 2015
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 9 November 2015 at 19:49:15 UTC, J.Frank wrote:

 I'm looking for a replacement for the deprecated inferfaces 
 InputStream and OutputStream.


Just keep using them. The deprecated note isn't really 
important... they will still be in Phobos for another year and 
you can copy the file out yourself to keep using it beyond that.

As of right now, there is no exact official replacement.
Nov 09 2015
prev sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 9 November 2015 at 08:49:08 UTC, J.Frank wrote:

 Hm. "Maybe the stream is seekable, maybe it is not" is not 
 really an option for a language that is supposed to be type 
 safe.


It just isn't known at compile time. Files, especially on Unix, 
are interchangeable at compile time but offer different 
capabilities at run time.


 Thank you for your solution, but this sounds more lika a hack.


Wrapper structs with alias this are basically how D does 
inheritance on a struct.
Nov 09 2015