www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - An IO Streams Library

reply Jason White <54f9byee3t32 gmail.com> writes:
I see the subject of IO streams brought up here occasionally. The 
general consensus seems to be that we need something better than 
what Phobos provides.

I wrote a library "io" that can work as a replacement for 
std.stdio, std.mmfile, std.cstream, and parts of std.stream:

     GitHub:  https://github.com/jasonwhite/io
     Package: https://code.dlang.org/packages/io

This library provides an input and output range interface for 
streams (which is more efficient if the stream is buffered). 
Thus, many of the wonderful range operations from std.range and 
std.algorithm can be used with this.

I'm interested in feedback on this library. What is it missing? 
How can be better?

I'm also interested in a discussion of what IO-related 
functionality people are missing in Phobos.

Please destroy!
Feb 06 2016
next sibling parent reply cym13 <cpicard openmailbox.org> writes:
On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:

 I see the subject of IO streams brought up here occasionally. 
 The general consensus seems to be that we need something better 
 than what Phobos provides.
 [...]


 From what I can see without testing it, very nice work, thanks!

More a little surprise than anything serious though, why did you 
choose to go with "println" instead of "writeln" and such? I find 
it more confusing than anything given phobos choice.
Feb 06 2016
parent Jason White <54f9byee3t32 gmail.com> writes:
On Sunday, 7 February 2016 at 01:01:21 UTC, cym13 wrote:

 From what I can see without testing it, very nice work, thanks!

 More a little surprise than anything serious though, why did 
 you choose to go with "println" instead of "writeln" and such? 
 I find it more confusing than anything given phobos choice.


Thanks!

There are a couple reasons for using print/println/etc. over 
write/writeln/etc.:
  1. A module-level definition of write(Stream s, ...) would clash 
with the stream's definition of write(...).
  2. Do we mean text-serialization or byte-for-byte output when we 
say write()? With print(), it's clear that we want the arguments 
to be converted to a text representation and have that written 
that to the stream. With write(), it's clear we're writing out 
the binary representation to the stream.
Feb 06 2016
prev sibling next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 07/02/16 1:48 PM, Jason White wrote:

 I see the subject of IO streams brought up here occasionally. The
 general consensus seems to be that we need something better than what
 Phobos provides.

 I wrote a library "io" that can work as a replacement for std.stdio,
 std.mmfile, std.cstream, and parts of std.stream:

      GitHub:  https://github.com/jasonwhite/io
      Package: https://code.dlang.org/packages/io

 This library provides an input and output range interface for streams
 (which is more efficient if the stream is buffered). Thus, many of the
 wonderful range operations from std.range and std.algorithm can be used
 with this.

 I'm interested in feedback on this library. What is it missing? How can
 be better?

 I'm also interested in a discussion of what IO-related functionality
 people are missing in Phobos.

 Please destroy!


I posted a link to your repo a couple days ago in IRC.
Honestly? I like it. It looks reasonably well made.

There is a bit of work regarding interfaces + ranges.
I.e. Sink really should be inheriting from OutputRange!ubyte

Its no where near Phobos quality and that is ok for now.
I do think given time it could be a reasonably good base to rework 
std.socket, std.stdio, std.stream, std.cstream and std.mmfile into a 
completely new set of modules.

Most of that code it would end up replacing is I think almost 10 years 
old either way its from D1 and I think we can do better.
Feb 06 2016
parent reply Jason White <54f9byee3t32 gmail.com> writes:
On Sunday, 7 February 2016 at 01:20:26 UTC, Rikki Cattermole 
wrote:

 I posted a link to your repo a couple days ago in IRC.
 Honestly? I like it. It looks reasonably well made.


Thanks. I saw a link to it in a recent thread in Learn. I figured 
I'd finally make a proper post on it.


 There is a bit of work regarding interfaces + ranges.
 I.e. Sink really should be inheriting from OutputRange!ubyte


I haven't had much use for the interfaces, which is why they 
aren't fleshed out. Do you have any particular use cases for this 
in mind?


 Its no where near Phobos quality and that is ok for now.


I agree. The documentation needs work and I imagine there are 
lots of use cases that aren't well supported. Increased 
visibility and usage definitely helps with finding the warts.


 I do think given time it could be a reasonably good base to 
 rework std.socket, std.stdio, std.stream, std.cstream and 
 std.mmfile into a completely new set of modules.

 Most of that code it would end up replacing is I think almost 
 10 years old either way its from D1 and I think we can do 
 better.
Feb 06 2016
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 07/02/16 2:55 PM, Jason White wrote:

 On Sunday, 7 February 2016 at 01:20:26 UTC, Rikki Cattermole wrote:

 I posted a link to your repo a couple days ago in IRC.
 Honestly? I like it. It looks reasonably well made.


 Thanks. I saw a link to it in a recent thread in Learn. I figured I'd
 finally make a proper post on it.


 There is a bit of work regarding interfaces + ranges.
 I.e. Sink really should be inheriting from OutputRange!ubyte


 I haven't had much use for the interfaces, which is why they aren't
 fleshed out. Do you have any particular use cases for this in mind?


I have no use case other then range compatibility.


 Its no where near Phobos quality and that is ok for now.


 I agree. The documentation needs work and I imagine there are lots of
 use cases that aren't well supported. Increased visibility and usage
 definitely helps with finding the warts.


Actually I think there are plenty of use cases not implemented.
Done properly as a full replacement and rework of Phobos will mean you 
need to do almost everything in e.g. std.stdio and std.socket but with 
better abstractions.

Of course your goal may not be inline with my assertions of reworking 
Phobos. So feel free to ignore, it just would be a shame since it really 
needs some love.
Feb 06 2016
parent reply Jason White <54f9byee3t32 gmail.com> writes:
On Sunday, 7 February 2016 at 01:59:43 UTC, Rikki Cattermole 
wrote:

 Actually I think there are plenty of use cases not implemented.
 Done properly as a full replacement and rework of Phobos will 
 mean you need to do almost everything in e.g. std.stdio and 
 std.socket but with better abstractions.


I think I'll tackle implementing sockets next. I might need that 
for another project of mine.

Once this gets polished enough, it would be great to eventually 
replace those modules in Phobos. However, it would be difficult 
to do this without compatibility breakages. For example, since 
std.stdio.File uses FILE* under the covers and this uses plain 
old file descriptors, programs that rely on that behavior would 
break.


 Of course your goal may not be inline with my assertions of 
 reworking Phobos. So feel free to ignore, it just would be a 
 shame since it really needs some love.


My primary goal is to provide a more useful and powerful IO 
library than what Phobos provides since that is what I need for 
my other projects. That goal is not necessarily counter to 
reworking Phobos. ;)
Feb 06 2016
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 07/02/16 3:43 PM, Jason White wrote:

 On Sunday, 7 February 2016 at 01:59:43 UTC, Rikki Cattermole wrote:

 Actually I think there are plenty of use cases not implemented.
 Done properly as a full replacement and rework of Phobos will mean you
 need to do almost everything in e.g. std.stdio and std.socket but with
 better abstractions.


 I think I'll tackle implementing sockets next. I might need that for
 another project of mine.


I wouldn't actually implement it based upon std.socket.
Use libasync instead.
https://github.com/etcimon/libasync

There has been talk about getting that into Phobos but it still needs 
time to mature.

One other important thing to note about sockets.
For anything performance related you need to have a central way to 
implement an event loop.

The one I've implemented is designed to work as a replacement for 
libasync's and to be used in Phobos.
https://github.com/rikkimax/alphaPhobos/blob/master/source/std/experimental/platform.d

If you do intend to make it compatible, you can ignore all of the 
windowing and related methods. Just keep things like optimizedEventLoop, 
eventLoopIteration, setAsDefault and thePlatform, defaultPlatform all in 
there.

We can combine later on, I just want it to be compatible when it comes time.


 Once this gets polished enough, it would be great to eventually replace
 those modules in Phobos. However, it would be difficult to do this
 without compatibility breakages. For example, since std.stdio.File uses
 FILE* under the covers and this uses plain old file descriptors,
 programs that rely on that behavior would break.


There is nothing wrong with breakage. Its old code. Its time to update.
But there must be a clear upgrade path.


 Of course your goal may not be inline with my assertions of reworking
 Phobos. So feel free to ignore, it just would be a shame since it
 really needs some love.


 My primary goal is to provide a more useful and powerful IO library than
 what Phobos provides since that is what I need for my other projects.
 That goal is not necessarily counter to reworking Phobos. ;)


Okay sweet. Also if you can, it would be great to see you on IRC.
Feb 06 2016
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 07 Feb 2016 00:48:54 +0000
schrieb Jason White <54f9byee3t32 gmail.com>:


 I see the subject of IO streams brought up here occasionally. The 
 general consensus seems to be that we need something better than 
 what Phobos provides.
 
 I wrote a library "io" that can work as a replacement for 
 std.stdio, std.mmfile, std.cstream, and parts of std.stream:
 
      GitHub:  https://github.com/jasonwhite/io
      Package: https://code.dlang.org/packages/io
 
 This library provides an input and output range interface for 
 streams (which is more efficient if the stream is buffered). 
 Thus, many of the wonderful range operations from std.range and 
 std.algorithm can be used with this.
 
 I'm interested in feedback on this library. What is it missing? 
 How can be better?
 
 I'm also interested in a discussion of what IO-related 
 functionality people are missing in Phobos.
 
 Please destroy!


I saw this on code.dlang.org some time ago and had a quick look. First
of all this would have to go into phobos to make sure it's used as some
kind of a standard. Conflicting stream libraries would only cause more
trouble.

Then if you want to go for phobos inclusion I'd recommend looking at
other stream implementations and learning from their mistakes ;-)
There's
https://github.com/schveiguy/phobos/tree/babe9fe338f03cafc0fb50fc0d37ea96505da3e3/std/io
which was supposed to be a stream replacement for phobos. Then there
are also vibe.d streams*.


Your Stream interfaces looks like standard stream implementations (which
is a good thing) which also work for unbuffered streams. I think it's a
good idea to support partial reads and writes. For an explanation why
partial reads, see the vibe.d rant below. Partial writes are useful
as a write syscall can be interrupted by posix signals to stop the
write. I'm not sure if the API should expose this feature (e.g. by
returning a partial write on EINTR) but it can sometimes be useful.
Still readExactly / writeAll helpers functions are useful. I would try
to implement these as UFCS functions instead of as a struct wrapper.

For some streams you'll need a TimeoutException. An interesting
question is whether users should be able to recover from
TimeoutExceptions. This essentially means if a read/write function
internally calls read/write posix calls more than once and only the
last one timed out, we already processed some data and it's not
possible to recover from a TimeoutException if the amount of already
processed data is unknown.
The simplest solution is using only one syscall internally. Then
TimeoutException => no data was processed. But this doesn't work for
read/writeExcatly (Another reason why read/writeExactly shouldn't be
the default. vibe.d...)

Regarding buffers / sliding windows I'd have a look at
https://github.com/schveiguy/phobos/blob/babe9fe338f03cafc0fb50fc0d37ea96505da3e3/std/io/buffer.d

Another design question is whether there should be an interface for
such buffered streams or whether it's OK to have only unbuffered
streams + one buffer struct / class. Basically the question is whether
there might be streams that can offer a buffer interface but can't  use
the standard implementation.




* vibe.d stream rant ahead:

vibe.d streams get some things right and some things very wrong. For
example their leastSize/empty/read combo means you might actually
have to implement reading data in any of these functions. Users have to
handle timeouts or other errors for any of these as well.

Then the API requires a buffered stream, it simply won't work for
unbuffered IO (leastSize, empty). And the fact that read reads exactly
n bytes makes stream implementations more complicated (re-reading until
enough data has been read should be done by a generic function, not
reimplemented in every stream). It even makes some user code more
complicated: I've implemented a serial port library for vibe-d.
If I don't know how many bytes will arrive with the next packet, the
read posix function usually returns the expected/available amount of
data. But now vibe.d requires me to specify a fixed length when calling
the stream read method. This leads to ugly code using peak...

Then vibe.d also mixes the sliding window / buffer concept into the
stream class, but does so in a bad way. A sliding window should expose
the internal buffer so that it's possible to consume bytes from the
buffer, skip bytes, refill... In vibe.d you can peak at the buffer. But
you can't discard data. You'll have to call read instead which copies
from the internal buffer to an external buffer, even if you only want
to skip data. Even worse, your external buffer size is limited. So you
have to implement some loop logic if you want to skip more data than
fits your buffer. And all you need is a discard(size_t n) function which
does _buffer = _buffer[n .. $] in the stream class...

TLDR: API design is very important.
Feb 07 2016
parent Jason White <54f9byee3t32 gmail.com> writes:
On Sunday, 7 February 2016 at 10:50:24 UTC, Johannes Pfau wrote:

 I saw this on code.dlang.org some time ago and had a quick 
 look. First of all this would have to go into phobos to make 
 sure it's used as some kind of a standard. Conflicting stream 
 libraries would only cause more trouble.

 Then if you want to go for phobos inclusion I'd recommend 
 looking at
 other stream implementations and learning from their mistakes 
 ;-)
 There's
 https://github.com/schveiguy/phobos/tree/babe9fe338f03cafc0fb50fc0d37ea96505da3e3/std/io
 which was supposed to be a stream replacement for phobos. Then 
 there
 are also vibe.d streams*.


I saw Steven's stream implementation quite some time ago and I 
had a look at vibe's stream implementation just now. I think it 
is a mistake to use classes over structs for this sort of thing. 
I briefly tried implementing it with classes, but ran into 
problems. The non-deterministic destruction of classes is 
probably the biggest issue. One has to be careful about calling 
f.close() in order to avoid accumulating too many open file 
descriptors in programs that open a lot of files. Reference 
counting takes care of this problem nicely and has less overhead. 
This is one area where classes relying on the GC is not ideal. 
Rust's ownership system solves this problem quite well. Python 
also solves this with "with" statements.


 Your Stream interfaces looks like standard stream 
 implementations (which
 is a good thing) which also work for unbuffered streams. I 
 think it's a
 good idea to support partial reads and writes. For an 
 explanation why
 partial reads, see the vibe.d rant below. Partial writes are 
 useful
 as a write syscall can be interrupted by posix signals to stop 
 the
 write. I'm not sure if the API should expose this feature (e.g. 
 by
 returning a partial write on EINTR) but it can sometimes be 
 useful.


I don't want to assume what the user wants to do in the event of 
an EINTR unless a certain behavior is desired 100% of the time. I 
don't think that is the case here. Thus, that is probably 
something the user should handle manually, if needed.


 Still readExactly / writeAll helpers functions are useful. I 
 would try
 to implement these as UFCS functions instead of as a struct 
 wrapper.


I agree. I went ahead and made that change.


 For some streams you'll need a TimeoutException. An interesting
 question is whether users should be able to recover from
 TimeoutExceptions. This essentially means if a read/write 
 function
 internally calls read/write posix calls more than once and only 
 the
 last one timed out, we already processed some data and it's not
 possible to recover from a TimeoutException if the amount of 
 already
 processed data is unknown.
 The simplest solution is using only one syscall internally. Then
 TimeoutException => no data was processed. But this doesn't 
 work for
 read/writeExcatly (Another reason why read/writeExactly 
 shouldn't be
 the default. vibe.d...)


In the current implementation of readExactly/writeExactly, one 
cannot assume how much was read or written in the event of an 
exception anyway. The only way around this I can see is to return 
the number of bytes read/written in the exception itself. In 
fact, that might solve the TimeoutException problem, too. Hmm...

I'd like to keep the fundamental read/write functions at just one 
system call each in order to guarantee that they are atomic in 
relation to each other.


 Regarding buffers / sliding windows I'd have a look at 
 https://github.com/schveiguy/phobos/blob/babe9fe338f03cafc0fb50fc0d37ea96505da3e3/std/io/buffer.d

 Another design question is whether there should be an interface 
 for such buffered streams or whether it's OK to have only 
 unbuffered streams + one buffer struct / class. Basically the 
 question is whether there might be streams that can offer a 
 buffer interface but can't  use the standard implementation.


I think it's OK to re-implement buffering for different types of 
streams where it is more efficient to do so. For example, there 
is no need to implement buffering for an in-memory stream 
because, by definition, it is already buffered.

I'm not sure if having multiple buffering strategies would be 
useful. Right now, there is only the fixed-sized sliding window. 
If multiple buffering strategies are useful, then it makes sense 
to have all streams unbuffered by default and have separate 
buffering implementations.

There is an interesting buffering approach here that is mainly 
geared towards parsing: 
https://github.com/DmitryOlshansky/datapicked/blob/master/dpick/buffer/buffer.d


 * vibe.d stream rant ahead:

 vibe.d streams get some things right and some things very 
 wrong. For
 example their leastSize/empty/read combo means you might 
 actually
 have to implement reading data in any of these functions. Users 
 have to
 handle timeouts or other errors for any of these as well.

 Then the API requires a buffered stream, it simply won't work 
 for
 unbuffered IO (leastSize, empty). And the fact that read reads 
 exactly
 n bytes makes stream implementations more complicated 
 (re-reading until
 enough data has been read should be done by a generic function, 
 not
 reimplemented in every stream). It even makes some user code 
 more
 complicated: I've implemented a serial port library for vibe-d.
 If I don't know how many bytes will arrive with the next 
 packet, the
 read posix function usually returns the expected/available 
 amount of
 data. But now vibe.d requires me to specify a fixed length when 
 calling
 the stream read method. This leads to ugly code using peak...

 Then vibe.d also mixes the sliding window / buffer concept into 
 the stream class, but does so in a bad way. A sliding window 
 should expose the internal buffer so that it's possible to 
 consume bytes from the buffer, skip bytes, refill... In vibe.d 
 you can peak at the buffer. But you can't discard data. You'll 
 have to call read instead which copies from the internal buffer 
 to an external buffer, even if you only want to skip data. Even 
 worse, your external buffer size is limited. So you have to 
 implement some loop logic if you want to skip more data than 
 fits your buffer. And all you need is a discard(size_t n) 
 function which does _buffer = _buffer[n .. $] in the stream 
 class...


These are the golden nuggets of experience I was looking for when 
making this post. They definitely help to guide an ergonomic API 
design. Standing on the shoulders of giants and such. Thanks!


 TLDR: API design is very important.


Completely agree.
Feb 07 2016
prev sibling next sibling parent reply Jakob Ovrum <jakobovrum gmail.com> writes:
On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:

 I'm interested in feedback on this library. What is it missing? 
 How can be better?


I like what I've seen so far, but I'd just like to note that it's 
easier to give feedback on the API when there is web 
documentation. GitHub Pages would be a natural place to host it.

A lot of D libraries on GitHub do this and not everyone uses the 
same tools, but for one example, here's LuaD[1] with reference 
documentation on GitHub pages, automatically generated and pushed 
by Travis-CI for the master branch.

https://github.com/JakobOvrum/LuaD
Feb 07 2016
parent Jason White <54f9byee3t32 gmail.com> writes:
On Monday, 8 February 2016 at 07:50:33 UTC, Jakob Ovrum wrote:

 I like what I've seen so far, but I'd just like to note that 
 it's easier to give feedback on the API when there is web 
 documentation. GitHub Pages would be a natural place to host it.

 A lot of D libraries on GitHub do this and not everyone uses 
 the same tools, but for one example, here's LuaD[1] with 
 reference documentation on GitHub pages, automatically 
 generated and pushed by Travis-CI for the master branch.

 https://github.com/JakobOvrum/LuaD


Done: https://jasonwhite.github.io/io/

(Nice work on bootDoc!)

I haven't tied it in with the Travis build yet, but that 
shouldn't be too hard to do.

Pretty docs typically helps motivate me to write more.
Feb 08 2016
prev sibling next sibling parent reply Dejan Lekic <dejan.lekic gmail.com> writes:
I think this is a huge task and requires a (huge) DIP, and 
collaborative effort of coming up with a good, really good, API 
for BOTH synchronous and asynchronous IO. As mentioned in the 
previous messages, there is already an asyncio library, although 
I am not sure it is good enough to be in Phobos...

I know people may not like Boost, but Boost.Asio is amazing and 
we should perhaps use it as a guide, if not copy the whole API...
Feb 08 2016
next sibling parent reply Jakob Ovrum <jakobovrum gmail.com> writes:
On Monday, 8 February 2016 at 09:08:53 UTC, Dejan Lekic wrote:

 I think this is a huge task and requires a (huge) DIP, and 
 collaborative effort of coming up with a good, really good, API 
 for BOTH synchronous and asynchronous IO. As mentioned in the 
 previous messages, there is already an asyncio library, 
 although I am not sure it is good enough to be in Phobos...


I don't think we've used DIPs for library additions before. I 
don't see what it would provide over module documentation.
Feb 08 2016
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Monday, 8 February 2016 at 09:12:25 UTC, Jakob Ovrum wrote:

 I don't think we've used DIPs for library additions before. I 
 don't see what it would provide over module documentation.


Probably because a standard library should keep things simple, 
meaning it should unify high level  functionality on aggregates 
as ranges/iterators or a generalization over streams/ranges that 
is as simple as possible across the board.

If not, then why have it in the standard library?
Feb 08 2016
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 8 February 2016 at 09:08:53 UTC, Dejan Lekic wrote:

 I think this is a huge task and requires a (huge) DIP, and 
 collaborative effort of coming up with a good, really good, API 
 for BOTH synchronous and asynchronous IO. As mentioned in the 
 previous messages, there is already an asyncio library, 
 although I am not sure it is good enough to be in Phobos...

 I know people may not like Boost, but Boost.Asio is amazing and 
 we should perhaps use it as a guide, if not copy the whole 
 API...


I like boost. Well, sometimes. I _severely_ dislike boost::asio. 
The new couroutines may make it easier to use, I'd have to try 
1st, but what's the point when vibe.d exists?

Atila
Feb 08 2016
parent reply Chris Wright <dhasenan gmail.com> writes:
On Mon, 08 Feb 2016 12:19:59 +0000, Atila Neves wrote:


 On Monday, 8 February 2016 at 09:08:53 UTC, Dejan Lekic wrote:

 I think this is a huge task and requires a (huge) DIP, and
 collaborative effort of coming up with a good, really good, API for
 BOTH synchronous and asynchronous IO. As mentioned in the previous
 messages, there is already an asyncio library, although I am not sure
 it is good enough to be in Phobos...

 I know people may not like Boost, but Boost.Asio is amazing and we
 should perhaps use it as a guide, if not copy the whole API...


 
 I like boost. Well, sometimes. I _severely_ dislike boost::asio.
 The new couroutines may make it easier to use, I'd have to try 1st, but
 what's the point when vibe.d exists?
 
 Atila


I can't use vibe.d for one of my projects because I need to keep tight 
control of coroutine scheduling. Better async IO routines in Phobos would 
help me -- though I'm mainly looking for higher level stuff, like a 
telnet handler.
Feb 08 2016
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 8 February 2016 at 17:11:56 UTC, Chris Wright wrote:

 On Mon, 08 Feb 2016 12:19:59 +0000, Atila Neves wrote:


 On Monday, 8 February 2016 at 09:08:53 UTC, Dejan Lekic wrote:

 [...]


 
 I like boost. Well, sometimes. I _severely_ dislike 
 boost::asio.
 The new couroutines may make it easier to use, I'd have to try 
 1st, but
 what's the point when vibe.d exists?
 
 Atila


 I can't use vibe.d for one of my projects because I need to 
 keep tight control of coroutine scheduling. Better async IO 
 routines in Phobos would help me -- though I'm mainly looking 
 for higher level stuff, like a telnet handler.


You can always use fibers yourself and just do the IO there. It's 
cooperative multithreading so you control when one of them yields.

Atila
Feb 08 2016
parent Chris Wright <dhasenan gmail.com> writes:
On Mon, 08 Feb 2016 18:52:52 +0000, Atila Neves wrote:


 On Monday, 8 February 2016 at 17:11:56 UTC, Chris Wright wrote:

 On Mon, 08 Feb 2016 12:19:59 +0000, Atila Neves wrote:


 On Monday, 8 February 2016 at 09:08:53 UTC, Dejan Lekic wrote:

 [...]


 
 I like boost. Well, sometimes. I _severely_ dislike boost::asio.
 The new couroutines may make it easier to use, I'd have to try 1st,
 but what's the point when vibe.d exists?
 
 Atila


 I can't use vibe.d for one of my projects because I need to keep tight
 control of coroutine scheduling. Better async IO routines in Phobos
 would help me -- though I'm mainly looking for higher level stuff, like
 a telnet handler.


 
 You can always use fibers yourself and just do the IO there. It's
 cooperative multithreading so you control when one of them yields.


Which is exactly what I'm doing now.

The point is, it's useful to have multiple types of concurrency handling 
with IO, and async IO that's likely to end up in Phobos doesn't 
necessarily obviate vibe.d or duplicate work.
Feb 08 2016
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:

     GitHub:  https://github.com/jasonwhite/io


In the output stream you compare output data length to the input 
data length. In case of a transcoding stream they can be 
different. Are you trying to account for partial writes?
Feb 08 2016
parent reply Jason White <54f9byee3t32 gmail.com> writes:
On Monday, 8 February 2016 at 12:02:08 UTC, Kagamin wrote:

 In the output stream you compare output data length to the 
 input data length. In case of a transcoding stream they can be 
 different. Are you trying to account for partial writes?


Reads and writes are not guaranteed to fill/write the entire 
buffer you throw at it. This is what readExactly/writeExactly are 
for. Those will throw an exception if the entire read/write 
cannot be done.
Feb 08 2016
parent reply Kagamin <spam here.lot> writes:
On Monday, 8 February 2016 at 20:21:31 UTC, Jason White wrote:

 Reads and writes are not guaranteed to fill/write the entire 
 buffer you throw at it. This is what readExactly/writeExactly 
 are for. Those will throw an exception if the entire read/write 
 cannot be done.


You mean posix non-blocking IO?
Feb 09 2016
parent Chris Wright <dhasenan gmail.com> writes:
On Tue, 09 Feb 2016 09:45:03 +0000, Kagamin wrote:


 On Monday, 8 February 2016 at 20:21:31 UTC, Jason White wrote:

 Reads and writes are not guaranteed to fill/write the entire buffer you
 throw at it. This is what readExactly/writeExactly are for. Those will
 throw an exception if the entire read/write cannot be done.


 
 You mean posix non-blocking IO?


No, Posix non-blocking IO will fill a buffer with all data that is 
currently available and tell you the number of bytes it managed to give 
you. And then it will set errno to EAGAIN.

readExactly will either fill the entire buffer or throw an exception 
saying that it was unable to do so.

It's using Posix functions underneath, so if it repeatedly calls them 
until it's sent the whole buffer (assuming EAGAIN is set), that saves me 
work sometimes (but I think phobos does that already).

But if it throws an exception even when errno is set to EAGAIN, that 
would be useful for writing to a file atomically.
Feb 09 2016
prev sibling next sibling parent Wyatt <wyatt.epp gmail.com> writes:
On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:

 This library provides an input and output range interface for 
 streams (which is more efficient if the stream is buffered). 
 Thus, many of the wonderful range operations from std.range and 
 std.algorithm can be used with this.


Ah, grand!  I love the idea and my impression from browsing the 
source a bit is positive enough to say I'm looking forward to 
what comes out of this.

Though I AM a little ambivalent-- I had a series of pretty 
in-depth conversations on this topic with a friend a while back 
and we came to a consensus that stream semantics are a tricky 
thing because of the historical baggage around them and how they 
tend to get conflated with other concepts.

Looking at your API design, I think you've hit close to a lot of 
the same conclusions we reached, but here are the notes I took 
for the sake of providing an additional perspective:
http://radiusic.com/doc/streamNotes (Sorry, I tried just pasting 
them and it was moderately unreadable even in the preview)

I think the most important things we hit upon are:
1. A stream is fundamentally unidirectional.
2. A stream is raw, untyped data that becomes a range through an 
adapter that mediates access.

-Wyatt
Feb 08 2016
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:

 I'm also interested in a discussion of what IO-related 
 functionality people are missing in Phobos.


This is still a very interesting approach that could even become 
a candidate for std.io at some point. Would be great if we could 
get a few more people behind this, for sure there are already 
plenty of D socket implementations out there.

https://github.com/jasonwhite/io/pull/3#issuecomment-234775601
Jul 24 2016
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:

 I'm interested in feedback on this library. What is it missing? 
 How can be better?


I think making the actual read/write/open/accept/et.al. functions 
used to talk to the kernel pluggable would be a good extension 
point to hook in evented/threaded IO frameworks.
This restricts the async integration to synchronous APIs, e.g. 
suspendable Fibers or async-await, but that might be a reasonable 
choice over nodejs-style inversion of control.
Jul 24 2016
parent Johannes Pfau <nospam example.com> writes:
Am Sun, 24 Jul 2016 13:03:01 +0000
schrieb Martin Nowak <code dawg.eu>:


 On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:

 I'm interested in feedback on this library. What is it missing? 
 How can be better?  


 
 I think making the actual read/write/open/accept/et.al. functions 
 used to talk to the kernel pluggable would be a good extension 
 point to hook in evented/threaded IO frameworks.
 This restricts the async integration to synchronous APIs, e.g. 
 suspendable Fibers or async-await, but that might be a reasonable 
 choice over nodejs-style inversion of control.
 


Can you please elaborate why this is useful? As streams already provide
a source-independent way to process data it should always possible to
just implement another stream to wrap low level calls. So why add
another API layer? And if the API slightly varies (e.g. different open
flags) implementing another wrapper is much simpler than having one
wrapper dealing with different low level functions.
Jul 26 2016
prev sibling parent reply ikod <geller.garry gmail.com> writes:
On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:

 I see the subject of IO streams brought up here occasionally. 
 The general consensus seems to be that we need something better 
 than what Phobos provides.

 I wrote a library "io" that can work as a replacement for 
 std.stdio, std.mmfile, std.cstream, and parts of std.stream:

     GitHub:  https://github.com/jasonwhite/io
     Package: https://code.dlang.org/packages/io

 This library provides an input and output range interface for 
 streams (which is more efficient if the stream is buffered). 
 Thus, many of the wonderful range operations from std.range and 
 std.algorithm can be used with this.

 I'm interested in feedback on this library. What is it missing? 
 How can be better?

 I'm also interested in a discussion of what IO-related 
 functionality people are missing in Phobos.

 Please destroy!


Hello,

I don't know if it is good practice or not, but sometimes it make 
life easier if you can put part of the data back into the input 
stream.
Jul 25 2016
parent reply Johannes Pfau <nospam example.com> writes:
Am Mon, 25 Jul 2016 13:10:42 +0000
schrieb ikod <geller.garry gmail.com>:


 On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:

 I see the subject of IO streams brought up here occasionally. 
 The general consensus seems to be that we need something better 
 than what Phobos provides.

 I wrote a library "io" that can work as a replacement for 
 std.stdio, std.mmfile, std.cstream, and parts of std.stream:

     GitHub:  https://github.com/jasonwhite/io
     Package: https://code.dlang.org/packages/io

 This library provides an input and output range interface for 
 streams (which is more efficient if the stream is buffered). 
 Thus, many of the wonderful range operations from std.range and 
 std.algorithm can be used with this.

 I'm interested in feedback on this library. What is it missing? 
 How can be better?

 I'm also interested in a discussion of what IO-related 
 functionality people are missing in Phobos.

 Please destroy!  


 
 Hello,
 
 I don't know if it is good practice or not, but sometimes it make 
 life easier if you can put part of the data back into the input 
 stream.
 


Writing data back to a stream is quite uncommon. The standard way to
solve such problems is a peek method for buffered streams:

auto buf = stream.peek(length)
// You can now look at the data in buf
stream.read() will still return the data read by peek, no need to write
data back into the stream.
Jul 26 2016
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 26.07.2016 um 16:50 schrieb Johannes Pfau:

 Am Mon, 25 Jul 2016 13:10:42 +0000

 Hello,

 I don't know if it is good practice or not, but sometimes it make
 life easier if you can put part of the data back into the input
 stream.


 Writing data back to a stream is quite uncommon. The standard way to
 solve such problems is a peek method for buffered streams:

 auto buf = stream.peek(length)
 // You can now look at the data in buf
 stream.read() will still return the data read by peek, no need to write
 data back into the stream.


With the notable exception of ungetc() for C's file streams. But working 
on the byte level is something that ranges should be used for in D's 
case and not streams, because the latter tend to have a high call 
overhead. So such a feature could make sense for a 
StreamInputRange/InputStreamRange wrapper.
Jul 26 2016
parent ikod <geller.garry gmail.com> writes:
On Wednesday, 27 July 2016 at 06:18:07 UTC, Sönke Ludwig wrote:

 Am 26.07.2016 um 16:50 schrieb Johannes Pfau:

 Am Mon, 25 Jul 2016 13:10:42 +0000

 Hello,

 I don't know if it is good practice or not, but sometimes it 
 make
 life easier if you can put part of the data back into the 
 input
 stream.


 Writing data back to a stream is quite uncommon. The standard 
 way to
 solve such problems is a peek method for buffered streams:

 auto buf = stream.peek(length)
 // You can now look at the data in buf
 stream.read() will still return the data read by peek, no need 
 to write
 data back into the stream.


 With the notable exception of ungetc() for C's file streams. 
 But working on the byte level is something that ranges should 
 be used for in D's case and not streams, because the latter 
 tend to have a high call overhead. So such a feature could make 
 sense for a StreamInputRange/InputStreamRange wrapper.


Yes, and here is simple use-case for this: you have to process 
input stream of data chunks mixed with separators from the 
socket. To read only data and not separators, you can either ask 
stream code to "read until separator", or you can read as much as 
possible, then split on separator, return split-tail back to 
stream and process split-head. Now all your reads from stream 
will either start from the new data chunk or continue reading 
from it.

Of course, application code can manage all this on its own, but 
why not to help application programmer keep his code clean?
Jul 27 2016