www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - iopipe v0.1.0 - now with Windows support!

reply Steven Schveighoffer <schveiguy yahoo.com> writes:
iopipe version 0.1.0 has been released.

iopipe is a high-performance pipe processing system that makes it easy 
to string together pipelines to process data with as little buffer 
copying as possible.

Nothing has really been changed, but it now has Windows i/o support. I 
will note at this time, however, that ring buffers are not yet supported 
on Windows.

This version deprecates the IODev type that I had included, in favor of 
a new io library that shows extreme promise.

This version ONLY builds on 2.080.1 or later (the bug fix that I 
submitted at dconf has been merged in that version, and so iopipe will 
now build against Martin Nowak's io library). In fact, iopipe 
development was kind of stalled due to this bug, so I'm super-happy to 
see it fixed and released!

Note that the new io library also supports sockets, which IODev did not 
have support for, AND has a pluggable driver system, so you could 
potentially use fiber-based async io without rebuilding. It just makes a 
lot of sense for D to have a standard low-level io library that 
everything can use without having to kludge together multiple types of 
io libraries.

Near future plans:

1. Utilize a CI to make sure it continues to work on all platforms.
2. Add RingBuffer support on Windows
3. Add more driver support for std.io.
4. Continue development of JSON library that depends on iopipe (not yet 
on code.dlang.org).

git - https://github.com/schveiguy/iopipe
dub - https://code.dlang.org/packages/iopipe
docs - http://schveiguy.github.io/iopipe/

-Steve
Jun 10 2018
next sibling parent reply Dejan Lekic <dejan.lekic gmail.com> writes:
On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer 
wrote:

 iopipe version 0.1.0 has been released.

 iopipe is a high-performance pipe processing system that makes 
 it easy to string together pipelines to process data with as 
 little buffer copying as possible.


All I can say (again, like I repeated on IRC many times) is that 
iopipe should be in the current form, or another, in Phobos. I 
just love it!
Jun 11 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/11/18 6:14 AM, Dejan Lekic wrote:

 On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote:

 iopipe version 0.1.0 has been released.

 iopipe is a high-performance pipe processing system that makes it easy 
 to string together pipelines to process data with as little buffer 
 copying as possible.


 
 All I can say (again, like I repeated on IRC many times) is that iopipe 
 should be in the current form, or another, in Phobos. I just love it!


Thanks for the kind words! If Martin's std.io makes it in to Phobos, 
there is a realistic chance iopipe could go as well.

Just curious, do you have any projects based on iopipe?

-Steve
Jun 11 2018
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/10/18 4:10 PM, Steven Schveighoffer wrote:


 Nothing has really been changed, but it now has Windows i/o support. I 
 will note at this time, however, that ring buffers are not yet supported 
 on Windows.
 


I just pushed v0.1.1 -- I realized that I never *actually* compiled on 
windows, and there were a couple things that didn't work.

Note: the examples still don't work as they rely on openDev, which is 
only available on Posix systems now.

I need to figure out a good way to open stdin/stdout in a cross platform 
way with std.io.

-Steve
Jun 11 2018
parent reply Jacob Carlborg <doob me.com> writes:
On 2018-06-11 16:45, Steven Schveighoffer wrote:


 I just pushed v0.1.1 -- I realized that I never *actually* compiled on
 windows, and there were a couple things that didn't work.

 Note: the examples still don't work as they rely on openDev, which is
 only available on Posix systems now.

 I need to figure out a good way to open stdin/stdout in a cross platform
 way with std.io.


You should setup AppVeyor [1] to make it works on Windows (when it works).

[1] https://www.appveyor.com

-- 
/Jacob Carlborg
Jun 19 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/19/18 7:18 AM, Jacob Carlborg wrote:

 On 2018-06-11 16:45, Steven Schveighoffer wrote:
 

 I just pushed v0.1.1 -- I realized that I never *actually* compiled on
 windows, and there were a couple things that didn't work.

 Note: the examples still don't work as they rely on openDev, which is
 only available on Posix systems now.

 I need to figure out a good way to open stdin/stdout in a cross platform
 way with std.io.


 
 You should setup AppVeyor [1] to make it works on Windows (when it works).
 
 [1] https://www.appveyor.com
 


I just set up travis to do the Linux/mac testing. I need to add appveyor 
as well, but haven't gotten to it. I'm a complete CI noob, so I'm 
learning slowly :)

-Steve
Jun 19 2018
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2018-06-19 15:04, Steven Schveighoffer wrote:


 I just set up travis to do the Linux/mac testing. I need to add appveyor
 as well, but haven't gotten to it. I'm a complete CI noob, so I'm
 learning slowly :)


To save you some trouble, AppVeyor supports both a YAML, like Travis, 
and a web UI to configure the CI system. If you're not including some 
parts of the YAML file, like "build_script", it will use the default, 
which is preforming some Visual Studio specific task. You also need to 
download the D compiler manually since AppVeyor doesn't have built-in 
support for D the same way as Travis.

-- 
/Jacob Carlborg
Jun 21 2018
prev sibling parent Seb <seb wilzba.ch> writes:
On Tuesday, 19 June 2018 at 13:04:24 UTC, Steven Schveighoffer 
wrote:

 On 6/19/18 7:18 AM, Jacob Carlborg wrote:

 On 2018-06-11 16:45, Steven Schveighoffer wrote:
 

 I just pushed v0.1.1 -- I realized that I never *actually* 
 compiled on
 windows, and there were a couple things that didn't work.

 Note: the examples still don't work as they rely on openDev, 
 which is
 only available on Posix systems now.

 I need to figure out a good way to open stdin/stdout in a 
 cross platform
 way with std.io.


 
 You should setup AppVeyor [1] to make it works on Windows 
 (when it works).
 
 [1] https://www.appveyor.com
 


 I just set up travis to do the Linux/mac testing. I need to add 
 appveyor as well, but haven't gotten to it. I'm a complete CI 
 noob, so I'm learning slowly :)

 -Steve


Just copy/paste one of the existing configuration scripts and you 
should be good to go, e.g.

https://github.com/dlang-community/DCD/blob/master/appveyor.yml

And there's also:

https://github.com/Abscissa/AppVeyor-D
Jun 21 2018
prev sibling next sibling parent reply DigitalDesigns <DigitalDesigns gmail.com> writes:
On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer 
wrote:

 iopipe version 0.1.0 has been released.

 iopipe is a high-performance pipe processing system that makes 
 it easy to string together pipelines to process data with as 
 little buffer copying as possible.

 Nothing has really been changed, but it now has Windows i/o 
 support. I will note at this time, however, that ring buffers 
 are not yet supported on Windows.

 This version deprecates the IODev type that I had included, in 
 favor of a new io library that shows extreme promise.

 This version ONLY builds on 2.080.1 or later (the bug fix that 
 I submitted at dconf has been merged in that version, and so 
 iopipe will now build against Martin Nowak's io library). In 
 fact, iopipe development was kind of stalled due to this bug, 
 so I'm super-happy to see it fixed and released!

 Note that the new io library also supports sockets, which IODev 
 did not have support for, AND has a pluggable driver system, so 
 you could potentially use fiber-based async io without 
 rebuilding. It just makes a lot of sense for D to have a 
 standard low-level io library that everything can use without 
 having to kludge together multiple types of io libraries.

 Near future plans:

 1. Utilize a CI to make sure it continues to work on all 
 platforms.
 2. Add RingBuffer support on Windows
 3. Add more driver support for std.io.
 4. Continue development of JSON library that depends on iopipe 
 (not yet on code.dlang.org).

 git - https://github.com/schveiguy/iopipe
 dub - https://code.dlang.org/packages/iopipe
 docs - http://schveiguy.github.io/iopipe/

 -Steve


Could you explain some benefits specific to this implementation 
and a bit of the functional aspects for a proper overview of it's 
capabilities and why I should chose this method over others?
Jun 11 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/12/18 1:51 AM, DigitalDesigns wrote:

 
 Could you explain some benefits specific to this implementation and a 
 bit of the functional aspects for a proper overview of it's capabilities 
 and why I should chose this method over others?


The things that I think make this approach better are:

1. Direct buffer access

Direct buffer access I have found is one of those ideas that doesn't 
seem like it's that impressive until you start using it. Many times, 
buffering is used in a generic library for optimization (basically 
amortizing reads) and is an implementation detail hidden from your view. 
Think of how FILE * keeps a large buffer of data inside itself, but only 
gives you access to one character at a time.

This forces you to create your *own* buffering scheme on top of that. 
What a waste! Iopipe allows you to use buffering for your purposes on 
top of the benefits of amortization. It's my belief that this is why 
iopipe's byline feature is 2x faster than Phobos'.

2. Using templates to their fullest

Iopipes are all templated on the buffer or iopipe underneath it. This 
makes tings easily swappable. It's really cool to be able to take your 
JSON or XML parser, and hook it onto an in-memory string in one line, 
and then hook it onto a socket, and everything is optimized for that 
situation. It takes the fun and flexibility of range programming and 
brings it to i/o.

This is why iopipe's byline handles all forms of UTF, compared to Phobos 
which only handles UTF8.

For example, I handle all forms of UTF with iopipe, with a decent set of 
utilities. Here is a complete program using iopipe that converts any 
type of UTF into another type, optimized for the specific situation:

https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d

3. Compiler optimization for everything

All parts of iopipe, except for the low-level reads and writes (which 
ironically are not really part of iopipe) are visible to the compiler 
for inlining and optimization. I'm leveraging the power of the decades 
of optimization experience that the compiler can provide. This makes it 
easy to write code that performs well.

An anecdote: For my talk on iopipe in 2017 
(http://dconf.org/2017/talks/schveighoffer.html) I wanted to have a live 
demo showing the performance power. I literally was still working on it 
2 or 3 days before, while at dconf. I had already written a JSON parser, 
which was part of my presentation, but when I was showing it to another 
D user (Daniel Murphy), I couldn't really answer the question "how does 
it perform?". So he gave me a challenge -- do pretty printing on a JSON 
file. Simple enough, with the parser I had already written, took me 
about 1 hour to write it. It performed poorly compared to what we would 
have expected, but tweaking a few things (almost all were due to using 
some algorithms incorrectly), I got it to go faster than RapidJson in 
certain use cases, and reasonably close in others. And I did nothing in 
terms of lookup tables or using any special instructions. All in all, it 
was probably 2 hours of work, and the code is beautiful IMO! 
https://github.com/schveiguy/jsoniopipe/blob/master/examples/formatjson/formatjson.d

I think anyone who is doing parsing should have a look at iopipe, it not 
only may make your code much simpler and easier to read and write, but 
it would help me tune iopipe to cater to parsers, which I think is its 
wheelhouse.

I plan to eventually finish the JSON parser for a releasable state, and 
eventually tackle XML and a few other things.

-Steve
Jun 12 2018
next sibling parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
 I plan to eventually finish the JSON parser for a releasable 
 state, and eventually tackle XML and a few other things.

 -Steve


You should definitely tackle xml by branching dxml. I'm really 
liking the api.
Jun 12 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/12/18 10:19 AM, Jesse Phillips wrote:

 I plan to eventually finish the JSON parser for a releasable state, 
 and eventually tackle XML and a few other things.


 
 You should definitely tackle xml by branching dxml. I'm really liking 
 the api.


Of course that is my plan! I would never want to have to build an xml 
parser from scratch :)

-Steve
Jun 12 2018
prev sibling parent DigitalDesigns <DigitalDesigns gmail.com> writes:
On Tuesday, 12 June 2018 at 13:51:56 UTC, Steven Schveighoffer 
wrote:

 On 6/12/18 1:51 AM, DigitalDesigns wrote:

 
 Could you explain some benefits specific to this 
 implementation and a bit of the functional aspects for a 
 proper overview of it's capabilities and why I should chose 
 this method over others?


 The things that I think make this approach better are:

 1. Direct buffer access

 Direct buffer access I have found is one of those ideas that 
 doesn't seem like it's that impressive until you start using 
 it. Many times, buffering is used in a generic library for 
 optimization (basically amortizing reads) and is an 
 implementation detail hidden from your view. Think of how FILE 
 * keeps a large buffer of data inside itself, but only gives 
 you access to one character at a time.

 This forces you to create your *own* buffering scheme on top of 
 that. What a waste! Iopipe allows you to use buffering for your 
 purposes on top of the benefits of amortization. It's my belief 
 that this is why iopipe's byline feature is 2x faster than 
 Phobos'.

 2. Using templates to their fullest

 Iopipes are all templated on the buffer or iopipe underneath 
 it. This makes tings easily swappable. It's really cool to be 
 able to take your JSON or XML parser, and hook it onto an 
 in-memory string in one line, and then hook it onto a socket, 
 and everything is optimized for that situation. It takes the 
 fun and flexibility of range programming and brings it to i/o.

 This is why iopipe's byline handles all forms of UTF, compared 
 to Phobos which only handles UTF8.

 For example, I handle all forms of UTF with iopipe, with a 
 decent set of utilities. Here is a complete program using 
 iopipe that converts any type of UTF into another type, 
 optimized for the specific situation:

 https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d

 3. Compiler optimization for everything

 All parts of iopipe, except for the low-level reads and writes 
 (which ironically are not really part of iopipe) are visible to 
 the compiler for inlining and optimization. I'm leveraging the 
 power of the decades of optimization experience that the 
 compiler can provide. This makes it easy to write code that 
 performs well.

 An anecdote: For my talk on iopipe in 2017 
 (http://dconf.org/2017/talks/schveighoffer.html) I wanted to 
 have a live demo showing the performance power. I literally was 
 still working on it 2 or 3 days before, while at dconf. I had 
 already written a JSON parser, which was part of my 
 presentation, but when I was showing it to another D user 
 (Daniel Murphy), I couldn't really answer the question "how 
 does it perform?". So he gave me a challenge -- do pretty 
 printing on a JSON file. Simple enough, with the parser I had 
 already written, took me about 1 hour to write it. It performed 
 poorly compared to what we would have expected, but tweaking a 
 few things (almost all were due to using some algorithms 
 incorrectly), I got it to go faster than RapidJson in certain 
 use cases, and reasonably close in others. And I did nothing in 
 terms of lookup tables or using any special instructions. All 
 in all, it was probably 2 hours of work, and the code is 
 beautiful IMO! 
 https://github.com/schveiguy/jsoniopipe/blob/master/examples/formatjson/formatjson.d

 I think anyone who is doing parsing should have a look at 
 iopipe, it not only may make your code much simpler and easier 
 to read and write, but it would help me tune iopipe to cater to 
 parsers, which I think is its wheelhouse.

 I plan to eventually finish the JSON parser for a releasable 
 state, and eventually tackle XML and a few other things.

 -Steve


Thanks!
Jun 12 2018
prev sibling next sibling parent reply Anton Fediushin <fediushin.anton yandex.ru> writes:
On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer 
wrote:

 iopipe version 0.1.0 has been released.

 iopipe is a high-performance pipe processing system that makes 
 it easy to string together pipelines to process data with as 
 little buffer copying as possible.


I saw iopipe a while back, but never looked at it closely. Now I 
did and... it implements its own kind of pattern and not ranges? 
I guess it is done for the sake of performance, but how easy it 
is to use iopipe with standard range-based methods from 
std.algorithm for example?

As long as it's easy to use with the rest of the phobos - I'd 
like to see it in the standard library.


 This version ONLY builds on 2.080.1 or later (the bug fix that 
 I submitted at dconf has been merged in that version, and so 
 iopipe will now build against Martin Nowak's io library). In 
 fact, iopipe development was kind of stalled due to this bug, 
 so I'm super-happy to see it fixed and released!


These are great news, keep it up!
Jun 12 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/12/18 3:08 AM, Anton Fediushin wrote:

 On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer wrote:

 iopipe version 0.1.0 has been released.

 iopipe is a high-performance pipe processing system that makes it easy 
 to string together pipelines to process data with as little buffer 
 copying as possible.


 
 I saw iopipe a while back, but never looked at it closely. Now I did 
 and... it implements its own kind of pattern and not ranges?


Correct, although it's very similar to ranges.


 I guess it 
 is done for the sake of performance, but how easy it is to use iopipe 
 with standard range-based methods from std.algorithm for example?


Very simple. You just have to define what is an "element" of a range 
that is a sliding window of data.

For example: 
https://github.com/schveiguy/iopipe/blob/master/source/iopipe/textpipe.d#L542

Note that asInputRange simply treats the current window as "front", and 
"popFront" discards that window and loads the next. It's a crude but 
effective tool to convert any iopipe into a range.

The reason iopipe is not based on ranges exactly (the window must be a 
random-access range), is because ranges don't handle i/o very 
performantly. E.g. I would never use lockingTextReader to process text 
data, it would be slow as hell, and too limiting.

File.byLine is fast, but only because of the underlying non-range tricks 
it uses to achieve performance. And iopipe still is 2x faster.


 As long as it's easy to use with the rest of the phobos - I'd like to 
 see it in the standard library.


It should be easy to use on its own, and with algorithms from phobos. 
I've done so in some of the toy parsers I've written.

I think the one sticking point (which I'm not sure how to reconcile, but 
we can probably hash it out) is that for iopipes, char and wchar arrays 
are arrays, not auto-decoding ranges.

-Steve
Jun 12 2018
parent Anton Fediushin <fediushin.anton yandex.ru> writes:
On Tuesday, 12 June 2018 at 14:00:32 UTC, Steven Schveighoffer 
wrote:

 File.byLine is fast, but only because of the underlying 
 non-range tricks it uses to achieve performance. And iopipe 
 still is 2x faster.


I wish iopipe was around a little bit earlier so I could use it 
in my small project. It doesn't use IO much, just reads large 
file (60GB+) and computes some hashes. Now I'd like to rewrite it 
using iopipe just to compare the performance.
Jun 13 2018
prev sibling next sibling parent reply bauss <jj_1337 live.dk> writes:
On Sunday, 10 June 2018 at 20:10:31 UTC, Steven Schveighoffer 
wrote:

 -Steve


Does iopipe work with CTFE?
Jun 13 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/13/18 8:35 AM, bauss wrote:

 Does iopipe work with CTFE?


It may work in some cases. Some of the things it does are not conducive 
to CTFE working well -- like using a buffer. But generally at compile 
time, you don't want to use a buffer.

But I would expect, for instance, using algorithms and iopipes on a 
string would probably work.

hm... just thought I'd try it:

     import std.range : walkLength;
     static 
assert("hello\nworld\nthis\nis\na\ntest".byLineRange.walkLength == 6);

It didn't work at first, as I'm using memchr for searching for the 
newlines, but with a nice little if(__ctfe) override, it works just great!

I'm going to push this (I'll do some tests for the other widths to make 
sure it works for all UTF), but if you have any more things you want to 
work at CTFE, submit some issues on the github project.

I don't expect a lot of the array casting stuff to work, but maybe there 
are ways. I actually never thought much about making it work at 
compile-time. I suppose though, it would be cool to import a file at 
compile time, and generate the JSON or XML DOM at compile time too :)

I think zip and unzip are never going to work since the underlying C 
calls are not CTFE-able.

-Steve
Jun 13 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/13/18 12:03 PM, Steven Schveighoffer wrote:


 I'm going to push this (I'll do some tests for the other widths to make 
 sure it works for all UTF), but if you have any more things you want to 
 work at CTFE, submit some issues on the github project.


v0.1.2 released

-Steve
Jun 13 2018
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 06/10/2018 10:10 PM, Steven Schveighoffer wrote:

 Note that the new io library also supports sockets, which IODev did not
 have support for, AND has a pluggable driver system, so you could
 potentially use fiber-based async io without rebuilding. It just makes a
 lot of sense for D to have a standard low-level io library that
 everything can use without having to kludge together multiple types of
 io libraries.


Note that the WIP std.io library is fully  nogc  safe, so it's a bit
edgy on using latest features. Soon want to move to use DIP10008 instead
of preallocated exceptions.
With that and  nogc in the Driver interface¹ it's still to be seen
whether we can adapt this well with vibe.d or need to adjust the
low-level design.

-Martin

¹: https://martinnowak.github.io/io/std/io/driver/Driver.html
Jun 16 2018
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Sunday, 17 June 2018 at 04:52:07 UTC, Martin Nowak wrote:

 On 06/10/2018 10:10 PM, Steven Schveighoffer wrote:

 Note that the new io library also supports sockets, which 
 IODev did not have support for, AND has a pluggable driver 
 system, so you could potentially use fiber-based async io 
 without rebuilding. It just makes a lot of sense for D to have 
 a standard low-level io library that everything can use 
 without having to kludge together multiple types of io 
 libraries.


 Note that the WIP std.io library is fully  nogc  safe, so it's 
 a bit
 edgy on using latest features. Soon want to move to use 
 DIP10008 instead
 of preallocated exceptions.


This is very encouraging. I’d like to see it working well with 
Photon (though my time is very limited atm). Any thoughts on what 
set of syscalls I need to support?

Maybe I could just provide my own “native” driver that fits your 
concept of I/O driver in io library.


 With that and  nogc in the Driver interface¹ it's still to be 
 seen
 whether we can adapt this well with vibe.d or need to adjust the
 low-level design.

 -Martin

 ¹: https://martinnowak.github.io/io/std/io/driver/Driver.html
Jun 17 2018