www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Streams - Need a little help here

reply Arcane Jill <Arcane_member pathlink.com> writes:
Hi,

I need to determine in advance the number of bytes I can read from a stream
without that stream blocking. Or, equivalently, receive a yes/no answer to the
question "if I attempt to read n bytes from this stream, will it block?"

Is there any way of doing that with a std.stream.Stream ? (I don't mind if the
answer is "Not yet, but I'll add that one at the next release").

Failing that, can anyone think of any other way of achieving this?

All offers of advice appreciated.
Jill
Jun 05 2004
parent reply Kevin Bealer <Kevin_member pathlink.com> writes:
In article <c9tqup$9cg$1 digitaldaemon.com>, Arcane Jill says...

Hi,

I need to determine in advance the number of bytes I can read from a stream
without that stream blocking. Or, equivalently, receive a yes/no answer to the
question "if I attempt to read n bytes from this stream, will it block?"

Is there any way of doing that with a std.stream.Stream ? (I don't mind if the
answer is "Not yet, but I'll add that one at the next release").

Failing that, can anyone think of any other way of achieving this?

All offers of advice appreciated.
Jill


I haven't tried this, but could you create a seperate thread that did a read,
and once it got data, it could (under synchronization) put the results in a
buffer?  Then your "main" thread could (under synch) check the buffer object.

You would still be doing a blocking read, but the main thread could stay out of
the way.  Assuming that the blocking doesn't block the whole D process.

I would think you could also use the "select" or "poll" system calls if you are
in a unix-like environment.

Kevin
Jun 05 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <c9u86n$r8r$1 digitaldaemon.com>, Kevin Bealer says...

I haven't tried this, but could you create a seperate thread that did a read,
and once it got data, it could (under synchronization) put the results in a
buffer?  Then your "main" thread could (under synch) check the buffer object.


A feasable workaround, though I'd hope it would only be temporary. Thanks for
the idea.



I would think you could also use the "select" or "poll" system calls if you are
in a unix-like environment.


Ah, well you see, that's a problem. I like to write code that is as platform
independent as possible. As it happens, I'm using Windows, but even if I
weren't, it would irritate the hell out of me to have to write Linux-only code.

And in actual fact, that wouldn't work anyway - even on Linux. The reason is
that a Stream doesn't have a filehandle. This is because not all Streams are
built around files or sockets. A Stream is a generic concept. It might simply
iterate through an array. Such a stream would have to block until more stuff got
appended to the array. I need a solution which will work for ALL Streams, and
ideally on all platforms.


Let me restate the question, because my need is actually less specific than I
originally asked. All I *ACTUALLY* want is a yes/no answer to the following
question: WILL THIS STREAM BLOCK ON THE NEXT CALL TO readBlock(...)?

There appears to be no way to find out under the existing API, but it strikes me
it ought to be quite easy for the Phobos folk to add an API function to Stream
which answered that very question. (Of course, it would have to be implemented
by all subclasses of Stream, which might be a bit of a headache).

In any case, I suspect that, in time, I won't be the only person who needs to do
this.

Arcane Jill
Jun 06 2004
next sibling parent Charlie <Charlie_member pathlink.com> writes:
I would think you could also use the "select" or "poll" system calls if you are
in a unix-like environment.




These work in win32 also.  


In any case, I suspect that, in time, I won't be the only person who needs to do
this.


This sounds like a great idea, good call!  .I think trying to convince Ben (H)
is your best bet ;).

C

In article <c9usbu$1mcd$1 digitaldaemon.com>, Arcane Jill says...

In article <c9u86n$r8r$1 digitaldaemon.com>, Kevin Bealer says...

I haven't tried this, but could you create a seperate thread that did a read,
and once it got data, it could (under synchronization) put the results in a
buffer?  Then your "main" thread could (under synch) check the buffer object.


A feasable workaround, though I'd hope it would only be temporary. Thanks for
the idea.



I would think you could also use the "select" or "poll" system calls if you are
in a unix-like environment.


Ah, well you see, that's a problem. I like to write code that is as platform
independent as possible. As it happens, I'm using Windows, but even if I
weren't, it would irritate the hell out of me to have to write Linux-only code.

And in actual fact, that wouldn't work anyway - even on Linux. The reason is
that a Stream doesn't have a filehandle. This is because not all Streams are
built around files or sockets. A Stream is a generic concept. It might simply
iterate through an array. Such a stream would have to block until more stuff got
appended to the array. I need a solution which will work for ALL Streams, and
ideally on all platforms.


Let me restate the question, because my need is actually less specific than I
originally asked. All I *ACTUALLY* want is a yes/no answer to the following
question: WILL THIS STREAM BLOCK ON THE NEXT CALL TO readBlock(...)?

There appears to be no way to find out under the existing API, but it strikes me
it ought to be quite easy for the Phobos folk to add an API function to Stream
which answered that very question. (Of course, it would have to be implemented
by all subclasses of Stream, which might be a bit of a headache).

In any case, I suspect that, in time, I won't be the only person who needs to do
this.

Arcane Jill
Jun 06 2004
prev sibling parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
[snip]

 Let me restate the question, because my need is actually less specific


than I

 originally asked. All I *ACTUALLY* want is a yes/no answer to the


following

 question: WILL THIS STREAM BLOCK ON THE NEXT CALL TO readBlock(...)?


In some sense readBlock by definition blocks until the data is read, so the
question should be "will the next readBlock block for more than xxx seconds"
which is a hard question to answer in any case.
Another possible solution would be to wrap the stream in a BufferedStream
and add a small function to BufferedStream that returns true if the buffer
contains xxx bytes from the current position.
Adding a general "will this block" function would be nice but I honestly
don't know what it would do to answer yes or no.


 There appears to be no way to find out under the existing API, but it


strikes me

 it ought to be quite easy for the Phobos folk to add an API function to


Stream

 which answered that very question. (Of course, it would have to be


implemented

 by all subclasses of Stream, which might be a bit of a headache).

 In any case, I suspect that, in time, I won't be the only person who needs


to do

 this.

 Arcane Jill
Jun 07 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <ca2don$r85$1 digitaldaemon.com>, Ben Hinkle says...

In some sense readBlock by definition blocks until the data is read, so the
question should be "will the next readBlock block for more than xxx seconds"


Not so. I want to know "Will the next readBlock block AT ALL?". As in, in zero
seconds.


which is a hard question to answer in any case.


I think it's not only easy, but /trivially/ easy. I hope to explain in the
course of this post.


Another possible solution would be to wrap the stream in a BufferedStream


I think I'm in the wrong language. Java has BufferedInputStream. D does not. Or
at least, Phobos does not. Of course it would be easy for me to write my own. I
could even add it to Deimos.


and add a small function to BufferedStream that returns true if the buffer
contains xxx bytes from the current position.


To be honest, I had rather assumed that the class std.stream.File *WAS*
buffered. Are you telling me it isn't? It's important that we know these things,
because lots of read()s from an unbuffered stream is horribly inefficient, but
adding buffering to an already buffered stream is ALSO horribly inefficient.


Adding a general "will this block" function would be nice but I honestly
don't know what it would do to answer yes or no.


I do. Java, as you probably know, has the abstract class InputStream. Now,
InputStream has exactly this funtion. It's called available(). It is documented
as, and I quote, "Returns the number of bytes that can be read (or skipped over)
from this input stream without blocking by the next caller of a method for this
input stream." Knowing HOW MANY bytes you can read is, of course, even better
than knowing whether or not you will be able to read just one.

Implementation is child's play. In std.stream.Stream, you do this:


       abstract ulong available();


In general, a non-buffered subclass would do this:


       override ulong available() { return 0; }


In any buffered or filtering subclass, you do this:


       override ulong available()
       {
           return numBytesRemainingInBufer + s.available();
       }


(where s is the underlying Stream - the one being buffered). In the special case
of std.stream.MemoryStream, you return the number of bytes to the end of the
array (although arguably this is a special case of buffering).

If Java can do this, D should be able to do it to. We're supposed to be trying
to make D /better/ than Java, not inferior to it in basic functionality. The
above suggested implementation, plus the addition to Phobos (or Deimos) of the
class BufferedStream, would do the job nicely.

Arcane Jill
Jun 07 2004
parent reply Sean Kelly <sean f4.ca> writes:
In article <ca3h7h$2lll$1 digitaldaemon.com>, Arcane Jill says...

In article <ca2don$r85$1 digitaldaemon.com>, Ben Hinkle says...

Another possible solution would be to wrap the stream in a BufferedStream


I think I'm in the wrong language. Java has BufferedInputStream. D does not. Or
at least, Phobos does not. Of course it would be easy for me to write my own. I
could even add it to Deimos.


Just user BufferedStream like Ben says.  I don't think it's mentioned in the
docs, but the class does exist.


To be honest, I had rather assumed that the class std.stream.File *WAS*
buffered. Are you telling me it isn't? It's important that we know these things,
because lots of read()s from an unbuffered stream is horribly inefficient, but
adding buffering to an already buffered stream is ALSO horribly inefficient.


None of the base classes are buffered.  There is a BufferedStream class that
adds buffering if you want it.


Sean
Jun 07 2004
next sibling parent reply Brad Anderson <brad sankaty.dot.com> writes:
Might I suggest Mango.io instead of Phobos modules?  I've been pretty 
happy with this part of the Mango tree, even though there's a nice http 
client, server, and servlet container there as well.

I'm not sure about a count of how many characters you have left to read 
w/o blocking, but Kris is pretty good with avoiding frivolous memory 
allocations and inefficient code in general.  (Live up to that, Mr. Bell)

BA

http://mango.dsource.org or http://www.dsource.org/projects/mango




Sean Kelly wrote:

 In article <ca3h7h$2lll$1 digitaldaemon.com>, Arcane Jill says...
 

In article <ca2don$r85$1 digitaldaemon.com>, Ben Hinkle says...


Another possible solution would be to wrap the stream in a BufferedStream


I think I'm in the wrong language. Java has BufferedInputStream. D does not. Or
at least, Phobos does not. Of course it would be easy for me to write my own. I
could even add it to Deimos.


 
 
 Just user BufferedStream like Ben says.  I don't think it's mentioned in the
 docs, but the class does exist.
 
 

To be honest, I had rather assumed that the class std.stream.File *WAS*
buffered. Are you telling me it isn't? It's important that we know these things,
because lots of read()s from an unbuffered stream is horribly inefficient, but
adding buffering to an already buffered stream is ALSO horribly inefficient.


 
 
 None of the base classes are buffered.  There is a BufferedStream class that
 adds buffering if you want it.
 
 
 Sean
Jun 07 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <ca3ia2$2nf5$1 digitaldaemon.com>, Brad Anderson says...

Might I suggest Mango.io instead of Phobos modules?  I've been pretty 
happy with this part of the Mango tree, even though there's a nice http 
client, server, and servlet container there as well.


Everyone on this newsgroup is really nice, and I really do appreciate all these
offers of help from people. (And I will look at mango as well). But, alas, you
have also completely misunderstood my needs.

I want a function equivalent to Java's java.io.InputSream.available() added to
D's std.stream.Stream() and its subclasses. [Or alternatively, a willBlock()
function]. Nothing less will suffice.

If it can't be done, then it can't be done. End of story. Then I just move on to
something else.

Arcane Jill
Jun 07 2004
parent reply "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
Jill,

What you're asking for is kinda' tricky to place in a class such as stream
(even though the Java lib has it). I think this boils down to a classic
producer/consumer issue, where the consumer wants to know if there's
anything ready for, uhhh, consumption ... traditional approaches place the
producer on a separate thread and use a synch-point, rendezvous, or exchange
mechanism to synchronize the pair; with a time-out period provided by the
consumer. That is, the consumer says "I'll wait X microseconds for something
to become available; then I'm outta here". This may have been what Ben
Hinkle was suggesting.

Perhaps Ben and/or Mike could advise you on the finer points within the
Concurrent library, but that's the place to go; Doug Lea seems to have spent
a lifetime perfecting the library that those two guys are converting. At
this point the Concurrent project does not have Rendezvous, but does have
Exchanger. The latter could be used to build a nice wee wrapper around a
producer thread that suits your purposes ...

- Kris


"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:ca3jo6$2pl3$1 digitaldaemon.com...

 In article <ca3ia2$2nf5$1 digitaldaemon.com>, Brad Anderson says...

Might I suggest Mango.io instead of Phobos modules?  I've been pretty
happy with this part of the Mango tree, even though there's a nice http
client, server, and servlet container there as well.


 Everyone on this newsgroup is really nice, and I really do appreciate all


these

 offers of help from people. (And I will look at mango as well). But, alas,


you

 have also completely misunderstood my needs.

 I want a function equivalent to Java's java.io.InputSream.available()


added to

 D's std.stream.Stream() and its subclasses. [Or alternatively, a


willBlock()

 function]. Nothing less will suffice.

 If it can't be done, then it can't be done. End of story. Then I just move


on to

 something else.

 Arcane Jill
Jun 08 2004
next sibling parent Sean Kelly <sean f4.ca> writes:
In article <ca4p2n$1smd$1 digitaldaemon.com>, Kris says...

Jill,

What you're asking for is kinda' tricky to place in a class such as stream
(even though the Java lib has it). I think this boils down to a classic
producer/consumer issue, where the consumer wants to know if there's
anything ready for, uhhh, consumption ... traditional approaches place the
producer on a separate thread and use a synch-point, rendezvous, or exchange
mechanism to synchronize the pair; with a time-out period provided by the
consumer. That is, the consumer says "I'll wait X microseconds for something
to become available; then I'm outta here". This may have been what Ben
Hinkle was suggesting.

Perhaps Ben and/or Mike could advise you on the finer points within the
Concurrent library, but that's the place to go; Doug Lea seems to have spent
a lifetime perfecting the library that those two guys are converting. At
this point the Concurrent project does not have Rendezvous, but does have
Exchanger. The latter could be used to build a nice wee wrapper around a
producer thread that suits your purposes ...


I think what Jill needs at the moment is far more simple.  A buffer always knows
how many bytes it has stored.  But for some things the producer/consumer model
is certainly the way to go.  I've got a multiplexed socket library I'm
considering porting to D, though I may decide on a redesign based on the
unfinished Boost api.  Either way, that's on my to do list.  Though it may need
to wait until Ben finishes with the Concurrent lib port.

Sean
Jun 08 2004
prev sibling parent "Ben Hinkle" <bhinkle mathworks.com> writes:
 Perhaps Ben and/or Mike could advise you on the finer points within the
 Concurrent library, but that's the place to go; Doug Lea seems to have


spent

 a lifetime perfecting the library that those two guys are converting. At
 this point the Concurrent project does not have Rendezvous, but does have
 Exchanger. The latter could be used to build a nice wee wrapper around a
 producer thread that suits your purposes ...


It's a little embarrassing but I've switched code-bases for that library.
Mike and I were porting Doug's original library, which contained Rendezvous,
andeverything was fine. When I emailed Doug he strongly suggested using the
newer JSR166 code which had Exchanger but no Rendezvous. When I initially
looked at the two libraries I thought they were essentially the same and so
I went with the stand-alone version but once Doug suggested going with the
newer code I took another look and realized the styles are actually pretty
different. For instance the old code was entirely based on the built-in
synchronized/wait/notify but the new stuff implements the synchronization
totally independently (and depending on an undocumented Sun package called
sun.misc.unsafe).

So to make a long story short I ripped out the Rendezvous and put in
Exchanger. Same goes for the other basic stuff. Doug indicated somewhere the
"more esoteric" classes like Rendezvous might get released in an optional
package sometime and when that happens we can port it. But for the time
being it's gonna be JSR166. The channel and fork/join code will probably
keep using the old code but we're looking into it.

-Ben
Jun 08 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <ca3hvs$2mse$1 digitaldaemon.com>, Sean Kelly says...


Just user BufferedStream like Ben says.  I don't think it's mentioned in the
docs, but the class does exist.


Thanks for the info that D has BufferedStream. I didn't know that, and it really
should be in the docs.

But as to the advice that I use it, I believe you have completely misunderstood
my needs. I guess that doesn't matter though - thanks for trying to help anyway.

Jill
Jun 07 2004
parent Sean Kelly <sean f4.ca> writes:
In article <ca3jd3$2p74$1 digitaldaemon.com>, Arcane Jill says...

In article <ca3hvs$2mse$1 digitaldaemon.com>, Sean Kelly says...


Just user BufferedStream like Ben says.  I don't think it's mentioned in the
docs, but the class does exist.


Thanks for the info that D has BufferedStream. I didn't know that, and it really
should be in the docs.

But as to the advice that I use it, I believe you have completely misunderstood
my needs. I guess that doesn't matter though - thanks for trying to help anyway.


BufferedStream doesn't have a method to return how many bytes are available in
the buffer, which is what I think you were asking for, but it wouldn't be too
hard to add (even as a temporary hack for your own uses).  I just started
looking at streams yesterday to see what they might be missing, so assuming my
submissions are accepted perhaps the wait won't be too long.  In this case,
perhaps soemthing along the lines of std::istream::readsome added to
BufferedStream would suit.  


Sean
Jun 08 2004