www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - My stream concept

reply Regan Heath <regan netwin.co.nz> writes:
Typical stream implementations I have seen use the following heirarchy:

class Stream {
}
class FileStream : Stream {
}
class SocketStream : Stream {
}

in other words you have to design/derive a class from Stream for each type 
of Stream you want to implement.

I always thought this was kinda backwards, and limited, in fact after 
Arcane Jill said something along the lines of "anything that reads and 
writes bytes _is_ a stream" I got to thinking surely I can use templates 
to make that statement true.


The way I have done it is to define some interfaces:

interface InputStream
{
	ulong read(void* buffer, ulong length);
	ulong available();
	bool eof();
}

interface OutputStream
{
	ulong write(void* buffer, ulong length);
}

and a Stream and BufferedStream template class. Any object that supports 
those interfaces (as does my RawFile class) can then be extended by the 
two template classes to give full stream functionality by some simple 
aliases, example:

alias Stream!(RawFile) FileStream;
alias BufferedStream!(FileStream,2048) File;

I have attached the code, it is far from complete and probably buggy as 
heck as I have not done any testing (or unittests - I know I'm bad) this 
was simply proof of concept.

Any and all comments are welcome/desired.

Regan

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 02 2004
next sibling parent "Matthew" <admin stlsoft.dot.dot.dot.dot.org> writes:
"Regan Heath" <regan netwin.co.nz> wrote in message
news:opsb4onie15a2sq9 digitalmars.com...

 Typical stream implementations I have seen use the following heirarchy:

 class Stream {
 }
 class FileStream : Stream {
 }
 class SocketStream : Stream {
 }

 in other words you have to design/derive a class from Stream for each type
 of Stream you want to implement.

 I always thought this was kinda backwards, and limited, in fact after
 Arcane Jill said something along the lines of "anything that reads and
 writes bytes _is_ a stream" I got to thinking surely I can use templates
 to make that statement true.


 The way I have done it is to define some interfaces:

 interface InputStream
 {
 ulong read(void* buffer, ulong length);
 ulong available();
 bool eof();
 }

 interface OutputStream
 {
 ulong write(void* buffer, ulong length);
 }

 and a Stream and BufferedStream template class. Any object that supports
 those interfaces (as does my RawFile class) can then be extended by the
 two template classes to give full stream functionality by some simple
 aliases, example:

 alias Stream!(RawFile) FileStream;
 alias BufferedStream!(FileStream,2048) File;

 I have attached the code, it is far from complete and probably buggy as
 heck as I have not done any testing (or unittests - I know I'm bad) this
 was simply proof of concept.

 Any and all comments are welcome/desired.


They are remarkably similar to two interfaces - IInputStream & IOutputStream -
that I've recently written for a client's
networking project.

I can't include the code, of course. ;)

I agree with the general principle you outline. IStream is one of the elegant
parts of COM. (IStorage one of the hideous
ones.<g>).
Aug 02 2004
prev sibling next sibling parent reply parabolis <parabolis softhome.net> writes:
Regan Heath wrote:


 Typical stream implementations I have seen use the following heirarchy:
 
 class Stream {
 }
 class FileStream : Stream {
 }
 class SocketStream : Stream {
 }
 
 in other words you have to design/derive a class from Stream for each 
 type of Stream you want to implement.
 
 I always thought this was kinda backwards, and limited, in fact after 
 Arcane Jill said something along the lines of "anything that reads and 
 writes bytes _is_ a stream" I got to thinking surely I can use templates 
 to make that statement true.
 
 
 The way I have done it is to define some interfaces:
 
 interface InputStream
 {
     ulong read(void* buffer, ulong length);
     ulong available();
     bool eof();
 }
 
 interface OutputStream
 {
     ulong write(void* buffer, ulong length);
 }
 
 and a Stream and BufferedStream template class. Any object that supports 
 those interfaces (as does my RawFile class) can then be extended by the 
 two template classes to give full stream functionality by some simple 
 aliases, example:
 
 alias Stream!(RawFile) FileStream;
 alias BufferedStream!(FileStream,2048) File;
 
 I have attached the code, it is far from complete and probably buggy as 
 heck as I have not done any testing (or unittests - I know I'm bad) this 
 was simply proof of concept.
 
 Any and all comments are welcome/desired.
 
 Regan


I am not entirely clear on how what you are doing is different 
from the typical stream implementations you have seen.

Look through the thread "OT scanf in Java" for a long post of 
mine towards the top that explains why the Stream 
implementations that you think of as being backwards actually do 
exactly what you want.

To give a hint you should ask yourself how you can combine these 
streams:

(--Stream means InputStream or OutputStream)

File--Stream
Network--Stream
Memory--Stream

Buffered--Stream
Compress--Stream
CRC32 Digest--Stream (eg CRC32 or MD5)

Image--Stream
Video--Stream
Audio--Stream


Sometimes I will want my data to come from a File, other times I 
want it to come from the Network. Sometimes I want a digest of 
the compressed data other times I want a digest of the 
uncompressed data.

These actions are performed in the order I want based on the 
order in which I construct them:

new ImageStream( DigestStream( CompressStream ( FileStream ) ) )

This would read file data, decompresses the data and then 
compute the CRC32 of the uncompressed data and read image data 
from it. However:

new ImageStream( CompressStream ( DigestStream( FileStream ) ) )

This would read file data, compute the CRC32 of the uncompressed 
data and then decompress the data and read image data from it.


I suspect the class you want to implement will either duplicate 
the way Stream libraries work or you will find yourself writing 
seperate classes to handle the two cases above which is bad. 
Just try to write out all the possible combinations that these 9 
classes have already accomplished and you will see what I mean.
Aug 02 2004
parent reply Regan Heath <regan netwin.co.nz> writes:
On Mon, 02 Aug 2004 20:21:28 -0400, parabolis <parabolis softhome.net> 
wrote:

 Regan Heath wrote:


 Typical stream implementations I have seen use the following heirarchy:

 class Stream {
 }
 class FileStream : Stream {
 }
 class SocketStream : Stream {
 }

 in other words you have to design/derive a class from Stream for each 
 type of Stream you want to implement.

 I always thought this was kinda backwards, and limited, in fact after 
 Arcane Jill said something along the lines of "anything that reads and 
 writes bytes _is_ a stream" I got to thinking surely I can use 
 templates to make that statement true.


 The way I have done it is to define some interfaces:

 interface InputStream
 {
     ulong read(void* buffer, ulong length);
     ulong available();
     bool eof();
 }

 interface OutputStream
 {
     ulong write(void* buffer, ulong length);
 }

 and a Stream and BufferedStream template class. Any object that 
 supports those interfaces (as does my RawFile class) can then be 
 extended by the two template classes to give full stream functionality 
 by some simple aliases, example:

 alias Stream!(RawFile) FileStream;
 alias BufferedStream!(FileStream,2048) File;

 I have attached the code, it is far from complete and probably buggy as 
 heck as I have not done any testing (or unittests - I know I'm bad) 
 this was simply proof of concept.

 Any and all comments are welcome/desired.

 Regan


 I am not entirely clear on how what you are doing is different from the 
 typical stream implementations you have seen.


I have written the code to read different types etc once in my Stream 
template, and can apply it to _any_ class at all that implements one read 
method of the form:

   ulong read(void* buffer, ulong length);

the same applies to writing.


So, if you added methods in the forms given in my interfaces to std.socket 
you could say:

alias Stream!(Socket) SocketStream;
alias BufferedStream!(SocketStream,2048) BufferedSocket;

and you have a socket you can instantiate like so:

   SocketStream s = new SocketStream();

which is a buffered stream.


 Look through the thread "OT scanf in Java" for a long post of mine 
 towards the top that explains why the Stream implementations that you 
 think of as being backwards actually do exactly what you want.

 To give a hint you should ask yourself how you can combine these streams:


Combination is the next step, what I have so far is not intended to handle 
combinations.


 (--Stream means InputStream or OutputStream)

 File--Stream
 Network--Stream
 Memory--Stream

 Buffered--Stream
 Compress--Stream
 CRC32 Digest--Stream (eg CRC32 or MD5)

 Image--Stream
 Video--Stream
 Audio--Stream


 Sometimes I will want my data to come from a File, other times I want it 
 to come from the Network. Sometimes I want a digest of the compressed 
 data other times I want a digest of the uncompressed data.

 These actions are performed in the order I want based on the order in 
 which I construct them:

 new ImageStream( DigestStream( CompressStream ( FileStream ) ) )

 This would read file data, decompresses the data and then compute the 
 CRC32 of the uncompressed data and read image data from it. However:

 new ImageStream( CompressStream ( DigestStream( FileStream ) ) )

 This would read file data, compute the CRC32 of the uncompressed data 
 and then decompress the data and read image data from it.


My next step is to implement what I call 'filters' they are like your 
CompressStream etc above, they will link one stream to another and filter 
the data passing through them.


 I suspect the class you want to implement will either duplicate the way 
 Stream libraries work or you will find yourself writing seperate classes 
 to handle the two cases above which is bad. Just try to write out all 
 the possible combinations that these 9 classes have already accomplished 
 and you will see what I mean.


As I said above, combinations are my next step. I may end up handling them 
in the same way as you have shown above but I'd rather not, I dislike 
instantiation in the form:

   new ImageStream( CompressStream ( DigestStream( FileStream ) ) )

Regan.

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 02 2004
parent reply parabolis <parabolis softhome.net> writes:
Regan Heath wrote:


 As I said above, combinations are my next step. I may end up handling 
 them in the same way as you have shown above but I'd rather not, I 
 dislike instantiation in the form:
 
   new ImageStream( CompressStream ( DigestStream( FileStream ) ) )


What will your eq. version of the above be?
Aug 02 2004
parent Regan Heath <regan netwin.co.nz> writes:
On Mon, 02 Aug 2004 21:00:57 -0400, parabolis <parabolis softhome.net> 
wrote:


 Regan Heath wrote:


 As I said above, combinations are my next step. I may end up handling 
 them in the same way as you have shown above but I'd rather not, I 
 dislike instantiation in the form:

   new ImageStream( CompressStream ( DigestStream( FileStream ) ) )


 What will your eq. version of the above be?


I'm not at all sure yet.

Regan.

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 02 2004
prev sibling parent reply Ben Hinkle <bhinkle4 juno.com> writes:
Regan Heath wrote:


 Typical stream implementations I have seen use the following heirarchy:
 
 class Stream {
 }
 class FileStream : Stream {
 }
 class SocketStream : Stream {
 }
 
 in other words you have to design/derive a class from Stream for each type
 of Stream you want to implement.
 
 I always thought this was kinda backwards, and limited, in fact after
 Arcane Jill said something along the lines of "anything that reads and
 writes bytes _is_ a stream" I got to thinking surely I can use templates
 to make that statement true.
 
 
 The way I have done it is to define some interfaces:
 
 interface InputStream
 {
 ulong read(void* buffer, ulong length);
 ulong available();
 bool eof();
 }
 
 interface OutputStream
 {
 ulong write(void* buffer, ulong length);
 }
 
 and a Stream and BufferedStream template class. Any object that supports
 those interfaces (as does my RawFile class) can then be extended by the
 two template classes to give full stream functionality by some simple
 aliases, example:
 
 alias Stream!(RawFile) FileStream;
 alias BufferedStream!(FileStream,2048) File;
 
 I have attached the code, it is far from complete and probably buggy as
 heck as I have not done any testing (or unittests - I know I'm bad) this
 was simply proof of concept.
 
 Any and all comments are welcome/desired.
 
 Regan
 


cool idea(s). I like the "bolt-in" (or it bolt-on?) stuff. One possible
problem, though, is that if read(out int x), for example, is in
Stream!(File) and it call super.read(...) then the
BufferedStream!(FileStream) subclass will not read ints from the buffer
(the super.read goes directly to RawFile.read - not BufferedStream.read). I
guess I'm a little lost about how the buffering works.

At one point there was the suggestion of moving the default implementations
of all the read(out int x) etc from Stream (the original one in
std/stream.d) to a mixin so that instead of subclassing to get the default
implementation you mix it in. I don't know how that approach would change
around your code but it's worth thinking about. 
To be more concrete, here's how that would work:
template DefaultInputStream() {
  void read(out int x) {readExact(...);}
  char[] readLine() {...}
  ...
}
template DefaultOutputStream() {
  void write(int x) {...}
  void writeLine(char[] x) {...}
  ...
}
class Stream : InputStream, OutputStream {
 mixin DefaultInputStream;
 mixin DefaultOutputStream;
 ...
}

Anyhow, keep pluggin'.

-Ben
Aug 02 2004
next sibling parent reply Regan Heath <regan netwin.co.nz> writes:
On Mon, 02 Aug 2004 22:19:19 -0400, Ben Hinkle <bhinkle4 juno.com> wrote:

 Regan Heath wrote:


 Typical stream implementations I have seen use the following heirarchy:

 class Stream {
 }
 class FileStream : Stream {
 }
 class SocketStream : Stream {
 }

 in other words you have to design/derive a class from Stream for each 
 type
 of Stream you want to implement.

 I always thought this was kinda backwards, and limited, in fact after
 Arcane Jill said something along the lines of "anything that reads and
 writes bytes _is_ a stream" I got to thinking surely I can use templates
 to make that statement true.


 The way I have done it is to define some interfaces:

 interface InputStream
 {
 ulong read(void* buffer, ulong length);
 ulong available();
 bool eof();
 }

 interface OutputStream
 {
 ulong write(void* buffer, ulong length);
 }

 and a Stream and BufferedStream template class. Any object that supports
 those interfaces (as does my RawFile class) can then be extended by the
 two template classes to give full stream functionality by some simple
 aliases, example:

 alias Stream!(RawFile) FileStream;
 alias BufferedStream!(FileStream,2048) File;

 I have attached the code, it is far from complete and probably buggy as
 heck as I have not done any testing (or unittests - I know I'm bad) this
 was simply proof of concept.

 Any and all comments are welcome/desired.

 Regan


 cool idea(s). I like the "bolt-in" (or it bolt-on?) stuff.


Yeah, I think they're a great idea.


 One possible
 problem, though, is that if read(out int x), for example, is in
 Stream!(File) and it call super.read(...) then the
 BufferedStream!(FileStream) subclass will not read ints from the buffer
 (the super.read goes directly to RawFile.read - not 
 BufferedStream.read). I
 guess I'm a little lost about how the buffering works.


You're right. :(

I think to fix it I change:

   alias Stream!(RawFile) FileStream;
   alias BufferedStream!(FileStream,2048) File;

to

   alias BufferedStream!(RawFile,2048) File;
   alias Stream!(File) FileStream;

and then change the signatures of the methods in BufferedStream from

   ulong read(ubyte[] data)

to

   ulong read(void* buffer, ulong length)

etc.

This highlights an 'inclusion order' weakness to this design which needs 
to be fixed.


 At one point there was the suggestion of moving the default 
 implementations
 of all the read(out int x) etc from Stream (the original one in
 std/stream.d) to a mixin so that instead of subclassing to get the 
 default
 implementation you mix it in. I don't know how that approach would change
 around your code but it's worth thinking about.


I think this might just be the solution to the inclusion order weakness in 
my current design.
I'm going to turn my Stream template class into a pair of mixins.


 To be more concrete, here's how that would work:
 template DefaultInputStream() {
   void read(out int x) {readExact(...);}
   char[] readLine() {...}
   ...
 }
 template DefaultOutputStream() {
   void write(int x) {...}
   void writeLine(char[] x) {...}
   ...
 }
 class Stream : InputStream, OutputStream {
  mixin DefaultInputStream;
  mixin DefaultOutputStream;
  ...
 }

 Anyhow, keep pluggin'.


Thanks. I'll post version 2 when I have one.

Regan

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 02 2004
parent reply "Vathix" <vathixSpamFix dprogramming.com> writes:
 and then change the signatures of the methods in BufferedStream from

    ulong read(ubyte[] data)

 to

    ulong read(void* buffer, ulong length)

 etc.


Why not use void[] for general read and write?
Aug 02 2004
parent Regan Heath <regan netwin.co.nz> writes:
On Tue, 3 Aug 2004 00:19:15 -0400, Vathix <vathixSpamFix dprogramming.com> 
wrote:

 and then change the signatures of the methods in BufferedStream from

    ulong read(ubyte[] data)

 to

    ulong read(void* buffer, ulong length)

 etc.


 Why not use void[] for general read and write?


So you can go:

bool read(out byte x)	{ if (this.read(&x,x.sizeof) ..etc..
bool read(out ubyte x)	{ if (this.read(&x,x.sizeof) ..etc..
bool read(out short x)	{ if (this.read(&x,x.sizeof) ..etc..

Regan.

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Aug 02 2004
prev sibling parent Sean Kelly <sean f4.ca> writes:
In article <cemsn6$kb9$1 digitaldaemon.com>, Ben Hinkle says...

At one point there was the suggestion of moving the default implementations
of all the read(out int x) etc from Stream (the original one in
std/stream.d) to a mixin so that instead of subclassing to get the default
implementation you mix it in.


This is kind of how the stream implementation I was working on is done.  If I
find the time to finish it I'll post it.

Sean
Aug 02 2004