www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Error: not enough data in stream

reply jicman <jicman_member pathlink.com> writes:
Greetings!

Seeing that everyone just want to argue, ;-), I hope some of you could help on
this one:  I am downloading a bunch of xml file using socket and datastream, but
from time to time, the program that I wrote would finish with

Error: not enough data in stream

I know that it's downloading lots of xml data every time this happens.  I don't
know how much, but my question is, is there a definite amount of data that a
stream can handle?  Don't tell me that this is because of memory, cuz this
server has over 1.5GB of memory and I know I am not downloading that much. ;-)

any help would be greatly appreciated.

thanks,

jic
Mar 08 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Tue, 8 Mar 2005 09:26:31 +0000 (UTC), jicman  
<jicman_member pathlink.com> wrote:

 Greetings!

 Seeing that everyone just want to argue, ;-), I hope some of you could  
 help on
 this one:  I am downloading a bunch of xml file using socket and  
 datastream, but
 from time to time, the program that I wrote would finish with

 Error: not enough data in stream


I cannot find the string "not enough data in stream" in the phobos  
source.. is that your own error message?


 I know that it's downloading lots of xml data every time this happens.   
 I don't
 know how much, but my question is, is there a definite amount of data  
 that a
 stream can handle?  Don't tell me that this is because of memory, cuz  
 this
 server has over 1.5GB of memory and I know I am not downloading that  
 much. ;-)

 any help would be greatly appreciated.


Perhaps posting some code would help? esp if you can cut the code down to  
a small sample which reproduces the problem. I find that attempting to do  
this, often helps find the problem.

Regan
Mar 08 2005
next sibling parent reply =?ISO-8859-15?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
Regan Heath wrote:


 Error: not enough data in stream


 
 I cannot find the string "not enough data in stream" in the phobos  
 source.. is that your own error message?


It's in std.stream:

   // reads block of data of specified size,
   // throws ReadException on error
   void readExact(void* buffer, uint size)
   {
     uint readsize = readBlock(buffer, size);
     if (readsize != size)
       throw new ReadException("not enough data in stream");
   }


I think it's a flawed design, that EOF is an Exception ?

I'd rather have "read" return 'false' on EOF, myself.
And save the exceptions for the I/O errors and such...

--anders
Mar 08 2005
next sibling parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Tue, 08 Mar 2005 10:51:26 +0100, Anders F Björklund <afb algonet.se>  
wrote:

 Regan Heath wrote:


 Error: not enough data in stream


  I cannot find the string "not enough data in stream" in the phobos   
 source.. is that your own error message?


 It's in std.stream:


Aha! That'll teach me to forget the "search subfolders" checkbox. :)


   // reads block of data of specified size,
   // throws ReadException on error
   void readExact(void* buffer, uint size)
   {
     uint readsize = readBlock(buffer, size);
     if (readsize != size)
       throw new ReadException("not enough data in stream");
   }


 I think it's a flawed design, that EOF is an Exception ?

 I'd rather have "read" return 'false' on EOF, myself.
 And save the exceptions for the I/O errors and such...


What if you're trying to read an 'int' i.e. 4 bytes and there is only 1, 2  
or 3 bytes available?
(as it might be doing above)

Regan
Mar 08 2005
parent =?ISO-8859-15?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
Regan Heath wrote:


 I think it's a flawed design, that EOF is an Exception ?

 I'd rather have "read" return 'false' on EOF, myself.
 And save the exceptions for the I/O errors and such...


 
 What if you're trying to read an 'int' i.e. 4 bytes and there is only 1, 
 2  or 3 bytes available?
 (as it might be doing above)


Actually, I was only thinking about bytes and lines...

You might be right. Either way, that's how it works now.

--anders
Mar 08 2005
prev sibling parent "Ben Hinkle" <ben.hinkle gmail.com> writes:
"Anders F Björklund" <afb algonet.se> wrote in message 
news:d0jsiv$1vrf$1 digitaldaemon.com...

 Regan Heath wrote:


 Error: not enough data in stream


 I cannot find the string "not enough data in stream" in the phobos 
 source.. is that your own error message?


 It's in std.stream:

   // reads block of data of specified size,
   // throws ReadException on error
   void readExact(void* buffer, uint size)
   {
     uint readsize = readBlock(buffer, size);
     if (readsize != size)
       throw new ReadException("not enough data in stream");
   }


 I think it's a flawed design, that EOF is an Exception ?

 I'd rather have "read" return 'false' on EOF, myself.
 And save the exceptions for the I/O errors and such...

 --anders


Hence the name "readExact". The function "readBlock", as you could guess 
from the pasted snippet, returns the number of bytes read. The reason this 
is exceptional is that by calling readExact the coder is asserting there are 
enough bytes in the stream to complete the read. If the stream closes 
prematurely there won't be enough bytes so it throws.
Mar 08 2005
prev sibling parent reply jicman <jicman_member pathlink.com> writes:
Regan Heath says...

On Tue, 8 Mar 2005 09:26:31 +0000 (UTC), jicman  
<jicman_member pathlink.com> wrote:

 Greetings!

 Seeing that everyone just want to argue, ;-), I hope some of you could  
 help on
 this one:  I am downloading a bunch of xml file using socket and  
 datastream, but
 from time to time, the program that I wrote would finish with

 Error: not enough data in stream


I cannot find the string "not enough data in stream" in the phobos  
source.. is that your own error message?


Nope.   Not my own message.  It may be what the server is giving my program
which d.DataStream is probably getting and printing.

But, I think I figured it out.  The server from where the XML data is coming
from may have had a cap on its maximum data output.  It was set to 128MB, so I
raised it to 1GB and now the error has disappeared, but the user which had this
problem previously is still downloading xml data and it's up to 500+MB!  YIKES!
This is bad design at its best. :-)


 I know that it's downloading lots of xml data every time this happens.   
 I don't
 know how much, but my question is, is there a definite amount of data  
 that a
 stream can handle?  Don't tell me that this is because of memory, cuz  
 this
 server has over 1.5GB of memory and I know I am not downloading that  
 much. ;-)

 any help would be greatly appreciated.


Perhaps posting some code would help? esp if you can cut the code down to  
a small sample which reproduces the problem. I find that attempting to do  
this, often helps find the problem.


I would, but I don't have the time, right now.  I am at a customer site and I am
trying to clean up a DB and I'm changing code on the fly.

Did I say that I love d?

thanks.

jic
Mar 08 2005
parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
 But, I think I figured it out.  The server from where the XML data is 
 coming
 from may have had a cap on its maximum data output.  It was set to 128MB, 
 so I
 raised it to 1GB and now the error has disappeared, but the user which had 
 this
 problem previously is still downloading xml data and it's up to 500+MB! 
 YIKES!
 This is bad design at its best. :-)


I don't understand - is there a bug in std.stream? I can't tell from that 
paragraph.
Mar 08 2005
parent jicman <jicman_member pathlink.com> writes:
Ben Hinkle says...

 But, I think I figured it out.  The server from where the XML data is 
 coming
 from may have had a cap on its maximum data output.  It was set to 128MB, 
 so I
 raised it to 1GB and now the error has disappeared, but the user which had 
 this
 problem previously is still downloading xml data and it's up to 500+MB! 
 YIKES!
 This is bad design at its best. :-)


I don't understand - is there a bug in std.stream? I can't tell from that 
paragraph.


Sorry.  typing and thinking at the same time has not been one of my strongest
showings ever! :-)

No, there is no bug on this post.  The problem was caused by the server's
software where I was getting the data from, which, by the way, is written in
java.  I was trying to acquire data from that server using d.   D acted
accordingly, so the problem was java memory cap setting.  The cap was set for
128MB of memory.  The amount of XML that I was downloading exceeded that amount
and I kept getting that "Error: not enough data in stream."  After raising the
memory cap to 1024MB, the problem went away.

Apparently, d was reading socket.datastream from the server ok, but when the
amount of data from the server exceeded 128MB, the server stopped sending data
and d aborted with "Error: not enough data in stream."  This confused me a bit
and I thought that it was something on my side.  (Perhaps we should change that
error.)  But, this only meant that no data at all and the server closed the
socket connection.  It didn't even send any http headers or anything.  Weird.
It just returned a null string.  Go figured.  Of course, the server also went
down hard. :-)  So, that's how the West was won.

Or something like that.

So, I am ok and you should ignore this post.  But, perhaps the error is a little
confusing, since I thought it was my problem, when it really was the other side.

Thanks.

jic
Mar 08 2005