www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - `Socket.receive` providing arbitrary packet sizes and hanging without

reply Unazed Spectaculum <unazed spec.org> writes:
ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
{
	ubyte[] buffer = new ubyte[receiveCount];
	size_t count = socket.receive(buffer);
	return buffer[0 .. count];
}

string receiveAll(T)(T socket, size_t segmentSize = 1024)
{
	ubyte[][] data;
	size_t count = 0;
	
	do
	{
		debug(1) writefln("Chunk %s", count);
		data ~= receiveBytes(socket, segmentSize);
		writeln(data[count]);
		if (!data)
			break;

	} while(data[count++]);

	char[] stringData;

	foreach (elem; data)
		stringData ~= elem;

	debug(1) writeln(`Exiting "receiveAll"`);

	return to!string(stringData);
}


I've tried many variations of the above code; both with the 
retrieve integrated into the do-while loop in receiveAll; however 
I was just seeing whether abstracting the code would let me spot 
my issues a bit faster because it won't require me to make two 
variables for the buffer and amount of bytes received, more like 
the typical interface you get.

Issue is; when receiving any size buffer from the end-point 
(tested with >1024 byte and <1024 byte buffers); there is always 
a superfluous chunk which is awaiting data.


Listening: 0.0.0.0:6969
Client: somebody:58769
Chunk 0
[123, 34, 109, 101, 116, 104, 111, 100, 34, 58, 32, 34, 114, 101, 
116, 114, 105, 101, 118, 101, 34, 44, 32, 34, 102, 105, 108, 101, 
110, 97, 109, 101, 34, 58, 32, 34, 51, 48, 50, 54, 57, 50, 48, 
49, 55, 50, 51, 54, 56, 54, 57, 49, 50, 49, 95, 97, 97, 97, 97, 
97, 34, 44, 32, 34, 100, 97, 116, 97, 34, 58, 32, 34, 34, 125]
Chunk 1
[PAUSE]


Listening: 0.0.0.0:6969
Client: somebody:58767
Chunk 0
[123, 34, 109, 101, 116, 104, 111, 100, 34, 58, 32, 34, 114, 101, 
116, 114, 105, 101, 118, 101, 34, 44, 32, 34, 102, 105, 108, 101, 
110, 97, 109, 101, 34, 58, 32, 34, ... 97]
Chunk 1
[97, ... 125]
Chunk 2
[PAUSE]


No matter what way I try; my code doesn't seem to know when to 
quit regardless of the check. Also for the arbitrary packet 
sizes, I would've expected that if I received N bytes X times, 
the first X-1 times would be perfectly N not some unusual integer.
Simply put, say I'm receiving 1024 bytes 5 times. The length of 
each item on the stack looks like:

[720,
  490,
  1024,
  103
]

Although I'd assume it'd be more like:

[1024,
  1024,
  289
]

What's up with this? It makes working with sockets so damn 
tedious since there's no room for assumptions; can anybody 
suggest an abstracted library for sockets in D or help with my 
former issue?
Dec 13 2017
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 12/13/2017 11:39 AM, Unazed Spectaculum wrote:

 ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
 {
      ubyte[] buffer = new ubyte[receiveCount];
      size_t count = socket.receive(buffer);


Don't trust code you find on newsgroups. :o) You have to check the 
returned value first. According to documentation, it can return 
Socket.ERROR:

   https://dlang.org/phobos/std_socket.html#.Socket.receive


 there is always a superfluous chunk
 which is awaiting data.


Can you show with complete code? Perhaps the stream is in blocking mode?


 No matter what way I try; my code doesn't seem to know when to quit
 regardless of the check. Also for the arbitrary packet sizes, I would've
 expected that if I received N bytes X times, the first X-1 times would
 be perfectly N not some unusual integer.
 Simply put, say I'm receiving 1024 bytes 5 times. The length of each
 item on the stack looks like:

 [720,
   490,
   1024,
   103
 ]


Posix read(2) man page says

"It is not an error if this number is smaller than the number of bytes
requested; this may happen for example because fewer bytes are actually
available right now (maybe because we were close to end-of-file, or because
we are reading from a pipe, or from a terminal), or because read() was
interrupted by a signal."

Ali
Dec 13 2017
parent reply Unazed Spectaculum <unazed spec.org> writes:
On Thursday, 14 December 2017 at 00:09:39 UTC, Ali Çehreli wrote:

 On 12/13/2017 11:39 AM, Unazed Spectaculum wrote:

 ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
 {
      ubyte[] buffer = new ubyte[receiveCount];
      size_t count = socket.receive(buffer);


 Don't trust code you find on newsgroups. :o) You have to check 
 the returned value first. According to documentation, it can 
 return Socket.ERROR:

   https://dlang.org/phobos/std_socket.html#.Socket.receive


 there is always a superfluous chunk
 which is awaiting data.


 Can you show with complete code? Perhaps the stream is in 
 blocking mode?


 No matter what way I try; my code doesn't seem to know when


 to quit

 regardless of the check. Also for the arbitrary packet sizes,


 I would've

 expected that if I received N bytes X times, the first X-1


 times would

 be perfectly N not some unusual integer.
 Simply put, say I'm receiving 1024 bytes 5 times. The length


 of each

 item on the stack looks like:

 [720,
   490,
   1024,
   103
 ]


 Posix read(2) man page says

 "It is not an error if this number is smaller than the number 
 of bytes
 requested; this may happen for example because fewer bytes are 
 actually
 available right now (maybe because we were close to 
 end-of-file, or because
 we are reading from a pipe, or from a terminal), or because 
 read() was
 interrupted by a signal."

 Ali


void main()
{
	auto socket = new TcpSocket();
	setupSocket(socket, "0.0.0.0", 6969);
	
	writefln("Listening: %s", socket.localAddress);

	while(true)
	{
		Socket client = socket.accept();
		debug(1) writefln("Client: %s", client.remoteAddress);

		auto data = receiveAll(client);
		writeln(data);
		JSONValue json;

		try {
			json = parseJSON(data);
		} catch (JSONException e) {
			debug(1) writefln("Failed parsing data as JSON, aborting.\n|| 
%s", e);
			client.close();
			continue;
		} catch (Exception e) {
			debug(1) writefln("Client caused exception:\n||%s", e);
			client.close();
			continue;
		}

		if (!verifyValues(json))
		{
			debug(1) writefln("Client missed out important key fields: 
%s", client.remoteAddress);
			client.close();
			continue;
		}

		debug(1) writeln("Client transacted successful JSON packet.");

		writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d",
				client.remoteAddress,
				json["filename"],
				json["method"],
				data.length
			);

		if (json["method"].str == "store")
			storeData(json["filename"].str, json["data"].str);
		else if (json["method"].str == "retrieve")
			retrieveData(client, json["filename"].str);

		client.close();
	}
}

This is the only function which has a call to `receiveAll`, also 
yeah I don't typically check return codes for error values, I 
just assume it'll all work and if it doesn't a fresh restart will 
fix it; but I'll include it in my code just in case.
Dec 14 2017
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 12/14/2017 11:55 AM, Unazed Spectaculum wrote:


 This is the only function which has a call to `receiveAll`



I marked my changes with [Ali]:

import std.stdio;
import std.socket;
import std.conv;
import std.json;

ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
{
     ubyte[] buffer = new ubyte[receiveCount];
     size_t count = socket.receive(buffer);
     // [Ali] Return null when there is no data
     if (count == Socket.ERROR || count == 0) {
         return null;
     }
     return buffer[0 .. count];
}

string receiveAll(T)(T socket, size_t segmentSize = 1024)
{
     ubyte[][] data;
     size_t count = 0;

     do
     {
         debug(1) writefln("Chunk %s", count);
         auto part = receiveBytes(socket, segmentSize);
         // [Ali] Done when there is no data
         if (!part) {
             break;
         }
         data ~= part;
         writeln(data[count]);
         if (!data)
             break;

     } while(data[count++]);

     char[] stringData;

     foreach (elem; data)
         stringData ~= elem;

     debug(1) writeln(`Exiting "receiveAll"`);

     return to!string(stringData);
}

void main()
{
     auto socket = new TcpSocket();
     // [Ali]    setupSocket(socket, "0.0.0.0", 6969);
     // The following had worked in a test program:
     socket.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, 
true);
     socket.bind(new InternetAddress(6969));
     socket.listen(1);

     writefln("Listening: %s", socket.localAddress);

     while(true)
     {
         Socket client = socket.accept();
         debug(1) writefln("Client: %s", client.remoteAddress);

         auto data = receiveAll(client);
         writeln(data);
         JSONValue json;

         try {
             json = parseJSON(data);
         } catch (JSONException e) {
             debug(1) writefln("Failed parsing data as JSON, 
aborting.\n|| %s", e);
             client.close();
             continue;
         } catch (Exception e) {
             debug(1) writefln("Client caused exception:\n||%s", e);
             client.close();
             continue;
         }

         /+ [Ali] Not important for this test
         if (!verifyValues(json))
         {
             debug(1) writefln("Client missed out important key fields: 
%s", client.remoteAddress);
             client.close();
             continue;
         }
         +/

         debug(1) writeln("Client transacted successful JSON packet.");

         writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d",
                 client.remoteAddress,
                 json["filename"],
                 json["method"],
                 data.length
             );

         // [Ali] Not important for this test
         /+
         if (json["method"].str == "store")
             storeData(json["filename"].str, json["data"].str);
         else if (json["method"].str == "retrieve")
             retrieveData(client, json["filename"].str);
         +/
         client.close();
     }
}

1) Start the program

2) $ telnet 0.0.0.0 6969
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.

3) Paste expected json to terminal:

{"filename":"/tmp/foo","method":"bar"}

4) Press the escape character

^]

5) Press Ctrl-D (Ctrl-Z on Windows) to terminate the connection

telnet> Connection closed.

Here is the output from the program:

Listening: 0.0.0.0:6969
[123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116, 
109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100, 
34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
{"filename":"/tmp/foo","method":"bar"}

127.0.0.1:47704:
	Filename: "\/tmp\/foo"
	Method: "bar"
	Data length: 40

I think it works! :)

Ali
Dec 14 2017
parent reply Unazed Spectaculum <unazed spec.org> writes:
On Thursday, 14 December 2017 at 20:27:36 UTC, Ali Çehreli wrote:

 On 12/14/2017 11:55 AM, Unazed Spectaculum wrote:


 This is the only function which has a call to `receiveAll`



 I marked my changes with [Ali]:

 import std.stdio;
 import std.socket;
 import std.conv;
 import std.json;

 ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
 {
     ubyte[] buffer = new ubyte[receiveCount];
     size_t count = socket.receive(buffer);
     // [Ali] Return null when there is no data
     if (count == Socket.ERROR || count == 0) {
         return null;
     }
     return buffer[0 .. count];
 }

 string receiveAll(T)(T socket, size_t segmentSize = 1024)
 {
     ubyte[][] data;
     size_t count = 0;

     do
     {
         debug(1) writefln("Chunk %s", count);
         auto part = receiveBytes(socket, segmentSize);
         // [Ali] Done when there is no data
         if (!part) {
             break;
         }
         data ~= part;
         writeln(data[count]);
         if (!data)
             break;

     } while(data[count++]);

     char[] stringData;

     foreach (elem; data)
         stringData ~= elem;

     debug(1) writeln(`Exiting "receiveAll"`);

     return to!string(stringData);
 }

 void main()
 {
     auto socket = new TcpSocket();
     // [Ali]    setupSocket(socket, "0.0.0.0", 6969);
     // The following had worked in a test program:
     socket.setOption(SocketOptionLevel.SOCKET, 
 SocketOption.REUSEADDR, true);
     socket.bind(new InternetAddress(6969));
     socket.listen(1);

     writefln("Listening: %s", socket.localAddress);

     while(true)
     {
         Socket client = socket.accept();
         debug(1) writefln("Client: %s", client.remoteAddress);

         auto data = receiveAll(client);
         writeln(data);
         JSONValue json;

         try {
             json = parseJSON(data);
         } catch (JSONException e) {
             debug(1) writefln("Failed parsing data as JSON, 
 aborting.\n|| %s", e);
             client.close();
             continue;
         } catch (Exception e) {
             debug(1) writefln("Client caused exception:\n||%s", 
 e);
             client.close();
             continue;
         }

         /+ [Ali] Not important for this test
         if (!verifyValues(json))
         {
             debug(1) writefln("Client missed out important key 
 fields: %s", client.remoteAddress);
             client.close();
             continue;
         }
         +/

         debug(1) writeln("Client transacted successful JSON 
 packet.");

         writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData 
 length: %d",
                 client.remoteAddress,
                 json["filename"],
                 json["method"],
                 data.length
             );

         // [Ali] Not important for this test
         /+
         if (json["method"].str == "store")
             storeData(json["filename"].str, json["data"].str);
         else if (json["method"].str == "retrieve")
             retrieveData(client, json["filename"].str);
         +/
         client.close();
     }
 }

 1) Start the program

 2) $ telnet 0.0.0.0 6969
 Trying 0.0.0.0...
 Connected to 0.0.0.0.
 Escape character is '^]'.

 3) Paste expected json to terminal:

 {"filename":"/tmp/foo","method":"bar"}

 4) Press the escape character

 ^]

 5) Press Ctrl-D (Ctrl-Z on Windows) to terminate the connection

 telnet> Connection closed.

 Here is the output from the program:

 Listening: 0.0.0.0:6969
 [123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 
 47, 116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 
 116, 104, 111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
 {"filename":"/tmp/foo","method":"bar"}

 127.0.0.1:47704:
 	Filename: "\/tmp\/foo"
 	Method: "bar"
 	Data length: 40

 I think it works! :)

 Ali


I've tried to integrate your code with mine; to no avail, I've 
directly tested your code; and yet again to no avail, so this 
should conclude the thread as it pinpoints an error with my 
actual OS.

Just in case, as a last resort in case I'm actually that dumb, 
here's me reproducing your steps:

1) Starting program
unazed unazed  /home/d/storage-server  dmd -debug -run app.d
Listening: 0.0.0.0:6969

2) telnet to the server
  unazed unazed  ~  telnet 0.0.0.0 6969
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.

3) paste '{"filename":"/tmp/foo","method":"bar"}'
  unazed unazed  /home/d/storage-server  dmd -debug -run app.d
Listening: 0.0.0.0:6969
Client: 127.0.0.1:55014
Chunk 0
[123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 
116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 
111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
Chunk 1
[pause]

4-5) press ESC + CTRL+D (I'm on Linux)
Chunk 1
[27]
Chunk 2
[pause]

So, big thanks , and sorry for taking longer to respond; I just 
get anxious about solving things sometimes and now it's kind of 
annoying I have to do some more research to fix this issue.
Dec 16 2017
parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 12/16/2017 05:21 PM, Unazed Spectaculum wrote:


 1) Starting program
 unazed unazed  /home/d/storage-server  dmd -debug -run app.d


Although I don't normally use the -run switch, as expected, it works 
with -run as well. (More below.)


 Listening: 0.0.0.0:6969

 2) telnet to the server
   unazed unazed  ~  telnet 0.0.0.0 6969
 Trying 0.0.0.0...
 Connected to 0.0.0.0.
 Escape character is '^]'.


Note that line! ;)


 3) paste '{"filename":"/tmp/foo","method":"bar"}'


Please also press Enter after that. I don't know what code is at fault 
but the json parser was not happy without that newline at the end.


   unazed unazed  /home/d/storage-server  dmd -debug -run app.d
 Listening: 0.0.0.0:6969
 Client: 127.0.0.1:55014
 Chunk 0
 [123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116,
 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100,
 34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
 Chunk 1
 [pause]


Same here...


 4-5) press ESC + CTRL+D (I'm on Linux)


Aha! That's the problem! You should enter the "Escape character", which 
is ^] as telnet indicates. You should press Ctrl-] (stay away from the 
ESC key). Otherwise, you're injecting an ESC character to the 
communication stream. (The 27 below.)

By the way, I'm on Linux as well! :) (Mint, Ubuntu based.)

And finally, press Ctrl-D to end the stream. It really works:

[...]
Chunk 1
Exiting "receiveAll"
{"filename":"/tmp/foo","method":"bar"}
[...]

Ali
Dec 16 2017