www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - problem with sockets under win32; premature connection termination

reply Downs <default_357-line yahoo.de> writes:
The following code should theoretically wait for a single TCP connection, and
transfer a file over it.

import std.stdio, std.socket, std.string, std.file;

void main(char[][] args) {
  auto l=new Socket(AddressFamily.INET, SocketType.STREAM, ProtocolType.TCP);
  l.blocking=true;
  with (l) {
   bind(new InternetAddress("0.0.0.0", atoi(args[1])));
   listen(16);
  }
  auto request=l.accept;
  request.send("HTTP/1.0 200 OK\r\n\r\n");
  request.send(read(args[2]));
  request.shutdown(SocketShutdown.BOTH);
  request.close;
}

Trying to use this program to host a file (images work best), I ran into the
following problem:
the image would transmit, partially, then the request.send call would return -
prematurely -
and the rest of the unsent data would be silently discarded.
Experimenting with linger and SetOption produced no discernible effect.
I've tried to find a solution to this problem for weeks now, but I don't seem
to be making headway.

Halp? ;_;

  -- downs

PS: If you try to replicate it, it is recommended to use a mid-latency
connection so that the transfer doesn't complete immediately.
PPS: Interestingly, it works with netcat. Firefox, IE and Opera break though.
May 10 2007
parent reply "Chris Miller" <chris dprogramming.com> writes:
On Thu, 10 May 2007 18:55:05 -0400, Downs <default_357-line yahoo.de>  =

wrote:


 The following code should theoretically wait for a single TCP  =



 connection, and transfer a file over it.

 import std.stdio, std.socket, std.string, std.file;

 void main(char[][] args) {
   auto l=3Dnew Socket(AddressFamily.INET, SocketType.STREAM,  =



 ProtocolType.TCP);
   l.blocking=3Dtrue;
   with (l) {
    bind(new InternetAddress("0.0.0.0", atoi(args[1])));
    listen(16);
   }
   auto request=3Dl.accept;
   request.send("HTTP/1.0 200 OK\r\n\r\n");
   request.send(read(args[2]));
   request.shutdown(SocketShutdown.BOTH);
   request.close;
 }

 Trying to use this program to host a file (images work best), I ran in=


to  =


 the following problem:
 the image would transmit, partially, then the request.send call would =


 =


 return - prematurely -
 and the rest of the unsent data would be silently discarded.
 Experimenting with linger and SetOption produced no discernible effect=


.

 I've tried to find a solution to this problem for weeks now, but I don=


't  =


 seem to be making headway.

 Halp? ;_;

   -- downs

 PS: If you try to replicate it, it is recommended to use a mid-latency=


  =


 connection so that the transfer doesn't complete immediately.
 PPS: Interestingly, it works with netcat. Firefox, IE and Opera break =


 =


 though.


Socket.send doesn't guarantee the whole thing will send; it returns how =
 =

many bytes are buffered to be sent. You should loop until it reports all=
  =

the bytes are buffered to be sent, or use SocketStream, which does this =
 =

for you. Also, using Socket.shutdown to shutdown sends/writes (included =
by  =

BOTH) might be aborting the unsent buffer. Finally, after these  =

appropriate changes, I believe there's still a chance that the whole  =

buffer won't be sent if the linger time expires before the whole buffer =
 =

was able to send.
May 10 2007
parent reply Downs <default_357-line yahoo.de> writes:
Chris Miller wrote:

 On Thu, 10 May 2007 18:55:05 -0400, Downs <default_357-line yahoo.de> 
 wrote:
 

 The following code should theoretically wait for a single TCP 
 connection, and transfer a file over it.

 import std.stdio, std.socket, std.string, std.file;

 void main(char[][] args) {
   auto l=new Socket(AddressFamily.INET, SocketType.STREAM, 
 ProtocolType.TCP);
   l.blocking=true;
   with (l) {
    bind(new InternetAddress("0.0.0.0", atoi(args[1])));
    listen(16);
   }
   auto request=l.accept;
   request.send("HTTP/1.0 200 OK\r\n\r\n");
   request.send(read(args[2]));
   request.shutdown(SocketShutdown.BOTH);
   request.close;
 }

 Trying to use this program to host a file (images work best), I ran 
 into the following problem:
 the image would transmit, partially, then the request.send call would 
 return - prematurely -
 and the rest of the unsent data would be silently discarded.
 Experimenting with linger and SetOption produced no discernible effect.
 I've tried to find a solution to this problem for weeks now, but I 
 don't seem to be making headway.

 Halp? ;_;

   -- downs

 PS: If you try to replicate it, it is recommended to use a mid-latency 
 connection so that the transfer doesn't complete immediately.
 PPS: Interestingly, it works with netcat. Firefox, IE and Opera break 
 though.


 
 Socket.send doesn't guarantee the whole thing will send; it returns how 
 many bytes are buffered to be sent.


Sorry, I omitted this. I tried it; didn't help.


 Also, using Socket.shutdown to shutdown sends/writes (included 
 by BOTH) might be aborting the unsent buffer.


Yes, on Linux. Not on Win32. (it's even recommended, see the std.socket docs)


 Finally, after these 
 appropriate changes, I believe there's still a chance that the whole 
 buffer won't be sent if the linger time expires before the whole buffer 
 was able to send.


I tried sending linger to 30s. Didn't help any. But thanks for your input.
  -- downs
May 10 2007
parent reply Regan Heath <regan netmail.co.nz> writes:
I'd like to think I have a bit of experience with sockets and yet this problem
has me a bit stumped too.  I decided to pull the calls to the winsock stuff out
of the classes and call them directly, eg

---------------------
import std.stdio, std.socket, std.string, std.file;

import std.c.windows.winsock, std.windows.syserror, std.c.windows.windows;

alias std.c.windows.winsock ws;

void main(char[][] args) {
	char[] buf = "TEST";
	socket_t l, request;
	sockaddr_in sin;	
	uint num = 0;
	int size;
	
	sin.sin_addr.s_addr = inet_addr("127.0.0.1");
	sin.sin_port = htons(atoi(args[1]));
	writef(std.string.toString(inet_ntoa(sin.sin_addr)),":",
		std.string.toString(ntohs(sin.sin_port)),"\n");
	
	l = cast(socket_t)ws.socket(AddressFamily.INET, SocketType.STREAM,
ProtocolType.TCP);
	writef(l,"\n");
	writef(ws.ioctlsocket(l, FIONBIO, &num),"\n");
	writef(ws.bind(l, cast(sockaddr*)&sin, sin.sizeof),"\n");
	writef(ws.listen(l, 16),"\n");

	request = cast(socket_t) ws.accept(l, null, null);

	size = sin.sizeof;
	ws.getsockname(request, cast(sockaddr*)&sin, &size);
	writef(std.string.toString(inet_ntoa(sin.sin_addr)),":",
		std.string.toString(ntohs(sin.sin_port)),"\n");

	writef(request,"\n");
	writef(ws.ioctlsocket(request, FIONBIO, &num),"\n");
	writef(ws.send(request, buf.ptr, buf.length, 0),"\n");
	Sleep(5000);
	writef(ws.shutdown(request, SocketShutdown.BOTH),"\n");
	writef(ws.closesocket(request),"\n");
	writef(ws.closesocket(l),"\n");
}
---------------------

Compiling this with:

  dmd bug.d wsock32.lib

and running with:

  bug 80

then telnetting to it, i.e.

  telnet 127.0.0.1 80

gives me this output on screen:

127.0.0.1:80
924
0
0
0
127.0.0.1:80
916
0
4
0
0
0

the actual socket handle values may differ for you but note the send reports 4
and the other calls report 0, no error.

The most interesting part is the pause I added, this prevent the socket being
closed immediately, however my telnet session connects and immediately closes
receiving no data.

I'm hoping by posting this someone can tell us both where we went wrong ;)

Regan Heath
May 12 2007
parent reply Regan Heath <regan netmail.co.nz> writes:
Still stumped I tried C this time with the same results.

----------
#include <winsock2.h>
#include <stdio.h>
#include <stdlib.h>
#include <process.h>

#define ISEOL(c) (c == '\r' || c == '\n')

void print_error()
{
	char bf[1000];
	int n,i;

	n = FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM,
			NULL,
			WSAGetLastError(),
			LANG_USER_DEFAULT,  //MAKELANGID(LANG_NEUTRAL,SUBLANG_DEFAULT),
			bf,
			1000,
			NULL);

	for(; n > 0 && ISEOL(bf[n-1]); n--) bf[n-1] = '\0';

	for(i = n; i >= 0; i--)
	{
		if (!ISEOL(bf[i])) continue;
		if (bf[i] == '\n') bf[i] = ' ';
		if (bf[i] == '\r') memmove(&bf[i], &bf[i+1], n-i);
	}

	printf("ERROR: %s\n",bf);
}

void init()
{
	WSADATA wd;
	if (WSAStartup(0x2020, &wd)) {
		print_error();
		exit(1);
	}
}

void main(int argc, char* args[])
{
	SOCKET l, request;
	SOCKADDR_IN sin;
	unsigned long num = 0;
	int size;

	init();

	sin.sin_addr.s_addr = inet_addr("127.0.0.1");
	sin.sin_port = htons(atoi(args[1]));
	sin.sin_family = AF_INET;
	printf("%s:%d\n",inet_ntoa(sin.sin_addr),ntohs(sin.sin_port));

	l = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
	printf("%d\n",l);
	printf("%d\n",ioctlsocket(l, FIONBIO, &num));
	printf("%d\n",bind(l, (SOCKADDR*)&sin, sizeof(sin)));
	printf("%d\n",listen(l, SOMAXCONN));

	request = accept(l, NULL, NULL);

	size = sizeof(sin);
	getsockname(request, (SOCKADDR*)&sin, &size);
	printf("%s:%d\n",inet_ntoa(sin.sin_addr),ntohs(sin.sin_port));

	printf("%d\n",request);
	printf("%d\n",ioctlsocket(request, FIONBIO, &num));
	printf("%d\n",send(request,"TEST\r\n",6,0));
	Sleep(5000);
	printf("%d\n",shutdown(request, SD_BOTH));
	printf("%d\n",closesocket(request));
	printf("%d\n",closesocket(l));
}
-----------

Compile with:

  dmc bug.c wsock32.lib

Run as:

  bug 80

Telnet with:

  telnet 127.0.0.1 80

Results:

127.0.0.1:80
924
0
0
0
127.0.0.1:80
916
0
6
0
0
0

Telnet window connects and closes, no data is seen.

Odd.

Regan Heath
May 12 2007
parent reply Downs <default_357-line yahoo.de> writes:
Yes, um, I kind of solved it for miniserv in the meantime ^^ But my solution is
specific to HTTP.

The solution is simply to do another attempt at receiving data right before
shutting down. ^^
Now, there are two possibilities.
a) The client transmits the header, then shuts down the connection
(SocketShutdown.SEND).
  In this case, the server-side recv call will return zero - connection closed.
  BUT, strangely, it will also block until all the missing data is sent.
b) The client transmits the header, then keeps the connection open (to send
another header?)
  In this case, we simply re-enqueue the socket and get persistent connections
practically for free :D

The respective miniserv code looks like thus:


     try {
       push; /// if this succeeds, we have a persistant connection on our hands
       scope(success) first(this, (BufferingLayer buf) { buf.push_back(recv);
logln("Re-enqueue!"); reenq(buf); });
     } catch (Exception e) close; /// nope; the client already shutdown his
SEND. Ah well. Terminate.


Push basically asks the lowest member of the filter-chain (the socket back-end)
to "push"
received data to the next higher element. It's basically a receive call.

Now, if there was a problem in-between, my socket back-end code will throw an
exception (after the blocking recv returns).
In this case, we simply close the socket and are done with it. Otherwise, we
add the additional data we received
to the front of our incoming buffer and re-enqueue the socket.

Easy, huh? ^^ I'm not sure how well this method translates to other protocols
though, mainly because I don't completely
understand why it works myself :p

Well, good luck ~
  -- downs, happy
May 12 2007
parent reply Regan Heath <regan netmail.co.nz> writes:
Downs Wrote:

 Yes, um, I kind of solved it for miniserv in the meantime ^^ But my solution
is specific to HTTP.
 
 The solution is simply to do another attempt at receiving data right before
shutting down. ^^
 Now, there are two possibilities.
 a) The client transmits the header, then shuts down the connection
(SocketShutdown.SEND).
   In this case, the server-side recv call will return zero - connection closed.
   BUT, strangely, it will also block until all the missing data is sent.
 b) The client transmits the header, then keeps the connection open (to send
another header?)
   In this case, we simply re-enqueue the socket and get persistent connections
practically for free :D
 
 The respective miniserv code looks like thus:
 
  >     try {
  >       push; /// if this succeeds, we have a persistant connection on our
hands
  >       scope(success) first(this, (BufferingLayer buf) {
buf.push_back(recv); logln("Re-enqueue!"); reenq(buf); });
  >     } catch (Exception e) close; /// nope; the client already shutdown his
SEND. Ah well. Terminate.
 
 Push basically asks the lowest member of the filter-chain (the socket
back-end) to "push"
 received data to the next higher element. It's basically a receive call.
 
 Now, if there was a problem in-between, my socket back-end code will throw an
exception (after the blocking recv returns).
 In this case, we simply close the socket and are done with it. Otherwise, we
add the additional data we received
 to the front of our incoming buffer and re-enqueue the socket.
 
 Easy, huh? ^^ I'm not sure how well this method translates to other protocols
though, mainly because I don't completely
 understand why it works myself :p


Well, I'm glad you're situation is resolved but I'm still puzzled as to what I
am/was doing wrong.  Maybe I'll take another look at it now, with fresh eyes,
to figure out where I am going wrong.

Regan Heath
May 13 2007
parent Regan Heath <regan netmail.co.nz> writes:
Aha!  Don't I feel like a fool.  The problems I was having were caused by the
Norton internet security product installed on this (not my own) machine.  After
disabling them it all worked as expected.
May 13 2007