www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Needing help with basic HTTP requests..

reply "AlexMcP" <derptyderp hotmail.co.uk> writes:
Hi guys,

I've only been trying to learn D for a short while, and compared 
to the other more popular programming languages, I've been able 
to find very little helpful and updated documentation or 
tutorials on it.

I basically need help getting data from a website, just the HTML 
of a webpage, in this case a PHP page. I'm using the following 
code which compiles fine, but when run it just encounters a 
'Socket.ERROR' with a value of -1.

I'm not sure what I'm doing wrong, but any help or suggestions 
would be appreciated. As I said, I'm quite new to D...

---
//readData.d
module main;

import std.stdio;
import std.socket;
import std.cstream;

int main(string[] args)
{
	string host = "www.dprogramming.com";
	ushort port = 80;
	
	Socket listener = new TcpSocket;
         assert(listener.isAlive);
         listener.blocking = false;
	
	listener.connect(new InternetAddress(host, port));
	
	char[] msg;
	char[] req = cast(char[]) "GET /internet.php HTTP/1.1\r\nHost: 
www.dprogramming.com\r\n\r\n";
	
	listener.send(req);
	
	while (1)
	{
		int bytes;
		char[1024] buf;
		bytes = listener.receive(buf);
		if (Socket.ERROR == bytes)
		{
			writeln("Connection Error ", Socket.ERROR);
			listener.close();
			din.readLine();
			return 1;
		}
		else if (0 == bytes)
		{
			try
			{
				writeln("Connection to target lost.");
			}
			catch (SocketException)
			{
				writeln("Connection closed.");
			}
			listener.close();
			din.readLine();
			return 1;
		}
		else
		{
			writeln("Recieved ", bytes, "bytes from ", 
listener.remoteAddress().toString(), ": \"", buf[0 .. bytes], 
"\"");
		}
	}
}
---

- Alex.
Jul 09 2013
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 9 July 2013 at 09:03:32 UTC, AlexMcP wrote:

 			writeln("Connection Error ", Socket.ERROR);


First tip: Socket.ERROR is a constant, so printing it doesn't  
help much (as you probably noticed). More helpful is the function 
lastSocketError(), which returns a string with some details.

                         writeln("Connection Error ", 
Socket.ERROR, " ", lastSocketError());

Which tells you:

Connection Error -1 Resource temporarily unavailable



The reason that is an error is because you turned off blocking.

         listener.blocking = false;


So, since the connection hasn't had a chance to get established 
and transfer data yet, instead of waiting for it to finish when 
you called receive, it returned the error telling you to try 
again later.

The simplest thing to do here is to comment out that blocking = 
false line, letting receive just wait until data is available. 
Another option is to use Socket.select to notify you when data is 
available, and only receive then.

After that, make sure the req literal is all on one line to avoid 
http 400 errors (not sure if that was a result of copy/paste or 
not, probably) and you should get a response.


Another note: casting a string literal to char[] is usually a bad 
idea, because you can't write to them anyway. Best to just leave 
them typed as string. It should all still work the same after 
doing that.



Finally, if you want a higher level library to do this, check out 
std.net.curl: http://dlang.org/phobos/std_net_curl.html
Jul 09 2013
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
BTW I also wrote a more complete lower level implementation too:

https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff/blob/master/http.d

My code is kinda ugly, but might be useful to refer to too. http 
has some other features you'll need to think about, like chunked 
data transfer, to have a more complete implementation, and I have 
some openssl code in there too. Of course, curl takes care of all 
this too.
Jul 09 2013
parent "AlexMcP" <derptyderp hotmail.co.uk> writes:
Thank you! It worked, finally, and I can't believe that was the 
only thing stopping it from working. But yeah, I'll take your 
advice and make the changes.

I've also looked at curl and your implementation, but I wanted to 
give this a go manually instead of just using external libraries, 
I feel it's just better to learn how to do it all from scratch.

So thank you again, and you can probably expect me to be back 
here in the near future!

- Alex.
Jul 09 2013