www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Network in phobos

reply Adam Ruppe <destructionator gmail.com> writes:
Something that I think would be nice is being able to treat a network
connection the same way we can work with files and stdin.

Is there a plan already in place for phobos? If no, I might have
something usable to get started.

I peered into std.stdio's source and hacked something up. My plan was
simple: open the network, then assign the FILE* to it to the File
struct.

This is just for Linux; I haven't done Windows code like this for a
long time and I don't remember how. But this might be a useful
starting point anyway, so I figure I'll share it.


What I'd ultimately like is something like this for all D platforms,
and an add-on facility for handling multiple connections, including
incoming connections. I actually have something for this too, but it
is incompatible with the File struct (and Linux only again).


=========
import std.stdio;
import std.string;

import std.conv;

version(linux):

import std.c.linux.linux;
import std.c.linux.socket;

alias std.c.linux.socket sock;
alias std.c.linux.linux linux;

enum int PF_INET = 2;
enum int AF_INET = PF_INET;

extern(C) FILE* fdopen(int, const(char)*);

File openNetwork(string host, ushort port) {
	hostent* h;
	sockaddr_in addr;

	h = gethostbyname(std.string.toStringz(host));
	if(h is null)
		throw new StdioException("gethostbyname");

	int s = socket(PF_INET, SOCK_STREAM, 0);
	if(s == -1)
		throw new StdioException("socket");

	scope(failure)
		close(s);

	addr.sin_family = AF_INET;
	addr.sin_port = htons(port);
	std.c.string.memcpy(&addr.sin_addr.s_addr, h.h_addr, h.h_length);

	if(sock.connect(s, cast(sockaddr*) &addr, addr.sizeof) == -1)
		throw new StdioException("Connect failed");

	FILE* fp = fdopen(s, "w+".ptr);

	File f;

	auto imp = new File.Impl(fp, 1, host ~ ":" ~ to!string(port));

	f.p = imp;

	return f;
}
========
Mar 27 2010
next sibling parent Bane <branimir.milosavljevic gmail.com> writes:
Adam Ruppe Wrote:


 Something that I think would be nice is being able to treat a network
 connection the same way we can work with files and stdin.
 


I think I saw exactly that in Tango lib, maybe you can save time by looking it
up. I don't know for sure, newer used it.
Mar 28 2010
prev sibling next sibling parent reply "Igor Lesik" <curoles yahoo.com> writes:
"Adam Ruppe"

 Something that I think would be nice is being able to treat a network
 connection the same way we can work with files and stdin.


IMHO, such oversimplified approach is not very useful.
Yes, it would be so nice to have a good network library, but many things
have to be considered in order to cook decent library. One important
aspect is support for asynchronous calls; for example TCP/UDP stuff
in Boost is placed in to ASIO package.


 This is just for Linux; I haven't done Windows code like this for a
 long time and I don't remember how. But this might be a useful
 starting point anyway, so I figure I'll share it.


As long as you use only socket API there is no problem to write generic
code for Linux and Windows. Windows socket API even has select() function.

Igor
Mar 28 2010
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sun, Mar 28, 2010 at 12:37:14PM -0700, Igor Lesik wrote:

 IMHO, such oversimplified approach is not very useful.


Depends on your needs. A lot of time, I just want to quickly whip
something together that fetches something, and I don't care how long
it takes*. It's the kind of thing I'd use the nc program on the linux
command line to do, but right there in the D.

* If I do care how long it takes, I'll just fork() the whole program and
let the child to it. I'm thinking about making a generic background!(delegate)
template that can do this automatically, but D2 should have threads that work
just as well soon anyway.


One example is I needed something very simple that speaks SMTP. I wrote
it in my fancier network library, and it was a fair amount of code that
was a bit overcomplicated.

With this kind of thing, it is a fairly brief list of writefln and readln
lines.


 Yes, it would be so nice to have a good network library, but many things
 have to be considered in order to cook decent library. One important
 aspect is support for asynchronous calls; for example TCP/UDP stuff
 in Boost is placed in to ASIO package.



In my fancier library <http://arsdnet.net/dcode/netman.d>, which needs some
work still to finish, but is usable, all write and read calls go to a buffer.

By the way, everything in my dcode directory there is free for the taking
if anyone wants it.

Anyway, you do something like this:

class MyConnection : Connection {
    override void onDataReceived() {
        auto a = read(); // gets the buffer of new data

        write("some data"); // puts that in the outgoing buffer. not
			   // actually written until the function returns

    	changeReadPosition(pos); // discards [0..pos] of the internal
    				// buffer, so the next time you get
    				// called, you deal with leftover info.
    				// good for handling incomplete requests
	}
}

The incoming and outgoing buffers aren't actually put on the network until
the next call to NetworkManager.proceed(). So your main has to look something
like this:

void main() {
	auto netman = new NetworkManager;
	netman.connect("site.com", 80);
	while(netman.proceed()) { }
}

The nice thing about the network manager class is it does incoming connections
too:

void main() {
	auto netman = new NetworkManager;
	netman.listen(80);
	while(netman.proceed() { }
}


And each incoming connection is a particular specialization of the Connection
class (which you create in a delegate, not shown here), which handles all the
work. netman.proceed() calls select() on all its sockets, then calls the
appropriate handler based on what happens: onDataReceived, onDisconnect, etc.





The downside is my read() and write() functions are very simple, and kinda
suck, and every call is async, so even if you don't care about speed, you
have to code it with all your functions returning instantly.

The various functions and ranges in the stdio File are much easier to work
with for lots of things.

Ideally, what I think would be nice is having File available for when you
want it, and then being able to attach it to a NetworkManager kind of thing
which adds async calls to the interface.

Or something like that. Maybe the one size can't really fit all, but I'd like
to have both options available somehow.


 As long as you use only socket API there is no problem to write generic
 code for Linux and Windows. Windows socket API even has select() function.


Cool. Most my code is just simple socket stuff, so it should be easy to
port then.


-- 
Adam D. Ruppe
http://arsdnet.net
Mar 29 2010
prev sibling parent reply BCS <none anon.com> writes:
Hello Adam,


 Something that I think would be nice is being able to treat a network
 connection the same way we can work with files and stdin.


how about this:

http://www.digitalmars.com/d/1.0/phobos/std_socketstream.html
or
http://www.digitalmars.com/d/2.0/phobos/std_socketstream.html


-- 
... <IXOYE><
Mar 28 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 03/28/2010 05:07 PM, BCS wrote:

 Hello Adam,


 Something that I think would be nice is being able to treat a network
 connection the same way we can work with files and stdin.


 how about this:

 http://www.digitalmars.com/d/1.0/phobos/std_socketstream.html
 or
 http://www.digitalmars.com/d/2.0/phobos/std_socketstream.html


I think Adam's implementation is more interesting because it integrates 
with File and consequently with the ranges that will come on top of it 
(e.g. byCharacter, byLine, byChunk).

Thanks Adam. If you're willing to make your implementation available to 
Phobos, you may want to put it in bugzilla.


Andrei
Mar 28 2010
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sun, Mar 28, 2010 at 05:43:22PM -0500, Andrei Alexandrescu wrote:

 I think Adam's implementation is more interesting because it integrates 
 with File and consequently with the ranges that will come on top of it 
 (e.g. byCharacter, byLine, byChunk).


Yes, that's what I really wanted (and write/readln). stdin.byLine is
possibly the greatest thing ever, and extending it to network connections,
and other processes too, is a nice thing to have available.

One size doesn't fit all


 Thanks Adam. If you're willing to make your implementation available to 
 Phobos, you may want to put it in bugzilla.


Yes, certainly: http://d.puremagic.com/issues/show_bug.cgi?id=4025

I've written a HTTP client on top of it too.
http://arsdnet.net/dcode/http.d
(the import network refers to the module now in bugzilla)

I'm not completely happy with it yet, but it works pretty well anyway. The
File based code is much simpler than the raw socket code I used to use, though
the downside is File's calls block. OK for me, but probably not for everyone.

-- 
Adam D. Ruppe
http://arsdnet.net
Mar 28 2010