digitalmars.D - Is there interest in a std.http?
- Tyler Jameson Little (38/38) Nov 19 2012 I'd like to see an HTTP module in Phobos, but I wanted to gauge
- Adam D. Ruppe (44/53) Nov 19 2012 I've been asked to put my cgi.d in phobos before (which includes
- Tyler Jameson Little (16/30) Nov 19 2012 I'm thinking of something a little more sophisticated than
- Adam D. Ruppe (13/16) Nov 19 2012 I've been using it for almost all my work apps for the last... I
- Pragma Tix (4/43) Nov 19 2012 Before you spend too much time :
- Vladimir Panteleev (15/20) Nov 19 2012 No Content-Size, no multiple requests per connection (unless you
- Tyler Jameson Little (25/44) Nov 19 2012 Not sure what you mean. I meant incoming stream. There would be a
- Nicolas Sicard (8/16) Nov 20 2012 BTW, the actual JSON module in viba.data.json in richer in
- Pragma Tix (4/23) Nov 20 2012 vibe.d is MIT licensed. (Like almost every other WEB related
- Rob T (6/12) Nov 20 2012 I have not yet looked at the vibe.d JSON yet, but I certainly
- Andrei Alexandrescu (19/21) Nov 19 2012 I can say the following. We sorely need a server expert on board with
- Andrei Alexandrescu (5/11) Nov 19 2012 To clarify (lest this would be interpreted as excluding vibe.d from the
- Adam D. Ruppe (25/32) Nov 20 2012 I agree with this but it really should be generic; working at
- H. S. Teoh (20/35) Nov 20 2012 The event loop itself should be a generic component in Phobos, not
- jerro (8/9) Nov 20 2012 Libev is a polling abstraction. It has select, epoll and kqueue
- Adam D. Ruppe (30/32) Nov 20 2012 I was thinking we'd send pointers and a type hash through a pipe.
- Adam D. Ruppe (15/17) Nov 20 2012 I have such a hard time focusing on work when there's something
- Dmitry Olshansky (43/64) Nov 20 2012 Having done quite some work on this before - both with frameworks and
- Dmitry Olshansky (5/8) Nov 20 2012 The obligatory link: https://netty.io/
- Jacob Carlborg (4/13) Nov 19 2012 * HTTPS
I'd like to see an HTTP module in Phobos, but I wanted to gauge interest first and see if this has been discussed before. An example of how I would like to interface with it (for creating a server): interface HTTPHandler { void serveHTTP(Request, Response); } class CustomHandler : HTTPHandler { .... void serveHTTP(Request req, Response res) { } } auto handler = new CustomHandler(); // implements http.Handler auto server = new HttpServer(handler); server.listen("0.0.0.0", 80); As long as serveHttp() is thread safe, the http server could be concurrent or evented (with libev or similar) and the custom handler code wouldn't have to change. I'm willing to put in the lion's share of the work (I've already written a bunch of it), but I'd naturally like to get some community input so I don't go in a completely wrong direction. I'm thinking of emulating Go's http library (http://golang.org/pkg/net/http), but of course D style. I think the following are necessary: * Access to underlying TcpSocket (for protocol upgrades, like WebSocket) * HTTP body is a stream * Simple HTTP requests * Errors can be recovered from (allow user-defined error handlers): * User settable size limits * Size of request line (1024 bytes by default) * Size of each header (1024 bytes by default) * Total size of header block (4096 bytes by default) So, basically I'm looking for two things: 1. Interest level: willing to writ code, willing to test, want to use 2. Suggestions for the API/behavior
Nov 19 2012
On Monday, 19 November 2012 at 20:38:56 UTC, Tyler Jameson Little wrote:I'd like to see an HTTP module in Phobos, but I wanted to gauge interest first and see if this has been discussed before.I've been asked to put my cgi.d in phobos before (which includes a http server as well as a cgi, fastcgi, and scgi implementation of its common interface), so there's probably some interest. I haven't put mine in just because I'm not particularly motivated to go through the red tape. The file is cgi.d in here: https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff Feel free to take whatever you want from it. There's also an extremely minimal http.d file in there, which is a very simple http client. (I didn't go far with this though because I tend to use curl for most my work anyway.) The way I did it is with a function: void handleRequest(Cgi cgi) { } And then there's a mixin called GenericMain that calls your function. mixin GenericMain!handleRequest; Depending on how you compile it, the main function constructs the cgi object differently. -version=fastcgi means it is plugged into a FastCGI loop. version=scgi ditto. -version=embedded_httpd includes a simple little threaded http server. No version builds it as a standard CGI binary, one process per request. However it is compiled though, the Cgi methods all remain the same, so your function shouldn't have to change.* Access to underlying TcpSocket (for protocol upgrades, like WebSocket)My cgi.d I don't think can do this yet. I've been thinking about adding it though, but haven't.* HTTP body is a streamclass Cgi has a method onRequestBodyDataReceived and one handleIncomingDataChunk which can be overridden by subclasses to do this kind of thing. It's a bit of a pain though, I made it thinking you'd want to handle it after it is all loaded.* Simple HTTP requestsThis is where I think my cgi.d is a winner because most the data is available so easily cgi.get["foo"] returns a string cgi.getArray["foo"] returns an array of strings ditto for post cgi.request("foo", 0) returns with a default value, converting types as requested There's also members like cookie, queryString, host, etc., that gives nice access to other headers.* Errors can be recovered from (allow user-defined error handlers): * User settable size limitseh, I'm partially there on these. There's some constructor args you can play with and some degree of exception catching but probably not fully what you have in mind.
Nov 19 2012
I've been asked to put my cgi.d in phobos before (which includes a http server as well as a cgi, fastcgi, and scgi implementation of its common interface), so there's probably some interest. I haven't put mine in just because I'm not particularly motivated to go through the red tape. The file is cgi.d in here: https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuffAwesome. I assume this hasn't gone through rigorous testing, but has it been used in production?I'm thinking of something a little more sophisticated than exceptions. Exceptions unwind the stack, so you can't just continue where you left off. How do you feel about function pointer callbacks? The http parser in node.js (https://github.com/joyent/http-parser) takes a struct of function pointers to handle errors/events. When the parser encounters a recoverable error (max header length reached), the programmer could opt to ignore it and continue the parse. Unrecoverable errors could throw exceptions (like trying to parse garbage data). If an error is recovered, it would just continue as if nothing happened. Of course, implementing this would increase complexity in the parser, but I think it's justified. Thoughts?* Errors can be recovered from (allow user-defined error handlers): * User settable size limitseh, I'm partially there on these. There's some constructor args you can play with and some degree of exception catching but probably not fully what you have in mind.
Nov 19 2012
On Monday, 19 November 2012 at 23:57:35 UTC, Tyler Jameson Little wrote:Awesome. I assume this hasn't gone through rigorous testing, but has it been used in production?I've been using it for almost all my work apps for the last... I think three years now. None of them so far have huge userbases, but it has been in steady use for a while. The others in this thread have pointed out the other options that have some users: Vladimir's library that runs the forum, and vibe.d that seems to be the most popular. I haven't used either of them though.How do you feel about function pointer callbacks?We could also do them as virtual functions... that's a little easier in the setup I have because the constructor arguments are messy, whereas subclassing is a lot simpler. But yeah, it sounds like a decent plan.
Nov 19 2012
Before you spend too much time : https://github.com/rejectedsoftware/vibe.d/tree/master/source/vibe/http On Monday, 19 November 2012 at 20:38:56 UTC, Tyler Jameson Little wrote:I'd like to see an HTTP module in Phobos, but I wanted to gauge interest first and see if this has been discussed before. An example of how I would like to interface with it (for creating a server): interface HTTPHandler { void serveHTTP(Request, Response); } class CustomHandler : HTTPHandler { .... void serveHTTP(Request req, Response res) { } } auto handler = new CustomHandler(); // implements http.Handler auto server = new HttpServer(handler); server.listen("0.0.0.0", 80); As long as serveHttp() is thread safe, the http server could be concurrent or evented (with libev or similar) and the custom handler code wouldn't have to change. I'm willing to put in the lion's share of the work (I've already written a bunch of it), but I'd naturally like to get some community input so I don't go in a completely wrong direction. I'm thinking of emulating Go's http library (http://golang.org/pkg/net/http), but of course D style. I think the following are necessary: * Access to underlying TcpSocket (for protocol upgrades, like WebSocket) * HTTP body is a stream * Simple HTTP requests * Errors can be recovered from (allow user-defined error handlers): * User settable size limits * Size of request line (1024 bytes by default) * Size of each header (1024 bytes by default) * Total size of header block (4096 bytes by default) So, basically I'm looking for two things: 1. Interest level: willing to writ code, willing to test, want to use 2. Suggestions for the API/behavior
Nov 19 2012
On Monday, 19 November 2012 at 20:38:56 UTC, Tyler Jameson Little wrote:* HTTP body is a streamNo Content-Size, no multiple requests per connection (unless you use chunked encoding?).* User settable size limits * Size of request line (1024 bytes by default) * Size of each header (1024 bytes by default) * Total size of header block (4096 bytes by default)I think this is pointless. Use an appender and the RFC limits. Unless you are serving VERY simple pages, the cost of a few allocations for handling the HTTP protocol's overhead will not be noticeable compared to the application's. (Slowloris is something to keep in mind, though.) My HTTP library, which is used for this forum and several other "production" projects, is here: https://github.com/CyberShadow/ae The main problem with getting it into Phobos is that it's tied to a lot of other related code, such as the asynchronous sockets module, the unmanaged memory wrapper type, etc. Also, what about vibe.d?
Nov 19 2012
Not sure what you mean. I meant incoming stream. There would be a request object with access to all other headers, it just wouldn't be read in until the user actually wanted it.* HTTP body is a streamNo Content-Size, no multiple requests per connection (unless you use chunked encoding?).I just made up those limits, but the point is I'd like the user to be able to tweak those. The default will work for most people though.* User settable size limits * Size of request line (1024 bytes by default) * Size of each header (1024 bytes by default) * Total size of header block (4096 bytes by default)I think this is pointless. Use an appender and the RFC limits. Unless you are serving VERY simple pages, the cost of a few allocations for handling the HTTP protocol's overhead will not be noticeable compared to the application's. (Slowloris is something to keep in mind, though.)My HTTP library, which is used for this forum and several other "production" projects, is here: https://github.com/CyberShadow/ae The main problem with getting it into Phobos is that it's tied to a lot of other related code, such as the asynchronous sockets module, the unmanaged memory wrapper type, etc.Cool, I didn't know that this was served by D code. I'll take a look at it.Also, what about vibe.d?vibe.d does a lot of things, and it probably does those things very well. It seems well maintained, and generally a good project. I think that a lot of the things it does well could be brought into the standard library. As this thread pointed out, there are several HTTP parsers floating around out there. For some, vibe.d might not be a perfect fit for whatever reason, but everyone can benefit from a simple HTTP library. Maybe it shouldn't be as high-level as Go's http library, but it should at least make writing a simple HTTP server trivial. Would a minor refactor of vibe.d be acceptable? This is pretty much what I'm looking for: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/common.d https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d Except that I'd remove some parts of it, like JSON parsing. I think that vibe.d could benefit from moving some of the code there into Phobos. I guess it comes down to whether it makes sense to make a standard HTTP library.
Nov 19 2012
On Tuesday, 20 November 2012 at 03:49:57 UTC, Tyler Jameson Little wrote:Would a minor refactor of vibe.d be acceptable? This is pretty much what I'm looking for: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/common.d https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d Except that I'd remove some parts of it, like JSON parsing. I think that vibe.d could benefit from moving some of the code there into Phobos. I guess it comes down to whether it makes sense to make a standard HTTP library.BTW, the actual JSON module in viba.data.json in richer in functionality and easier to use than the one in std.json. The module has virtually no dependency on other parts of vibe.d. I whish it was also moved to Phobos or at least inspired a refactoring of std.json. Nicolas
Nov 20 2012
On Tuesday, 20 November 2012 at 12:12:27 UTC, Nicolas Sicard wrote:On Tuesday, 20 November 2012 at 03:49:57 UTC, Tyler Jameson Little wrote:vibe.d is MIT licensed. (Like almost every other WEB related library) Guess you have to ask Soenke for a permission.Would a minor refactor of vibe.d be acceptable? This is pretty much what I'm looking for: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/common.d https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d Except that I'd remove some parts of it, like JSON parsing. I think that vibe.d could benefit from moving some of the code there into Phobos. I guess it comes down to whether it makes sense to make a standard HTTP library.BTW, the actual JSON module in viba.data.json in richer in functionality and easier to use than the one in std.json. The module has virtually no dependency on other parts of vibe.d. I whish it was also moved to Phobos or at least inspired a refactoring of std.json. Nicolas
Nov 20 2012
On Tuesday, 20 November 2012 at 12:12:27 UTC, Nicolas Sicard wrote:BTW, the actual JSON module in viba.data.json in richer in functionality and easier to use than the one in std.json. The module has virtually no dependency on other parts of vibe.d. I whish it was also moved to Phobos or at least inspired a refactoring of std.json. NicolasI have not yet looked at the vibe.d JSON yet, but I certainly will. I agree with you that the std.json could use some improvement, it's not all that fun to use that's for sure. --rt
Nov 20 2012
On 11/19/12 3:38 PM, Tyler Jameson Little wrote:I'd like to see an HTTP module in Phobos, but I wanted to gauge interest first and see if this has been discussed before.I can say the following. We sorely need a server expert on board with the time and inclination to write a good server-side networking framework. We have a couple, but for various reasons I have been unable to convince them (through newsgroup and private exchanges) to invest time in such an endeavor. That's mostly because lack of time - speaking for myself, I wish I found the time to learn about all that, which as far as I understand gravitates around notions such as asynchronous I/O, libevent/libevt, select(), and the such. I've never used them so I'd need to start from scratch. We really need an expert. Look at the Go programming language - it's not remarkable, but it benefits of full-time dedication of experts in server-side programming, so it has acquired excellent library support for networking servers. And they milk that for all it's worth: any discussion, article, or blog post about Go gravitates toward the five-lines HTTP server with the same implacable reach as conversations with ideologists, which inevitably converge towards their ideological stronghold. Andrei
Nov 19 2012
On 11/19/12 11:26 PM, Andrei Alexandrescu wrote:On 11/19/12 3:38 PM, Tyler Jameson Little wrote:To clarify (lest this would be interpreted as excluding vibe.d from the category of good server-side networking frameworks): ... for inclusion in the standard library. AndreiI'd like to see an HTTP module in Phobos, but I wanted to gauge interest first and see if this has been discussed before.I can say the following. We sorely need a server expert on board with the time and inclination to write a good server-side networking framework.
Nov 19 2012
On Tuesday, 20 November 2012 at 04:26:02 UTC, Andrei Alexandrescu wrote:We sorely need a server expert on board with the time and inclination to write a good server-side networking framework.I agree with this but it really should be generic; working at probably the file descriptor level, so we can do whatever protocols along with files, pipes, ... and ideally, any kind of injected event. I'd really like it if we could run one event loop and be able to do keyboard/joystick input, timers, GUI, threads, etc., including third party libraries, as well as networking. (Combining two different event loops is a pain in the butt.) I have an implementation in mind (I'd use select() because I don't know libev, but we should be able to change that without changing the interface)... ... but I'm behind on my other work and have yet another deadline looming so can't do it myself right now. I can't wait till I'm retired!And they milk that for all it's worth: any discussion, article, or blog post about Go gravitates toward the five-lines HTTP server with the same implacable reach as conversations with ideologists, which inevitably converge towards their ideological stronghold.You know, I'll never understand this. Anybody can do a five line http server. Hell, with my lib: import arsd.cgi; void hello(Cgi cgi) { cgi.write("hello!"); } mixin GenericMain!hello; Look, tiny http server (on port 8080)! Also works as cgi, fastcgi, scgi, and easy command line testing with a different compile option. But not really that special language wise... yay, you can call a library function. Who can't do that?
Nov 20 2012
On Tue, Nov 20, 2012 at 03:09:23PM +0100, Adam D. Ruppe wrote:On Tuesday, 20 November 2012 at 04:26:02 UTC, Andrei Alexandrescu wrote:The event loop itself should be a generic component in Phobos, not specific to anything else. It should work at the file descriptor level, perhaps even combining with signal handling (using the self-pipe trick), and handle timers and scheduled events as well. Everything else should be pluggable into this generic module. I have written such things before in C/C++, but in D this will be much easier to implement, thanks to delegates (which make writing event-loop based programs so much less painful). Having a standard event loop will make it much less painful to integrate many components together in a seamless way. Combining two event loops, as Adam says, is a royal pain in the neck. Making the generic event loop a Phobos standard will eliminate this problem.We sorely need a server expert on board with the time and inclination to write a good server-side networking framework.I agree with this but it really should be generic; working at probably the file descriptor level, so we can do whatever protocols along with files, pipes, ... and ideally, any kind of injected event. I'd really like it if we could run one event loop and be able to do keyboard/joystick input, timers, GUI, threads, etc., including third party libraries, as well as networking. (Combining two different event loops is a pain in the butt.)I have an implementation in mind (I'd use select() because I don't know libev, but we should be able to change that without changing the interface)...The interface should be generic enough that you don't have to worry about whether select or poll (or whatever the windows equivalent is) is used. I think it's a good idea to implement the select version first, as that's generic across Posixes, whereas libev is Linux-specific IIRC. T -- 2+2=4. 2*2=4. 2^2=4. Therefore, +, *, and ^ are the same operation.
Nov 20 2012
whereas libev is Linux-specific IIRC.Libev is a polling abstraction. It has select, epoll and kqueue backends (and there may be others I don't know about). So it is particulary useful for unices, where epoll (linux) and kqueue (OSX and BSD) are preferred async IO mechanisms. It can also be used on Windows, but with the select() backend, which doesn't scale well with many file descriptors. select() is also limited to 64 file descriptors by default on windows IIRC (that limit can be raised, but only to 4096 if I am not mistaken).
Nov 20 2012
On Tuesday, 20 November 2012 at 15:21:12 UTC, H. S. Teoh wrote:It should work at the file descriptor level, perhaps even combining with signal handling (using the self-pipe trick)I was thinking we'd send pointers and a type hash through a pipe. It'd keep the read/write very simple. Now, the event data must be defined to be transient, unless specifically told otherwise (e.g. immutable types), because then we can reuse the data structures to avoid allocations. But take a gander at this: http://arsdnet.net/dcode/typehash.d We can take any type tuple, do a mangle of and get a unique name for any combination of arguments. Then hash it, and we have a nicely unique, fixed size message to send down the pipe. One one end, we'll cast to a void* and send that along with the hash. On the other side, only listener functions who's ParameterTypeTuple matches the typehash will receive a message. They can then cast the void* back to the argument type with confidence. This lets us define custom messages as plain types: struct MyMessage { string text; } event.listen((MyMessage m) { writeln("Received: ", m.text); } event.send(MyMessage("hello!")); event.loop(); // we should receive the message and call the above func If there's no listener that matches the hash, we ignore the message. Now there'd be some care needed to not send pointers to stack data down the pipe, as that could be destroyed, but eh. Some details gotta be worked out but I really like the idea of using type names like this.
Nov 20 2012
On Tuesday, 20 November 2012 at 16:09:37 UTC, Adam D. Ruppe wrote:But take a gander at this: http://arsdnet.net/dcode/typehash.dI have such a hard time focusing on work when there's something like this to play with.... updated the link to show the concept actually working, not just hashing. All event listeners must take exactly one argument and it is all statically typed, but it works and can do the hash+pointer message. The hash function down there is borrowed from druntime. I couldn't find it accessible as a stdlib import so I copy/pasted. But ignoring that, the rest is actually simple code. Problem is this probably isn't safe... and might not be trusted either because you can send stack info.. maybe we could do further constraints, or copy the data into a queue, or something. I really think we could make it work though and it'd be pretty cool.
Nov 20 2012
11/20/2012 8:26 AM, Andrei Alexandrescu пишет:On 11/19/12 3:38 PM, Tyler Jameson Little wrote:Having done quite some work on this before - both with frameworks and barehanded. The recipe for scalable and fast server itself is not that hard: asynchronous I/O + event-based notification + separate thread pool for user-defined handlers Asynchronous I/O is well understood to have far less overhead then thread per client/descriptor model. Both in terms of memory usage and time spent on the context switching. Event based notification can be traded for select() polling but then you'd waste a lot of cycles (in kernel mode) walking through huge descriptor sets when number of simultaneous clients is high enough. A separate "worker" thread pool insures your event-loop is not ever blocked thus insuring responsiveness w.r.t dispatching I/O requests. Arguably this could be an opt-in. If this combo is implemented even half-decently it should give us great speed. The fine-tuning and topping the performance however is the other 70% of work and that makes the real difference. The devil is in the details. Not to mention fiddling with socket options and imposing limits on transfer rate/timeouts etc. I currently had to work with Java (a painful journey after D) to create a couple of server components and surprisingly found the Netty project to be quite nice. Though being overly Java-escue it has some great ideas (aside from the above basic concepts): - The composable pipeline abstraction that allows passing user data through a series of transformations seamlessly. Think the stack of TCP|HTTP|encryption/compression or TCP|framing|ProtocolBuffers etc. - Flexible and zero-copy (quite hard for Java I guess) buffer abstraction. It reminds the power of ranges but in a heck of a more limited scale - it's more about chaining and slicing. These limitations are mostly because of the way the OS works.I'd like to see an HTTP module in Phobos, but I wanted to gauge interest first and see if this has been discussed before.I can say the following. We sorely need a server expert on board with the time and inclination to write a good server-side networking framework. We have a couple, but for various reasons I have been unable to convince them (through newsgroup and private exchanges) to invest time in such an endeavor. That's mostly because lack of time - speaking for myself, I wish I found the time to learn about all that, which as far as I understand gravitates around notions such as asynchronous I/O, libevent/libevt, select(), and the such. I've never used them so I'd need to start from scratch.We really need an expert. Look at the Go programming language - it's not remarkable, but it benefits of full-time dedication of experts in server-side programming, so it has acquired excellent library support for networking servers.Can help with a proper asynchronous (network) I/O support on Windows. I doubt I'll find time to flesh out the whole design though. What it irks me that I haven't seen a good enough backend for Windows in the open source (of those I've peeked at the source of). A lot of event-based backends seem to just ignore this OS or fallback to select-loop. Meanwhile M$ introduced the new RIO socket API targeted at high-load servers. Was planing to try it out since the day it was introduced.And they milk that for all it's worth: any discussion, article, or blog post about Go gravitates toward the five-lines HTTP server with the same implacable reach as conversations with ideologists, which inevitably converge towards their ideological stronghold.I see no reason we can't beat them in their favorite game. -- Dmitry Olshansky
Nov 20 2012
11/20/2012 11:45 PM, Dmitry Olshansky пишет: [snip]I currently had to work with Java (a painful journey after D) to create a couple of server components and surprisingly found the Netty project to be quite nice.The obligatory link: https://netty.io/ -- Dmitry Olshansky
Nov 20 2012
On 2012-11-19 21:38, Tyler Jameson Little wrote:I think the following are necessary: * Access to underlying TcpSocket (for protocol upgrades, like WebSocket) * HTTP body is a stream * Simple HTTP requests * Errors can be recovered from (allow user-defined error handlers): * User settable size limits * Size of request line (1024 bytes by default) * Size of each header (1024 bytes by default) * Total size of header block (4096 bytes by default)* HTTPS -- /Jacob Carlborg
Nov 19 2012