www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - web development in D

reply joe <joe studiofortress.com> writes:
I currently do most of my web development in PHP, with some work in Ruby with
RoR. Right now I'm starting to think about building my own stack for web dev
(I already use my own MVC framework and libs in PHP), but I'd really like to
move to something faster and more powerful. Java or ASP.NET are two obvious
choices, but I'm currently researching other alternatives (like D).

So first I was wondering if there has been any web development using D, and if
there are many (or any) libraries/frameworks/other resources already available
for this?

Second, from your own experience using D do think it would make a good or bad
choice for a web development language?
May 21 2011
next sibling parent Trass3r <un known.com> writes:
Have a look at the recent thread titled 'How To Dynamic Web Rendering?'
Adam Ruppe has created a package for web development with D and it seems  
to work like a charm.
May 21 2011
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Yea, the Dynamic Web Rendering thread just went over this, so
shouldn't be hard to find that thread.

It went into discussing my method and got a little long, but the
summary is:

You can write web apps in D using the standard CGI interface or
a long lived process (embedded http server or whatever.)

My code works best with standard CGI, cgi.d in here:
http://arsdnet.net/dcode/

I also have an httpd.d (requires netman.d) that lets the cgi
interface work with an embedded server, but it's single threaded
so that can be a problem. It's not production ready.

Regular cgi though works with other existing web servers, is
easy to use, and actually performs quite well.

(You'll find a lot of people complaining on the internet that
CGI is slow, but check the date on those articles and see if they
are using Perl or native compiled binaries.

Usually they are very old and using an interpreter - that's why
it is slow. A native binary over cgi is quite fast.)



The stuff you build on top of cgi can be whatever you want. I
personally use my dom.d and web.d modules for templating and
some automatic code generation, but you can spin your own libs
too.
May 21 2011
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sat, 21 May 2011 19:18:39 +0300, Adam D. Ruppe  
<destructionator gmail.com> wrote:


 (You'll find a lot of people complaining on the internet that
 CGI is slow, but check the date on those articles and see if they
 are using Perl or native compiled binaries.

 Usually they are very old and using an interpreter - that's why
 it is slow. A native binary over cgi is quite fast.)


Regular CGI still has the overhead of starting a new process for every  
request. PHP without CGI is likely to be faster than D with CGI.

OP is looking for best performance, so CGI is not a good recommendation -  
FastCGI/webserver module/built-in webserver are the way to go.

I've had great experience with using an HTML/jQuery frontend with a D  
backend (built-in HTTP server which only serves static files and AJAX  
requests).

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 22 2011
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Vladimir Panteleev wrote:

 Regular CGI still has the overhead of starting a new process for
 every request.


That's a very small cost - between 1 and 5 milliseconds, depending
on your hardware.

Next to the network latency involved in making a request, that's
literally nothing - it's less than random network variation, so it
has no impact on the end user experience.

It also disappears quickly if your program does any processing,
since it's like a line: y = mx + b. The startup cost is b, and
the speed of the language is m.

If x is even slightly big, reducing m means you'll get better
end results than reducing b. (unless b is *huge*... like network
latency. Adding an "Expires:" header to your responses gives the
biggest gains of all.)

My work application runs about 40% faster on CGI than the fastest
comparable product we've found in mod_php, on average, with certain
CPU intensive portions being up to 5x faster.

That said, an embedded webserver /is/ even faster, but it's
also harder to use.

The end result depends on your app. Only actually benchmarking it
can tell for sure what matters. My cgi.d provides the same interface
regardless of how it is called - switching to built in httpd is
easy if needed.



 I've had great experience with using an HTML/jQuery frontend with
 a D backend


From what I've seen in my benchmarks, you would probably get
a bigger usability boost by ditching jQuery than by ditching CGI.
Even when cached, jQuery takes a while for the browser to parse -
comparable to starting up a native process on the server.
May 22 2011
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I've never done any CGI stuff before, but can't you employ some kind
of messaging mechanism with an already running process? Maybe use
memory-mapped files or some other technique to communicate with the
process?
May 22 2011
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
 I've never done any CGI stuff before, but can't you employ some kind
 of messaging mechanism with an already running process?


Yes. I find it's quite useful when you want instant notifications
on something, like a web chat application, but it could do some
speed boosts too.

I'm working on a post showing some of my benchmarks too on built-in
vs cgi. Some of them are over 20x faster with a built in webserver!
(I wonder if this is cgi's fault or if Apache is just not optimized
for it... I'm changing the i/o code too so we can make sure it's
not just stdio being slow.)
May 22 2011
parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 22/05/2011 19:11, Adam D. Ruppe wrote:

 I've never done any CGI stuff before, but can't you employ some kind
 of messaging mechanism with an already running process?


 Yes. I find it's quite useful when you want instant notifications
 on something, like a web chat application, but it could do some
 speed boosts too.

 I'm working on a post showing some of my benchmarks too on built-in
 vs cgi. Some of them are over 20x faster with a built in webserver!
 (I wonder if this is cgi's fault or if Apache is just not optimized
 for it... I'm changing the i/o code too so we can make sure it's
 not just stdio being slow.)


Have you tried FastCGI? It's essentially CGI but with a long running 
process rather than re-spawning for each request (and it still has the 
advantage that it can be re-spawned by the web server). I'm using it for 
serenity, albeit the FCGX interface rather than the interface that 
emulates CGI.

-- 
Robert
http://octarineparrot.com/
May 22 2011
parent Adam D. Ruppe <destructionator gmail.com> writes:
Robert Clipsham wrote:

 Have you tried FastCGI?


I haven't. I read about it but concluded it would actually be just
about as difficult to implement as a minimalist http server, so
I never went into it.

(http itself is hard to get all the corner cases right, but if you
live behind something like Apache, you can get by without a full
implementation.

But in both cases, you have to write a fast network layer, and
that's the harder part.)


Besides though, there's a penalty between D in CGI and D with
long lived processes.... but D in CGI beats PHP a lot of the
time, so it still wins.

I wonder how D/cgi stacks up against a long lived Python or
Ruby process. I doubt they'd do better than embedded PHP, but
I haven't tried them so I'm not sure.
May 22 2011
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
I wrote:

 That said, an embedded webserver /is/ even faster


I measured this using one of my in-development apps recompiled
to use the built in server I have, picking a page with moderate
processing requirements.

CGI:
Time per request:       17.375 [ms] (mean)

built in webserver:
Time per request:       8.727 [ms] (mean)

I wonder why the difference is bigger than the startup I measured
when isolated? I think I/O contributes to slowdown a little too.
My startup cost was based on hello world, no significant output.
This outputs several kilobytes of data.

That's without any optimization. Let's try that again, making the
mysql and reflection objects static so they needn't be recreated
on every request when embedded....

Cgi:
Time per request:       17.178 [ms] (mean)

Built-in:
Time per request:       5.079 [ms] (mean)


Very nice.


Those are all pretty small numbers, but we do see a clear speedup
with the built in webserver.


Since my webserver is single threaded, I don't think this is
really fit for production use. If one user's request went too
long - which it shouldn't, but could - it'd slow everyone down.
It also can't serve up staticish files while waiting on something
else to process.

Another worry I have with a long lived process is what if it dies?
With CGI, you can trust Apache to start a fresh copy for you.

That's why I call the cgi one easier to use. You just drop it in
there, no need to worry about it dying, no need to start or restart
the process for updates, etc.


For fun, let's run it on some simpler pages too

Serving the auto-generated Javascript api file:

CGI:
Time per request:       13.678 [ms] (mean)

Built in:
Time per request:       1.070 [ms] (mean)


Big multiplier, but since it's a trivial program, we can see it's
about the same difference as we saw above - a constant ~10-13 ms
savings.

I also serve the stylesheet through it, because then it's
deployed all as one file. This is done before the database and
reflection objects are created anyway, so that optimization
shouldn't have an effect here.

CGI:
Time per request:       10.493 [ms] (mean)

Built in:
Time per request:       0.774 [ms] (mean)

And indeed, it didn't - the 10 ms constant cost is all we see.

I wonder if I can do anything to improve the I/O
performance.... nope, changing to the syscall from stdio made
no significant difference.

Doing
 time PATH_INFO=/style.css ./cgi-app > /dev/null

averages to ~10 ms too, so I can't blame Apache either.


So startup + object construction + I/O overhead for a real
world page gives CGI about a 10ms penalty. This ranges from a
small to a very big multiplier of total time, depending on
what the program is.

Next to real world network conditions, this is consistently
a small number in the big picture - the end user is unlikely to
notice it.
May 22 2011
parent Trass3r <un known.com> writes:
Very interesting benchmarks!
May 22 2011
prev sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 22 May 2011 20:10:07 +0300, Adam D. Ruppe  
<destructionator gmail.com> wrote:


 Vladimir Panteleev wrote:

 Regular CGI still has the overhead of starting a new process for
 every request.


 That's a very small cost - between 1 and 5 milliseconds, depending
 on your hardware.

 Next to the network latency involved in making a request, that's
 literally nothing - it's less than random network variation, so it
 has no impact on the end user experience.


Yes, of course - the difference is negligible if the server load is low.  
But if your website is getting enough hits to generate more requests than  
the server can process, technology choice matters a lot.


 It also disappears quickly if your program does any processing,
 since it's like a line: y = mx + b. The startup cost is b, and
 the speed of the language is m.


Yes, I should have added that I meant for very simple requests. The speed  
advantage of a compiled language will always overtake most initialization  
overheads for non-trivial requests.


 The end result depends on your app. Only actually benchmarking it
 can tell for sure what matters. My cgi.d provides the same interface
 regardless of how it is called - switching to built in httpd is
 easy if needed.


Is there any reason you didn't go for FastCGI or SCGI? The main advantage  
I can see is that CGI is much simpler for edit-compile-test iterations.


 I've had great experience with using an HTML/jQuery frontend with
 a D backend


 From what I've seen in my benchmarks, you would probably get
 a bigger usability boost by ditching jQuery than by ditching CGI.
 Even when cached, jQuery takes a while for the browser to parse -
 comparable to starting up a native process on the server.


The difference here is that HTML, CSS and jQuery will only be loaded once.  
I was talking about AJAX web-apps.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 22 2011
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Vladimir Panteleev wrote:

 But if your website is getting enough hits to generate more
 requests than the server can process, technology choice matters a
 lot.


Yeah. I've never had that happen, so I don't really know. If it
happens, it's easy enough to change later. (it was a two line change
just now to switch it to built-in webserver - none of the actual
app code needs to change at all)


 Is there any reason you didn't go for FastCGI or SCGI?


The biggest reason is they are harder to implement, and it's
premature optimization. I didn't want to spend a lot of extra
time writing code that would never be needed!


Secondary benefits are simplicity and reliability. A CGI process
can segfault without affecting anyone else. It can go completely
wild on memory, and it doesn't matter because it's short-lived
anyway. It can deployed to any server with ease - just copy
the binary in there. Worst case, you add a couple lines to
.htaccess.

Apache (or IIS) also handles logging and other details for a CGI
program.


It's just simpler in a lot of ways.
May 22 2011
parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 22/05/2011 22:40, Adam D. Ruppe wrote:

 Vladimir Panteleev wrote:

 But if your website is getting enough hits to generate more
 requests than the server can process, technology choice matters a
 lot.


 Yeah. I've never had that happen, so I don't really know. If it
 happens, it's easy enough to change later. (it was a two line change
 just now to switch it to built-in webserver - none of the actual
 app code needs to change at all)


 Is there any reason you didn't go for FastCGI or SCGI?


 The biggest reason is they are harder to implement, and it's
 premature optimization. I didn't want to spend a lot of extra
 time writing code that would never be needed!


FastCGI has an interface available that emulates CGI - that's not 
exactly harder to implement! ;)


 Secondary benefits are simplicity and reliability. A CGI process
 can segfault without affecting anyone else. It can go completely
 wild on memory, and it doesn't matter because it's short-lived
 anyway. It can deployed to any server with ease - just copy
 the binary in there. Worst case, you add a couple lines to
 .htaccess.


Same for FCGI, for the most part, except it's a long running process. 
You can set it up to periodically re-spawn if you like, or it will 
automatically re-spawn when you exit (error or otherwise).


 Apache (or IIS) also handles logging and other details for a CGI
 program.


Same for FCGI.


 It's just simpler in a lot of ways.


That's debatable! :D

-- 
Robert
http://octarineparrot.com/
May 22 2011
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Robert Clipsham wrote:

 FastCGI has an interface available that emulates CGI - that's not
 exactly harder to implement


What scared me was that the data comes in on a socket... so wouldn't
that come with the same threading problems my simple http server
has? (Where one slow request means everyone else has to wait)

I guess I'll have to try it!

[a little later]

So I changed cgi.d to include support for fast cgi by linking in
one of the C libraries. That wasn't hard.

It was kinda painful to get the C library and apache module working
though...


Anyway, I recompiled my same in-development work application to
use fcgi, so let's run the benchmarks on it and see what we got.


Same page as I used in the last post.

7 ms per request

under regular cgi it ran in about 20 ms. My embedded httpd served
it up in 5 ms per request.

Turning up the concurrent requests to 50 (I'm using the program ab
for these benchmarks btw) shows me that Apache spawned more copies
of the process automatically. Excellent.

Got 150 ms with fcgi. 35 ms with embedded. 370 ms with plain cgi.
(I wonder why the plain cgi shows weirdness here)


With 100, my embedded one still wins by a lot in 90% of requests,
but in the other ones, it gets brutally slow - 600 ms for 5% and
9 full seconds for 1% of them. Ouch! fcgi came in at 250 for most
requests, with the longest one being a little over a second.


Getting the generated Javascript file:

-c 50, CGI: 300 ms
-c 50, FCGI: 50 ms (nice!)
-c 50, embedded: 6 ms for 95%, but one took 3 seconds.


Getting the static stylesheet, still -c 50:

CGI: 150 ms
FCGI: 50 ms
embedded: 4ms

-c 1

CGI: 12 ms
FCGI: 1 ms
embedded: 1 ms


Keep in mind, this program was *already* beating comparable mod_php
applications, even in plain CGI. Now, it's just a curbstomp.


Overall, that's pretty ok, there is a consistent boost over plain
CGI. My simple embedded server seems to do consistently better
though, until it hits the concurrency wall. (I'd be willing to bet
a big part of the reason is that the embedded one uses slices for a
lot of stuff. The other ones copy data at least once.)


Not bad at all, but I'm not sure it's good enough for me to
switch to it in general. Regardless, this proves something I've
been saying though: my cgi.d adapts with ease to different
environments.

I'm running the same program in these benchmarks. Only difference
is compiling with -version=fastcgi or -version=embedded_httpd. The
app code is completely unchanged.
May 22 2011
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Mon, 23 May 2011 03:49:49 +0300, Adam D. Ruppe  
<destructionator gmail.com> wrote:


 What scared me was that the data comes in on a socket... so wouldn't
 that come with the same threading problems my simple http server
 has? (Where one slow request means everyone else has to wait)


Asynchronous networking is all the rage now. lighttpd is completely  
asynchronous and VERY fast.

Have you heard about NodeJS? It uses asynchronous IO as the primary  
(only?) way to do IO. As closures are first-class JavaScript language  
features, async IO works pretty well there. I dream about creating  
something like this in D as well one day.

Anyway, the idea is that nothing should block your event loop for more  
than a millisecond or so. If something takes longer, then either you need  
to look into "asynchronizing" it (file access?), or putting it in a  
separate, isolated thread.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 22 2011
parent Adam D. Ruppe <destructionator gmail.com> writes:
Vladimir Panteleev wrote:

 Have you heard about NodeJS?


Yeah. It's actually not too much different than the way my network
manager code works in D (which the embedded http server is made on)
although their implementation is much better than mine - I just use
select() in the event loop.

The key to using it is to always return from your function as soon
as possible. It will then add to your buffer and call the function
again as new stuff comes in.

It actually works quite well for it's simplicity. The problem with
using it for the cgi apps is they aren't really written to work
in that style.
May 22 2011
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Vladimir Panteleev wrote:

 PHP without CGI is likely to be faster than D with CGI.


Nope.

In my last post, I did cgi/D vs builtin/D and we saw a 10 ms
difference (running on my home computer. I did the same thing on
the live company server just now and saw only a 2 ms difference -
proper configuration and hardware helps cut it down too.)

I mentioned a 40% speedup on comparable products D vs PHP. Let's
list some more.


One of the company websites is run by Wordpress. 246 ms per request.

Another uses phpBB3. 73 ms per request.

One of them uses a PHP graph library to make a quick line graph.
136 ms per request.


Let's contrast to an assortment of D sites I've written for them,
all on that same server.

Dynamic homepage: 11 ms per request

Scheduling and conferencing app: 14 ms per request

Gradient generator outputting small pngs: 4 ms per request


Most the D apps would be too hard to rewrite in PHP to do a direct
comparison, so we'll have to settle for the 10x difference we
see between these and the popular PHP packages...


But the gradient generator is simple. I found a PHP program on
the web that does the same thing with basic options, utilizing
the GD library.

I downloaded and benchmarked it.... 19 ms per request, creating
identical images. (average of 2000 requests) Recall the D program,
running on CGI, did the very same thing in 4 ms per request, about
5x faster.


We can ignore the network, embrace CGI, and still *easily* crush
PHP performance with anything more complicated than hello world.
May 22 2011
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 22 May 2011 22:41:12 +0300, Adam D. Ruppe  
<destructionator gmail.com> wrote:


 We can ignore the network, embrace CGI, and still *easily* crush
 PHP performance with anything more complicated than hello world.


Interesting. Thanks.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
May 22 2011
prev sibling next sibling parent "Nick Sabalausky" <a a.a> writes:
"joe" <joe studiofortress.com> wrote in message 
news:ir8frr$ait$1 digitalmars.com...

 Second, from your own experience using D do think it would make a good or 
 bad
 choice for a web development language?


I'd really say that a good choice for a web development language is whatever 
you find to be a good choice for a language. Personally, I love D. It's my 
favorite language, and out of all the languages I've used (many over the 
years) it's really the only one I'm overall happy with (but then, I'm 
notoriously picky ;) ). So I find D to be an excellent choice for a web 
development language since I personally find D to be an excellect language. 
OTOH, if someone happened to absolutely hate D, then D probably wouldn't be 
a very good choice for them for a web development language.

There is one other factor to consider, though: You need to think about what 
server or servers it will need to run on:

If, for example, you need it to be able to run on pretty much any shared web 
host out there, then you're pretty much stuck with either PHP or something 
like Haxe that compiles down to PHP. And that defintiely won't help you with 
performance, and it'll only help alleviate *some* of PHP's probelms, not all 
of them. That's the unfortunate state of shared web hosting :/

If, OTOH, you only need it to run on one specific server and you don't have 
complete control over it (for example, if it's a shared web host, or if it's 
run by some separate IT department thet you're not part of), then naturally 
you'll need to see what the server supports and make your choice from those. 
But keep in mind one thing I've learned the hard way: Just because they say 
they support CGI does *not* necessarily mean they support CGI from a 
natively compiled langauge (which D is). So make sure to get clarification 
on that.

And, of course, if you have total control over the server, then naturally 
you can choose whatever you want.

Oh, but unless you do have direct access to the server and total control 
over it, I would recommend against ASP.NET. Not because I really have 
anything against ASP.NET, Windows or MS, per se. It's just that ASP.NET 
typically requires a Windows server, and shared Windows hosts usually don't 
provide any way to set up things like file permissions and internal URL 
rewriting. And not being able to do those things can often be a problem.
May 21 2011
prev sibling parent reply Lutger Blijdestijn <lutger.blijdestijn gmail.com> writes:
joe wrote:


 I currently do most of my web development in PHP, with some work in Ruby
 with RoR. Right now I'm starting to think about building my own stack for
 web dev (I already use my own MVC framework and libs in PHP), but I'd
 really like to move to something faster and more powerful. Java or ASP.NET
 are two obvious choices, but I'm currently researching other alternatives
 (like D).
 
 So first I was wondering if there has been any web development using D,
 and if there are many (or any) libraries/frameworks/other resources
 already available for this?


As mentioned Adam D. Ruppe has nice libs for web development, he managed to 
make a living out of it too so he must be doing something right :)

There is one other web framework for D2 called serenity, I'm not sure how 
far it is but you can check it out here:

https://github.com/mrmonday/serenity

Perhaps this is unnecessary to mention, but would you choose ASP.NET, I'd 
recommend against webforms and look into ASP.NET mvc. If haven't used the 
latter myself but I didn't like webforms too much and heard good things 
about the mvc framework. 

Good luck!
May 22 2011
parent Robert Clipsham <robert octarineparrot.com> writes:
On 22/05/2011 13:09, Lutger Blijdestijn wrote:

 joe wrote:


 I currently do most of my web development in PHP, with some work in Ruby
 with RoR. Right now I'm starting to think about building my own stack for
 web dev (I already use my own MVC framework and libs in PHP), but I'd
 really like to move to something faster and more powerful. Java or ASP.NET
 are two obvious choices, but I'm currently researching other alternatives
 (like D).

 So first I was wondering if there has been any web development using D,
 and if there are many (or any) libraries/frameworks/other resources
 already available for this?


 As mentioned Adam D. Ruppe has nice libs for web development, he managed to
 make a living out of it too so he must be doing something right :)

 There is one other web framework for D2 called serenity, I'm not sure how
 far it is but you can check it out here:

 https://github.com/mrmonday/serenity

 Perhaps this is unnecessary to mention, but would you choose ASP.NET, I'd
 recommend against webforms and look into ASP.NET mvc. If haven't used the
 latter myself but I didn't like webforms too much and heard good things
 about the mvc framework.

 Good luck!


Thanks for mentioning it! :D I held off mentioning it due to its 
immaturity in comparison to Adam's stuff, I'm gradually working on it. 
The latest thing I've added is an ORM to remove about 100 lines of code 
for simple stuff (still a work in progress). As a result I'm probably 
going to be reworking the overall design (It started off being MVC, it's 
not going to be now).

-- 
Robert
http://octarineparrot.com/
May 22 2011