www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Measuring the page generation of the forum

reply "SomeDude" <lovelydear mailmetrash.com> writes:
Hi,

I've noticed a couple of times (i.e very rarely, mind you), that 
the page generation of the forum had hiccups.
Have other people experienced such hiccups ?
If yes, I wonder if we saw the GC kicking in, or if it was 
SQLite's fault (or was it a network latency ?).

Anyhow, the forum seems to me a good opportunity to measure the D 
runtime in action on the server side, and in particular the GC.
A few questions :
Is the server regularly restarted or is it running smoothly 
without having to be restarted at all ?
Does it measure the time for the page generation ? Would it be 
interesting to see at the bottom of the page something like "Page 
generated in xxx ms", when I (or other people) come accross those 
hiccups, and to collect these data for statistics over a 
week/month ?
Apr 12 2012
next sibling parent reply "SomeDude" <lovelydear mailmetrash.com> writes:
On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:

 Hi,

 I've noticed a couple of times (i.e very rarely, mind you), 
 that the page generation of the forum had hiccups.
 Have other people experienced such hiccups ?
 If yes, I wonder if we saw the GC kicking in, or if it was 
 SQLite's fault (or was it a network latency ?).

 Anyhow, the forum seems to me a good opportunity to measure the 
 D runtime in action on the server side, and in particular the 
 GC.
 A few questions :
 Is the server regularly restarted or is it running smoothly 
 without having to be restarted at all ?
 Does it measure the time for the page generation ? Would it be 
 interesting to see at the bottom of the page something like 
 "Page generated in xxx ms", when I (or other people) come 
 accross those hiccups, and to collect these data for statistics 
 over a week/month ?


So noone thinks this could be a good idea ?
Apr 12 2012
next sibling parent reply "Nick Sabalausky" <SeeWebsiteToContactMe semitwist.com> writes:
"SomeDude" <lovelydear mailmetrash.com> wrote in message 
news:mjectyruiwxebfrkegvt forum.dlang.org...

 On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:

 Hi,

 I've noticed a couple of times (i.e very rarely, mind you), that the page 
 generation of the forum had hiccups.
 Have other people experienced such hiccups ?
 If yes, I wonder if we saw the GC kicking in, or if it was SQLite's fault 
 (or was it a network latency ?).

 Anyhow, the forum seems to me a good opportunity to measure the D runtime 
 in action on the server side, and in particular the GC.
 A few questions :
 Is the server regularly restarted or is it running smoothly without 
 having to be restarted at all ?
 Does it measure the time for the page generation ? Would it be 
 interesting to see at the bottom of the page something like "Page 
 generated in xxx ms", when I (or other people) come accross those 
 hiccups, and to collect these data for statistics over a week/month ?


 So noone thinks this could be a good idea ?


I like it. (And the StopWatch in std.datetime would make it super easy. 
StopWatch is freaking awesome.) But newgroup/forum posts don't always get 
responses after only a few hours. And even a total lack of replies shouldn't 
be mistaken for a "no" vote. So I wouldn't worry too much about that.

Could you be more specific what you mean by "hiccup"? I assume you mean that 
once in a while a page load will be slower than usual. If so, about how 
long?

My wild random guesses, aside from your ideas, are that maybe your request 
just happens to occur at the right time for the forum to decide to refresh 
its cache (if it even works that way). Or maybe it could be related to the 
newsgroup server (which the web forum is just a frontend for) being 
temporarily down due to high load (which unfortunately has been happening a 
lot lately). Vladimir would probably have a better idea what might be going 
on than I would, though.
Apr 12 2012
parent Somedude <lovelydear mailmetrash.com> writes:
Le 13/04/2012 01:55, Nick Sabalausky a écrit :

 "SomeDude" <lovelydear mailmetrash.com> wrote in message 
 news:mjectyruiwxebfrkegvt forum.dlang.org...

 On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:
 So noone thinks this could be a good idea ?


 
 I like it. (And the StopWatch in std.datetime would make it super easy. 
 StopWatch is freaking awesome.) But newgroup/forum posts don't always get 
 responses after only a few hours. And even a total lack of replies shouldn't 
 be mistaken for a "no" vote. So I wouldn't worry too much about that.
 


OK


 Could you be more specific what you mean by "hiccup"? I assume you mean that 
 once in a while a page load will be slower than usual. If so, about how 
 long?
 


Yeah, once I was surprised, because it took several seconds I believe.


 My wild random guesses, aside from your ideas, are that maybe your request 
 just happens to occur at the right time for the forum to decide to refresh 
 its cache (if it even works that way). Or maybe it could be related to the 
 newsgroup server (which the web forum is just a frontend for) being 
 temporarily down due to high load (which unfortunately has been happening a 
 lot lately). Vladimir would probably have a better idea what might be going 
 on than I would, though.
 


Ah, I forgot about the newsgroup server.
Apr 12 2012
prev sibling parent Kevin Cox <kevincox.ca gmail.com> writes:
On Apr 12, 2012 4:29 PM, "SomeDude" <lovelydear mailmetrash.com> wrote

 So noone thinks this could be a good idea ?


I like it.  I usually put a comment in the bottom of my pages but since the
form is implemented in D it would nice to actually display it somewhere.
Apr 12 2012
prev sibling next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:

 Hi,

 I've noticed a couple of times (i.e very rarely, mind you), 
 that the page generation of the forum had hiccups.
 Have other people experienced such hiccups ?
 If yes, I wonder if we saw the GC kicking in, or if it was 
 SQLite's fault (or was it a network latency ?).


Long page loads are caused by a combination of high server load + 
RAM cache misses + SQLite queries that require a lot of seeks. I 
still haven't found the time for the migration to a more powerful 
server.


 Is the server regularly restarted or is it running smoothly 
 without having to be restarted at all ?


It seems to run fine for weeks on end.


 Does it measure the time for the page generation ? Would it be 
 interesting to see at the bottom of the page something like 
 "Page generated in xxx ms", when I (or other people) come 
 accross those hiccups, and to collect these data for statistics 
 over a week/month ?


There are some numbers in the reddit discussion:

http://www.reddit.com/r/programming/comments/ppre5/the_new_d_online_forum_software_written_in_d/c3rfwte
Apr 12 2012
next sibling parent reply Somedude <lovelydear mailmetrash.com> writes:
Le 13/04/2012 03:40, Vladimir Panteleev a écrit :

 On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:

 Hi,

 I've noticed a couple of times (i.e very rarely, mind you), that the
 page generation of the forum had hiccups.
 Have other people experienced such hiccups ?
 If yes, I wonder if we saw the GC kicking in, or if it was SQLite's
 fault (or was it a network latency ?).


 
 Long page loads are caused by a combination of high server load + RAM
 cache misses + SQLite queries that require a lot of seeks. I still
 haven't found the time for the migration to a more powerful server.
 


Well, it's not really urgent. It's fast enough.
Still, collecting statistics over time would have been nice, so that one
could analyse the general behaviour of the GC, for instance, by getting
an idea of the standard deviation and maximum response time over weeks
of use. I guess since it can handle much larger loads, the forum server
is not stressed enough to give very meaningful insight under heavy load.
But I have experienced a response of several seconds once, I don't know
where this comes from (although it's most likely one of your reasons).
So collecting real time data would have to be done for each component of
the chain (SQLite, cache, D generated page). Such data would be useful
for any web server anyway.

BTW, could you repost the source code address ? Thx.


 Is the server regularly restarted or is it running smoothly without
 having to be restarted at all ?


 
 It seems to run fine for weeks on end.


Nice.


 

 Does it measure the time for the page generation ? Would it be
 interesting to see at the bottom of the page something like "Page
 generated in xxx ms", when I (or other people) come accross those
 hiccups, and to collect these data for statistics over a week/month ?


 
 There are some numbers in the reddit discussion:
 
 http://www.reddit.com/r/programming/comments/ppre5/the_new_d_online_forum_software_written_in_d/c3rfwte
 


Nice, now I remember having read this post.
Apr 12 2012
next sibling parent reply James Miller <james aatch.net> writes:
* Somedude <lovelydear mailmetrash.com> [2012-04-13 04:25:19 +0200]:

 Well, it's not really urgent. It's fast enough.
 Still, collecting statistics over time would have been nice, so that one
 could analyse the general behaviour of the GC, for instance, by getting
 an idea of the standard deviation and maximum response time over weeks
 of use. I guess since it can handle much larger loads, the forum server
 is not stressed enough to give very meaningful insight under heavy load.
 But I have experienced a response of several seconds once, I don't know
 where this comes from (although it's most likely one of your reasons).
 So collecting real time data would have to be done for each component of
 the chain (SQLite, cache, D generated page). Such data would be useful
 for any web server anyway.


I doubt that the GC is the source of any real problems. As a web
developer (in PHP no less :S) I can say with authority that most slow
downs are due to waiting on external services and data processing. Some
of the slowest parts of our site are due to heavy database usage where
we have to perform over 100 SQL queries to generate a page. Other
slowdowns are caused by the need to process the data.

I understand that you want to see how the GC affects the server
performance, but it will be so small that it will be lost in the noise,
a blip in traffic because it's lunchtime and people are watching youtube
in your area will cause a bigger delay than the GC kicking in.

--
James Miller
Apr 12 2012
parent Somedude <lovelydear mailmetrash.com> writes:
Le 13/04/2012 04:43, James Miller a écrit :

 I doubt that the GC is the source of any real problems. As a web
 developer (in PHP no less :S) I can say with authority that most slow
 downs are due to waiting on external services and data processing. Some
 of the slowest parts of our site are due to heavy database usage where
 we have to perform over 100 SQL queries to generate a page. Other
 slowdowns are caused by the need to process the data.
 
 I understand that you want to see how the GC affects the server
 performance, but it will be so small that it will be lost in the noise,
 a blip in traffic because it's lunchtime and people are watching youtube
 in your area will cause a bigger delay than the GC kicking in.
 
 --
 James Miller


I agree, you're probably right.
Apr 12 2012
prev sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 13 April 2012 at 02:25:19 UTC, Somedude wrote:

 Well, it's not really urgent. It's fast enough.
 Still, collecting statistics over time would have been nice, so 
 that one could analyse the general behaviour of the GC, for 
 instance, by getting an idea of the standard deviation and 
 maximum response time over weeks of use.


Such statistics wouldn't be very meaningful due to the highly 
varying server load, caused by a large number of other websites 
and services running on the same machine.


 But I have experienced a response of several seconds once, I 
 don't know where this comes from (although it's most likely one 
 of your reasons).


Long page loads are caused by a combination of high server load +
RAM cache misses + SQLite queries that require a lot of seeks. 
I'm quite sure this is the case.


 BTW, could you repost the source code address ? Thx.


There is a link to the source code on the Help page.


 Oh, I just saw you wrote the max page generation was around 50 
 ms.


When there are no cache misses, yes.
Apr 12 2012
prev sibling parent Somedude <lovelydear mailmetrash.com> writes:
Le 13/04/2012 03:40, Vladimir Panteleev a écrit :

 On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:
 There are some numbers in the reddit discussion:
 
 http://www.reddit.com/r/programming/comments/ppre5/the_new_d_online_forum_software_written_in_d/c3rfwte
 


Oh, I just saw you wrote the max page generation was around 50 ms.
Apr 12 2012
prev sibling parent reply "Jesse Phillips" <jessekphillips+D gmail.com> writes:
On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:

 Hi,

 I've noticed a couple of times (i.e very rarely, mind you), 
 that the page generation of the forum had hiccups.
 Have other people experienced such hiccups ?
 If yes, I wonder if we saw the GC kicking in, or if it was 
 SQLite's fault (or was it a network latency ?).

 Anyhow, the forum seems to me a good opportunity to measure the 
 D runtime in action on the server side, and in particular the 
 GC.
 A few questions :
 Is the server regularly restarted or is it running smoothly 
 without having to be restarted at all ?
 Does it measure the time for the page generation ? Would it be 
 interesting to see at the bottom of the page something like 
 "Page generated in xxx ms", when I (or other people) come 
 accross those hiccups, and to collect these data for statistics 
 over a week/month ?


I'm pretty sure it is the NG itself. Every news reader I've used 
has shown some form of "hiccup" If you try making a post under a 
heavy load you'll get a nice page telling you the load and what 
it needed to be.
Apr 12 2012
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 13 April 2012 at 03:24:19 UTC, Jesse Phillips wrote:

 I'm pretty sure it is the NG itself. Every news reader I've 
 used has shown some form of "hiccup" If you try making a post 
 under a heavy load you'll get a nice page telling you the load 
 and what it needed to be.


Long page loads are caused by a combination of high server load +
RAM cache misses + SQLite queries that require a lot of seeks.
Network operations, including fetching new posts from the NG and
posting to the NG, are asynchronous, and never block web requests.
Apr 13 2012
parent reply "Jesse Phillips" <jessekphillips+D gmail.com> writes:
On Friday, 13 April 2012 at 08:09:09 UTC, Vladimir Panteleev 
wrote:

 On Friday, 13 April 2012 at 03:24:19 UTC, Jesse Phillips wrote:

 I'm pretty sure it is the NG itself. Every news reader I've 
 used has shown some form of "hiccup" If you try making a post 
 under a heavy load you'll get a nice page telling you the load 
 and what it needed to be.


 Long page loads are caused by a combination of high server load 
 +
 RAM cache misses + SQLite queries that require a lot of seeks.
 Network operations, including fetching new posts from the NG and
 posting to the NG, are asynchronous, and never block web 
 requests.


I use the split-thread view, I haven't had long page loads so 
maybe I've miss understood what he means by a hiccup. But I have 
seen it take time to retrieve a message and display it to me. I 
would expect a page could be considered "loading" if it is still 
waiting on fetching the post from the NG.
Apr 13 2012
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Saturday, 14 April 2012 at 03:31:29 UTC, Jesse Phillips wrote:

 I use the split-thread view, I haven't had long page loads so 
 maybe I've miss understood what he means by a hiccup. But I 
 have seen it take time to retrieve a message and display it to 
 me. I would expect a page could be considered "loading" if it 
 is still waiting on fetching the post from the NG.


All posts are stored in a local SQLite database. Posts are 
retrieved from the NG when the program starts, and as they are 
posted (a listening connection is kept open). If a message is 
listed in the forum, then it is already in the local database.
Apr 13 2012