www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - First big PITA in reallife D project

reply Stephan <spam extrawurst.org> writes:
Hey guys I wanted to discuss my first major depressing experience with D 
in a reallife project.

First off here is the according bug report : 
http://d.puremagic.com/issues/show_bug.cgi?id=4951

I have to mention that it was by far not the first bug i encountered in 
D but it has never been such a PITA. I spent hours and hours of 
searching, rewriting and crawling my code for the cause of this issue. 
Just to finally find out that it was not my code but some bug in druntime.

What i do:
I spawn multiple threads all periodically connecting to some webservers 
waiting for the request to finish. Every request is totally independent 
of the others. There is no data sharing in my code. Now i have a 
protocol that makes my process send one last important request when 
shutting down. For a clean shut down i wanted to make sure that all the 
other threads are shut down first. That way the final request of the 
process is the very last. So i wait for every thread to finish their 
request and stop them.
BUT it wont work like that. AFTER i stopped all other threads some 
strange behavior of druntime makes every creation of an InternetAddress 
instance (internally trying to resolve host) impossible (throws an 
exception). Additionally even if the Internet Addresses were created 
upfront the TcpSockets wont connect but throw.
I have no idea what the heck is going on inside of the druntime that 
could cause this but in my opinion it makes designing a big server 
environment pretty much impossible because it would mean that i cannot 
let any of my user generated threads exit unless i want to shutdown 
completely anyway.

Perhaps one side note: I am testing and having these issues under Windows.


I have never been so tempted to use another language while coding in D 
as i was this time and that makes me sad.


- Stephan


(Sorry for reposting but it belongs to this NG and not the Announcement NG)
Oct 19 2010
next sibling parent "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Tue, 19 Oct 2010 20:33:29 +0900, Stephan <spam extrawurst.org> wrote:


 Hey guys I wanted to discuss my first major depressing experience with D  
 in a reallife project.

 First off here is the according bug report :  
 http://d.puremagic.com/issues/show_bug.cgi?id=4951


I think this bug is related to  
http://d.puremagic.com/issues/show_bug.cgi?id=4344
I replied similar post in NG.
http://lists.puremagic.com/pipermail/digitalmars-d/2010-September/082814.html

I want to know the best solution.
Oct 19 2010
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 19 Oct 2010 07:33:29 -0400, Stephan <spam extrawurst.org> wrote:


 Hey guys I wanted to discuss my first major depressing experience with D  
 in a reallife project.

 First off here is the according bug report :  
 http://d.puremagic.com/issues/show_bug.cgi?id=4951

 I have to mention that it was by far not the first bug i encountered in  
 D but it has never been such a PITA. I spent hours and hours of  
 searching, rewriting and crawling my code for the cause of this issue.  
 Just to finally find out that it was not my code but some bug in  
 druntime.

 What i do:
 I spawn multiple threads all periodically connecting to some webservers  
 waiting for the request to finish. Every request is totally independent  
 of the others. There is no data sharing in my code. Now i have a  
 protocol that makes my process send one last important request when  
 shutting down. For a clean shut down i wanted to make sure that all the  
 other threads are shut down first. That way the final request of the  
 process is the very last. So i wait for every thread to finish their  
 request and stop them.
 BUT it wont work like that. AFTER i stopped all other threads some  
 strange behavior of druntime makes every creation of an InternetAddress  
 instance (internally trying to resolve host) impossible (throws an  
 exception). Additionally even if the Internet Addresses were created  
 upfront the TcpSockets wont connect but throw.
 I have no idea what the heck is going on inside of the druntime that  
 could cause this but in my opinion it makes designing a big server  
 environment pretty much impossible because it would mean that i cannot  
 let any of my user generated threads exit unless i want to shutdown  
 completely anyway.

 Perhaps one side note: I am testing and having these issues under  
 Windows.


 I have never been so tempted to use another language while coding in D  
 as i was this time and that makes me sad.


That is unfortunate.  These kinds of bugs need to be fixed ASAP, and they  
should be much easier than compiler bugs.

The issue at the moment is that nobody owns or wants to own the network  
API portion of phobos (it's not druntime btw).  So your pleas are falling  
on sympathetic ears, but not ones that will make changes.  These growing  
pains will pass, I think some have mentioned wanting to rewrite the  
network API, but streams in general are up in the air at the moment...

I'll try and spend a few minutes to see if I can spot the problem.  Having  
a minimal test case as you have posted helps.

-Steve
Oct 19 2010
parent reply Stephan <spam extrawurst.org> writes:
On 19.10.2010 14:22, Steven Schveighoffer wrote:

 On Tue, 19 Oct 2010 07:33:29 -0400, Stephan <spam extrawurst.org> wrote:


 Hey guys I wanted to discuss my first major depressing experience with
 D in a reallife project.

 First off here is the according bug report :
 http://d.puremagic.com/issues/show_bug.cgi?id=4951

 I have to mention that it was by far not the first bug i encountered
 in D but it has never been such a PITA. I spent hours and hours of
 searching, rewriting and crawling my code for the cause of this issue.
 Just to finally find out that it was not my code but some bug in
 druntime.

 What i do:
 I spawn multiple threads all periodically connecting to some
 webservers waiting for the request to finish. Every request is totally
 independent of the others. There is no data sharing in my code. Now i
 have a protocol that makes my process send one last important request
 when shutting down. For a clean shut down i wanted to make sure that
 all the other threads are shut down first. That way the final request
 of the process is the very last. So i wait for every thread to finish
 their request and stop them.
 BUT it wont work like that. AFTER i stopped all other threads some
 strange behavior of druntime makes every creation of an
 InternetAddress instance (internally trying to resolve host)
 impossible (throws an exception). Additionally even if the Internet
 Addresses were created upfront the TcpSockets wont connect but throw.
 I have no idea what the heck is going on inside of the druntime that
 could cause this but in my opinion it makes designing a big server
 environment pretty much impossible because it would mean that i cannot
 let any of my user generated threads exit unless i want to shutdown
 completely anyway.

 Perhaps one side note: I am testing and having these issues under
 Windows.


 I have never been so tempted to use another language while coding in D
 as i was this time and that makes me sad.


 That is unfortunate. These kinds of bugs need to be fixed ASAP, and they
 should be much easier than compiler bugs.

 The issue at the moment is that nobody owns or wants to own the network
 API portion of phobos (it's not druntime btw). So your pleas are falling
 on sympathetic ears, but not ones that will make changes. These growing
 pains will pass, I think some have mentioned wanting to rewrite the
 network API, but streams in general are up in the air at the moment...

 I'll try and spend a few minutes to see if I can spot the problem.
 Having a minimal test case as you have posted helps.

 -Steve


Thanks for the reply and that you gonna take a look. I added the 
relation of bug http://d.puremagic.com/issues/show_bug.cgi?id=4344 to 
the bug aswell because it reallys seems to have the same underlying root.

- Stephan
Oct 19 2010
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 19 Oct 2010 08:35:09 -0400, Stephan <spam extrawurst.org> wrote:

 Thanks for the reply and that you gonna take a look. I added the  
 relation of bug http://d.puremagic.com/issues/show_bug.cgi?id=4344 to  
 the bug aswell because it reallys seems to have the same underlying root.


I fixed both with the suggested update in 4344.  I also found a race  
condition on Linux that as far as I know was never reported.

See changeset http://www.dsource.org/projects/phobos/changeset/2107

That was an easy one :)

-Steve
Oct 19 2010
next sibling parent reply Stephan <spam extrawurst.org> writes:
On 19.10.2010 15:27, Steven Schveighoffer wrote:

 On Tue, 19 Oct 2010 08:35:09 -0400, Stephan <spam extrawurst.org> wrote:

 Thanks for the reply and that you gonna take a look. I added the
 relation of bug http://d.puremagic.com/issues/show_bug.cgi?id=4344 to
 the bug aswell because it reallys seems to have the same underlying root.


 I fixed both with the suggested update in 4344. I also found a race
 condition on Linux that as far as I know was never reported.

 See changeset http://www.dsource.org/projects/phobos/changeset/2107

 That was an easy one :)

 -Steve


Awesome, thank you.
Is there a new release planned anytime soon that includes the patch ?

- Stephan
Oct 19 2010
next sibling parent "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Tue, 19 Oct 2010 23:02:31 +0900, Stephan <spam extrawurst.org> wrote:


 On 19.10.2010 15:27, Steven Schveighoffer wrote:

 On Tue, 19 Oct 2010 08:35:09 -0400, Stephan <spam extrawurst.org> wrote:

 Thanks for the reply and that you gonna take a look. I added the
 relation of bug http://d.puremagic.com/issues/show_bug.cgi?id=4344 to
 the bug aswell because it reallys seems to have the same underlying  
 root.


 I fixed both with the suggested update in 4344. I also found a race
 condition on Linux that as far as I know was never reported.

 See changeset http://www.dsource.org/projects/phobos/changeset/2107

 That was an easy one :)

 -Steve


 Awesome, thank you.
 Is there a new release planned anytime soon that includes the patch ?

 - Stephan


Discussion is here.
http://lists.puremagic.com/pipermail/phobos/2010-October/002957.html


Masahiro
Oct 19 2010
prev sibling parent Mike Chaten <mchaten gmail.com> writes:
I doubt a new version of dmd will be out until November at the earliest, as
there is no dmd beta out at the moment.
In the mean time, you should be able to download the latest source code of
phobos at
http://www.dsource.org/projects/phobos/changeset/head/trunk?old_path=%2F&format=zip
and build the code.

I'm not familiar with dmd under windows, but I believe you can just replace
the phobos .lib/.dll/.whateverextension file with the one you built.

On Tue, Oct 19, 2010 at 10:02 AM, Stephan <spam extrawurst.org> wrote:


 On 19.10.2010 15:27, Steven Schveighoffer wrote:


 On Tue, 19 Oct 2010 08:35:09 -0400, Stephan <spam extrawurst.org> wrote:


 Thanks for the reply and that you gonna take a look. I added the
 relation of bug http://d.puremagic.com/issues/show_bug.cgi?id=4344 to
 the bug aswell because it reallys seems to have the same underlying root.


 I fixed both with the suggested update in 4344. I also found a race
 condition on Linux that as far as I know was never reported.

 See changeset http://www.dsource.org/projects/phobos/changeset/2107

 That was an easy one :)

 -Steve


 Awesome, thank you.
 Is there a new release planned anytime soon that includes the patch ?

 - Stephan
Oct 19 2010
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Steven Schveighoffer wrote:

 On Tue, 19 Oct 2010 08:35:09 -0400, Stephan <spam extrawurst.org> wrote:

 Thanks for the reply and that you gonna take a look. I added the 
 relation of bug http://d.puremagic.com/issues/show_bug.cgi?id=4344 to 
 the bug aswell because it reallys seems to have the same underlying root.


 
 I fixed both with the suggested update in 4344.  I also found a race 
 condition on Linux that as far as I know was never reported.
 
 See changeset http://www.dsource.org/projects/phobos/changeset/2107
 
 That was an easy one :)


Awesome, thanks for taking care of this one.
Oct 19 2010
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
Stephan Wrote:


 Hey guys I wanted to discuss my first major depressing experience with D 
 in a reallife project.
 
 First off here is the according bug report : 
 http://d.puremagic.com/issues/show_bug.cgi?id=4951
 
 I have to mention that it was by far not the first bug i encountered in 
 D but it has never been such a PITA. I spent hours and hours of 
 searching, rewriting and crawling my code for the cause of this issue. 


Huh? I can understand when compiler really stays in you way. That's pig bita.
But... phobos? Yes, phobos doesn't work, so what? If something doesn't work,
you just roll your own code, right? And trust me, socket is not an AI, one can
easily boot it in a day or two of lazy coding.
I'm working with .net 3.5 and we *do* write our own utils because .net
functionality is not sufficient: type converters, home-grown inheritance,
dataset serializers. And, think, it's .net!
Oct 19 2010
parent dennis luehring <dl.soluz gmx.net> writes:
Am 19.10.2010 17:04, schrieb Kagamin:

 Stephan Wrote:


  Hey guys I wanted to discuss my first major depressing experience with D
  in a reallife project.

  First off here is the according bug report :
  http://d.puremagic.com/issues/show_bug.cgi?id=4951

  I have to mention that it was by far not the first bug i encountered in
  D but it has never been such a PITA. I spent hours and hours of
  searching, rewriting and crawling my code for the cause of this issue.


 Huh? I can understand when compiler really stays in you way. That's pig bita.
But... phobos? Yes, phobos doesn't work, so what? If something doesn't work,
you just roll your own code, right? And trust me, socket is not an AI, one can
easily boot it in a day or two of lazy coding.
 I'm working with .net 3.5 and we *do* write our own utils because .net
functionality is not sufficient: type converters, home-grown inheritance,
dataset serializers. And, think, it's .net!


maybe it was not clear in the first shot, if it is an phobos or dmd 
problem...combined with threading...
Oct 19 2010
prev sibling parent reply FeepingCreature <default_357-line yahoo.de> writes:
On 19.10.2010 13:33, Stephan wrote:

 BUT it wont work like that. AFTER i stopped all other threads some
 strange behavior of druntime makes every creation of an InternetAddress
 instance (internally trying to resolve host) impossible (throws an
 exception). Additionally even if the Internet Addresses were created
 upfront the TcpSockets wont connect but throw.


Terminating threads has always been iffy in D. As a simple workaround, use a
threadpool to keep a set of threads around blocking on a semaphore (and
consuming no resources), and rewrite your problem in terms of tasks (void
delegate()).

I'm sorry you had such a bad experience with this. There really should be a
list of common D pitfalls, and "Don't ever destroy threads" should be among the
top ten.
Oct 19 2010
next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 19 Oct 2010 12:07:04 -0400, FeepingCreature  
<default_357-line yahoo.de> wrote:


 On 19.10.2010 13:33, Stephan wrote:

 BUT it wont work like that. AFTER i stopped all other threads some
 strange behavior of druntime makes every creation of an InternetAddress
 instance (internally trying to resolve host) impossible (throws an
 exception). Additionally even if the Internet Addresses were created
 upfront the TcpSockets wont connect but throw.


 Terminating threads has always been iffy in D. As a simple workaround,  
 use a threadpool to keep a set of threads around blocking on a semaphore  
 (and consuming no resources), and rewrite your problem in terms of tasks  
 (void delegate()).

 I'm sorry you had such a bad experience with this. There really should  
 be a list of common D pitfalls, and "Don't ever destroy threads" should  
 be among the top ten.


This one was a simple bug.  He was not terminating threads, rather he was  
letting a thread exit normally.

-Steve
Oct 19 2010
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/19/10 11:07 CDT, FeepingCreature wrote:

 On 19.10.2010 13:33, Stephan wrote:

 BUT it wont work like that. AFTER i stopped all other threads some
 strange behavior of druntime makes every creation of an
 InternetAddress instance (internally trying to resolve host)
 impossible (throws an exception). Additionally even if the Internet
 Addresses were created upfront the TcpSockets wont connect but
 throw.


 Terminating threads has always been iffy in D.


Terminating threads has always been iffy in all languages that use 
native threads. I've discussed with Sean a couple of times about ways to 
define a little protocol and API for orderly thread termination. We need 
something like a callback mechanism - atShutdown().

Andrei
Oct 19 2010
next sibling parent reply Kagamin <spam here.lot> writes:
Andrei Alexandrescu Wrote:


 Terminating threads has always been iffy in all languages that use 
 native threads. I've discussed with Sean a couple of times about ways to 
 define a little protocol and API for orderly thread termination. We need 
 something like a callback mechanism - atShutdown().


Why? Just call delete on the thread object and in destructor do everything you
want to put in atShutdown.
Oct 19 2010
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/19/10 13:59 CDT, Kagamin wrote:

 Andrei Alexandrescu Wrote:


 Terminating threads has always been iffy in all languages that use
 native threads. I've discussed with Sean a couple of times about
 ways to define a little protocol and API for orderly thread
 termination. We need something like a callback mechanism -
 atShutdown().


 Why? Just call delete on the thread object and in destructor do
 everything you want to put in atShutdown.


The problem with that is that it scales bad when you have many threads - 
the last thread stopped would have to wait for all others to stop.

Andrei
Oct 19 2010
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
Kagamin Wrote:


 Andrei Alexandrescu Wrote:
 

 Terminating threads has always been iffy in all languages that use 
 native threads. I've discussed with Sean a couple of times about ways to 
 define a little protocol and API for orderly thread termination. We need 
 something like a callback mechanism - atShutdown().


 
 Why? Just call delete on the thread object and in destructor do everything you
want to put in atShutdown.


Thread objects are tracked by the runtime.  If you explicitly delete them,
expect a segfault.  I suppose I could use a pimpl approach to avoid this, but
it would be a tad tricky (Thread.getThis() for example).  More appropriate is
to use a non-shared static dtor to signal thread shutdown, which is basically
how OwnerTerminated works anyway.  It's just taken care of for you.
Oct 22 2010
prev sibling parent Fawzi Mohamed <fawzi gmx.ch> writes:
On 19-ott-10, at 19:17, Andrei Alexandrescu wrote:


 On 10/19/10 11:07 CDT, FeepingCreature wrote:

 On 19.10.2010 13:33, Stephan wrote:

 BUT it wont work like that. AFTER i stopped all other threads some
 strange behavior of druntime makes every creation of an
 InternetAddress instance (internally trying to resolve host)
 impossible (throws an exception). Additionally even if the Internet
 Addresses were created upfront the TcpSockets wont connect but
 throw.


 Terminating threads has always been iffy in D.


 Terminating threads has always been iffy in all languages that use  
 native threads. I've discussed with Sean a couple of times about  
 ways to define a little protocol and API for orderly thread  
 termination. We need something like a callback mechanism -  
 atShutdown().


As feep wrote using thread pools is often better.
With D1/tango one can use blip (whose release is hopefully just a few  
days away) it becomes very nice to do.
Unfortunately this is not for Stephan, as most likely it will not work  
on windows, because I used and developed it only on
mac/linux, and windows will require some porting.

Anyway checking several addresses in parallel becomes simply:

import blip.parallel.smp.PLoopHelpers;
import blip.io.Socket;
//...
foreach(addr;pLoopArray(adressesToCheck){
	auto s=BasicSocket(addr,"80");
	// do stuff
}

and this will use optimally all processors of your computer, and will  
use libev to never block waiting for input...
Also making a server is not so difficult as shown by
http://github.com/fawzi/blip/blob/master/tests/EchoServer.d

the checking operation can spawn new tasks without problems.

Fawzi
Oct 19 2010