www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - 10's of threads + network == segfault (sample code)

reply prefetch <prefetch_member pathlink.com> writes:
not sure if this is the right place to post this, let me know if i should be
posting this somewhere else.

please try running the code below and see if you get segfaults every 5th or 6th
try.  not exactly sure what is going on or what to blame.

i tried running in gdb, but i can't reproduce it there.

has anyone used attempted to build very large systems (hundreds of concurrent
threads) using D?

let me know if you are able to reproduce this segfault (it should only take
about 10 times or so with a few dozen threads as the arg.)  i'd really
appreciate it.

----8<-----

import std.string, std.conv, std.stream;
import std.thread, std.socket, std.socketstream;

int netthread(void*nothing) {
Socket sock = new TcpSocket(new InternetAddress(server, port));
Stream ss = new SocketStream(sock);
/* do something with sockets */
return 0;
}

uint port = 8080;
char[] server = "localhost";

/* run this with 10 or so threads, and it will seg fault after a few tries. */
/* Linux 2.6.8-2-686-smp i686 GNU/Linux */
/* dmd v0.129 */

int main(char[][] args)
{
if(args.length < 2)
{
printf("Usage:\n   netthreadcrash numthreads\n");
return 0;
}
uint numthreads = std.conv.toUint(args[1]);

int i = 0;
while (i++ < numthreads) {
Thread mythread = new Thread(&netthread, null);
mythread.start();
}
return 0;
}
Aug 17 2005
parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
"prefetch" <prefetch_member pathlink.com> wrote in message 
news:ddvq51$2q0$1 digitaldaemon.com...

 not sure if this is the right place to post this, let me know if i should 
 be
 posting this somewhere else.

 please try running the code below and see if you get segfaults every 5th 
 or 6th
 try.  not exactly sure what is going on or what to blame.

 i tried running in gdb, but i can't reproduce it there.

 has anyone used attempted to build very large systems (hundreds of 
 concurrent
 threads) using D?

 let me know if you are able to reproduce this segfault (it should only 
 take
 about 10 times or so with a few dozen threads as the arg.)  i'd really
 appreciate it.

 ----8<-----

 import std.string, std.conv, std.stream;
 import std.thread, std.socket, std.socketstream;

 int netthread(void*nothing) {
 Socket sock = new TcpSocket(new InternetAddress(server, port));
 Stream ss = new SocketStream(sock);
 /* do something with sockets */
 return 0;
 }

 uint port = 8080;
 char[] server = "localhost";

 /* run this with 10 or so threads, and it will seg fault after a few 
 tries. */
 /* Linux 2.6.8-2-686-smp i686 GNU/Linux */
 /* dmd v0.129 */

 int main(char[][] args)
 {
 if(args.length < 2)
 {
 printf("Usage:\n   netthreadcrash numthreads\n");
 return 0;
 }
 uint numthreads = std.conv.toUint(args[1]);

 int i = 0;
 while (i++ < numthreads) {
 Thread mythread = new Thread(&netthread, null);
 mythread.start();
 }
 return 0;
 }


D doesn't wait for child threads to finish before trying to do a gc and 
exit. This causes nasty seg-v's on exit. Get around the problem by making 
sure all child threads are done before returning from main().
See for example 
http://www.digitalmars.com/d/archives/digitalmars/D/13528.html and 
http://www.digitalmars.com/d/archives/digitalmars/D/6419.html and probably 
more.
Aug 17 2005
parent reply prefetch <prefetch_member pathlink.com> writes:
thanks ben.  sounds like D is broken in this regard.  exiting main w/o cleaning
up threads should not result in a segfault.  just to make supre duper sure, i
wrote a quick test in C using pthreads, and it has no problems.

so, is there a D bug tracker?
how do i submit a bug?

i know there is a D bug newsgroup, but i'm assuming that's a discussion forum
and not an actual bug tracking system (i hope..)

In article <ddvv95$77p$1 digitaldaemon.com>, Ben Hinkle says...


D doesn't wait for child threads to finish before trying to do a gc and 
exit. This causes nasty seg-v's on exit. Get around the problem by making 
sure all child threads are done before returning from main().
See for example 
http://www.digitalmars.com/d/archives/digitalmars/D/13528.html and 
http://www.digitalmars.com/d/archives/digitalmars/D/6419.html and probably 
more.
Aug 17 2005
next sibling parent reply Derek Parnell <derek psych.ward> writes:
On Wed, 17 Aug 2005 21:10:46 +0000 (UTC), prefetch wrote:



 so, is there a D bug tracker?
 how do i submit a bug?
 
 i know there is a D bug newsgroup, but i'm assuming that's a discussion forum
 and not an actual bug tracking system (i hope..)


Sorry to be the bearer of bad news, but digitalmars.D.bugs *is* the bug
tracking 'system'. Fortunately, Thomas Kühne is helping us with the
'tracking' part.

-- 
Derek Parnell
Melbourne, Australia
18/08/2005 7:16:40 AM
Aug 17 2005
parent reply prefetch <prefetch_member pathlink.com> writes:
In article <1bl4gtbuuy6ek.12qm3vcv0nnxa$.dlg 40tude.net>, Derek Parnell says...


Sorry to be the bearer of bad news, but digitalmars.D.bugs *is* the bug
tracking 'system'. Fortunately, Thomas Kühne is helping us with the
'tracking' part.


holy crap batman, who is driving this boat?? ;-)

how on earth can you expect to develop a language and runtime library without a
bug tracking system?  i mean, this is just plain stupid.  we all know, there are
a whole bunch of free, web based bug trackers.

don't get me wrong, i've really enjoyed learning about D, and i think it's a
stellar language, but COME ON PEOPLE - let's get it together here.  somewhere on
the list of "how to start a super successful grassroots technology" is 'setup a
web based bug tracking system so the community can help identify and fix bugs
for the primary developer(s).'

are my expectations just way to high or something?  shouldn't this project
reflect the quality of the language concept?  i mean, it's a great concept - but
how about we help it not fade into obscurity before it even gets going by
building tools (docs, bug tracker, etc.) that can allow the community to grow?
Aug 17 2005
next sibling parent reply Manfred Nowak <svv1999 hotmail.com> writes:
prefetch <prefetch_member pathlink.com> wrote:

[...]

 holy crap batman, who is driving this boat?? ;-)


This superfast offshore-racer is punched through the waves solely by 
Walter Bright. Because of the mere mass of tiny fish that is spread 
all over the water, wanting the boat to race even faster and Walters 
limited capabilities only fish that is able to mimic a big whale 
pointing to fast routes through dangerous cliffs and shallows will be 
recognized.

Furthermore everything ever written from that tiny fish becomes 
immediately a private property of Walters DigitalMars, except 
expressively stated otherwise.

Therefore you just presnted another famous creature to become a 
private property :-)

-manfred
Aug 17 2005
parent reply prefetch <prefetch_member pathlink.com> writes:
hm.  

so let me translate: 

1. "this is a one man show."
2. "if you want the one man to do something, you better figure out how to get
his attention."
3. "anything you do to help becomes private property of the one man show."

is this correct?

i snooped around and couldn't find much licensing info - except the compiler
comes with a license.txt file that implies that it is owned by Digital Mars and
Symantec.  i wonder what symantec has to do with anything...??

In article <de0j9o$ldt$1 digitaldaemon.com>, Manfred Nowak says...

prefetch <prefetch_member pathlink.com> wrote:

[...]

 holy crap batman, who is driving this boat?? ;-)


This superfast offshore-racer is punched through the waves solely by 
Walter Bright. Because of the mere mass of tiny fish that is spread 
all over the water, wanting the boat to race even faster and Walters 
limited capabilities only fish that is able to mimic a big whale 
pointing to fast routes through dangerous cliffs and shallows will be 
recognized.

Furthermore everything ever written from that tiny fish becomes 
immediately a private property of Walters DigitalMars, except 
expressively stated otherwise.

Therefore you just presnted another famous creature to become a 
private property :-)

-manfred
Aug 17 2005
next sibling parent Derek Parnell <derek psych.ward> writes:
On Thu, 18 Aug 2005 00:27:12 +0000 (UTC), prefetch wrote:


 hm.  
 
 so let me translate: 
 
 1. "this is a one man show."
 2. "if you want the one man to do something, you better figure out how to get
 his attention."
 3. "anything you do to help becomes private property of the one man show."
 
 is this correct?


Pretty much, as I understand it.


 i snooped around and couldn't find much licensing info - except the compiler
 comes with a license.txt file that implies that it is owned by Digital Mars and
 Symantec.  i wonder what symantec has to do with anything...??


I believe that Walter had a hand in writing a C++ compiler for Symantec.

 http://www.walterbright.com/


-- 
Derek
(skype: derek.j.parnell)
Melbourne, Australia
18/08/2005 10:31:34 AM
Aug 17 2005
prev sibling parent "Ben Hinkle" <ben.hinkle gmail.com> writes:
"prefetch" <prefetch_member pathlink.com> wrote in message 
news:de0kl0$ml2$1 digitaldaemon.com...

 hm.

 so let me translate:

 1. "this is a one man show."


who responds to email (eventually or usually) and sometimes responds to 
newsgroup threads.


 2. "if you want the one man to do something, you better figure out how to 
 get
 his attention."


For localized things I've had luck posting about changes, getting feedback, 
making the changes, making the unittests, updating the doc and emailing 
Walter with a link to the discussion. For bigger changes I'm at a loss how 
to proceed.


 3. "anything you do to help becomes private property of the one man show."

 is this correct?


Various parts of phobos have non-Walter (or non-DigitalMars) copyrights. 
Some of it is in the public domain. I like to put my stuff in the public 
domain since I don't plan on selling it and I don't care if I lose interest 
and/or someone else starts selling it. You can also make your own libraries 
with whatever copyright/license you want.


 i snooped around and couldn't find much licensing info - except the 
 compiler
 comes with a license.txt file that implies that it is owned by Digital 
 Mars and
 Symantec.  i wonder what symantec has to do with anything...??


Different parts come with different licenses. For example poke around in the 
etc phobos directory and I think none of it is DigitalMars.
Aug 17 2005
prev sibling next sibling parent AJG <AJG_member pathlink.com> writes:
Hi,


Sorry to be the bearer of bad news, but digitalmars.D.bugs *is* the bug
tracking 'system'. Fortunately, Thomas Kühne is helping us with the
'tracking' part.


holy crap batman, who is driving this boat?? ;-)

how on earth can you expect to develop a language and runtime library without a
bug tracking system?  i mean, this is just plain stupid.  we all know, there
area whole bunch of free, web based bug trackers.

don't get me wrong, i've really enjoyed learning about D, and i think it's a
stellar language, but COME ON PEOPLE - let's get it together here.  somewhere
the list of "how to start a super successful grassroots technology" is 'setup a
web based bug tracking system so the community can help identify and fix bugs
for the primary developer(s).'

are my expectations just way to high or something?  shouldn't this project
reflect the quality of the language concept?  i mean, it's a great concept
how about we help it not fade into obscurity before it even gets going by
building tools (docs, bug tracker, etc.) that can allow the community to grow?


FWIW, I concur with this general sentiment. The PHP community has done some
truly outstanding work in this regard. I wish some of that would be emulated for
D in the form of better web usage (forum/tracker/docs/feedback/etc.).

Cheers,
--AJG.
Aug 17 2005
prev sibling parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
"prefetch" <prefetch_member pathlink.com> wrote in message 
news:de0fam$ivb$1 digitaldaemon.com...

 In article <1bl4gtbuuy6ek.12qm3vcv0nnxa$.dlg 40tude.net>, Derek Parnell 
 says...


Sorry to be the bearer of bad news, but digitalmars.D.bugs *is* the bug
tracking 'system'. Fortunately, Thomas Kühne is helping us with the
'tracking' part.


 holy crap batman, who is driving this boat?? ;-)

 how on earth can you expect to develop a language and runtime library 
 without a
 bug tracking system?  i mean, this is just plain stupid.  we all know, 
 there are
 a whole bunch of free, web based bug trackers.

 don't get me wrong, i've really enjoyed learning about D, and i think it's 
 a
 stellar language, but COME ON PEOPLE - let's get it together here. 
 somewhere on
 the list of "how to start a super successful grassroots technology" is 
 'setup a
 web based bug tracking system so the community can help identify and fix 
 bugs
 for the primary developer(s).'

 are my expectations just way to high or something?  shouldn't this project
 reflect the quality of the language concept?  i mean, it's a great 
 concept - but
 how about we help it not fade into obscurity before it even gets going by
 building tools (docs, bug tracker, etc.) that can allow the community to 
 grow?


One issue with the community fixing bugs is that many of the bugs are in the 
compiler and those we can't really help with - and besides I think probably 
only David Friedman (author of gdc) knows compiler stuff well enough to 
help. About phobos and the doc the community can definitely help but I see 
several reasons why there hasn't been more progress:
1) the process for making changes isn't well communicated - eg - post 
proposal, write unittests, update doc, etc
2) the scope of what Walter expects isn't communicated - what is he willing 
to change? I think he has said phobos is temporary - what's the plan for it? 
Who should fix the doc?
3) the community likes to do what they want - there are several different 
libraries that are fairly complete with overlapping features with phobos but 
different philosophies so little effort is being put into phobos and the 
doc. Individuals will either submit a new module or submit some bug fixes 
but it's fairly quiet. Personally I'm waiting on hearing more about item 2 
before really helping on phobos.
Aug 18 2005
parent John Reimer <terminal.node gmail.com> writes:
Ben Hinkle wrote:


 
 One issue with the community fixing bugs is that many of the bugs are in the 
 compiler and those we can't really help with - and besides I think probably 
 only David Friedman (author of gdc) knows compiler stuff well enough to 
 help. About phobos and the doc the community can definitely help but I see 
 several reasons why there hasn't been more progress:
 1) the process for making changes isn't well communicated - eg - post 
 proposal, write unittests, update doc, etc
 2) the scope of what Walter expects isn't communicated - what is he willing 
 to change? I think he has said phobos is temporary - what's the plan for it? 
 Who should fix the doc?
 3) the community likes to do what they want - there are several different 
 libraries that are fairly complete with overlapping features with phobos but 
 different philosophies so little effort is being put into phobos and the 
 doc. Individuals will either submit a new module or submit some bug fixes 
 but it's fairly quiet. Personally I'm waiting on hearing more about item 2 
 before really helping on phobos. 
 
 


I should stay out of this, because usually when I open my mouth in this 
regard, I get myself in trouble.  Yet, I wanted to mention what I've 
observed.

It seems that Walter has been very hesitant to include code/bug fixes in 
phobos from the community if the submitter doesn't clearly indicate the 
"openness" of the submitted code.  He appears to fear future litigation 
or the submitter demanding code rights later or something to that effect.

Perhaps many of the bug fixers are not putting his mind adequately to 
rest.  I'm not quite sure.  That or it's just the huge burden he 
carries, and he's only able to audit so many fixes.

-JJR
Aug 19 2005
prev sibling parent "Ben Hinkle" <ben.hinkle gmail.com> writes:
"prefetch" <prefetch_member pathlink.com> wrote in message 
news:de094m$em7$1 digitaldaemon.com...

 thanks ben.  sounds like D is broken in this regard.  exiting main w/o 
 cleaning
 up threads should not result in a segfault.  just to make supre duper 
 sure, i
 wrote a quick test in C using pthreads, and it has no problems.


That's because C doesn't try to run a final GC when main exits - which hoses 
the threads that could be still running. Mike Swieton posted a fix for 
src/phobos/internal/dmain2.d that inserted some code after main returned and 
before the moduleDtor and gc_term calls. I don't actually remember the 
details, though. See the thread 
http://www.digitalmars.com/d/archives/digitalmars/D/bugs/156.html.
I don't think the code was ever incoporated by Walter - and I don't think he 
ever commented on the issue in general.


 so, is there a D bug tracker?
 how do i submit a bug?

 i know there is a D bug newsgroup, but i'm assuming that's a discussion 
 forum
 and not an actual bug tracking system (i hope..)

 In article <ddvv95$77p$1 digitaldaemon.com>, Ben Hinkle says...


D doesn't wait for child threads to finish before trying to do a gc and
exit. This causes nasty seg-v's on exit. Get around the problem by making
sure all child threads are done before returning from main().
See for example
http://www.digitalmars.com/d/archives/digitalmars/D/13528.html and
http://www.digitalmars.com/d/archives/digitalmars/D/6419.html and probably
more.
Aug 17 2005