digitalmars.D - Bug in phobos Thread module?
- Babele Dunnit (7/7) May 11 2007 Hi all,
- Regan Heath (2/2) May 11 2007 I suspect you have found a bug, in which case it should be reported in t...
- david (4/7) May 11 2007 look here:
- Chris Miller (6/10) May 11 2007 =
- Walter Bright (3/10) May 11 2007 I'm not too experienced with threading, so if you could post a patch
- kenny (2/26) May 12 2007 I also experience the same issue in linux. It happens after I create/del...
- Regan Heath (3/4) May 12 2007 Each operating system has a maximum number of handles, it differs for op...
- kenny (2/8) May 12 2007 wow, now it's only giving me 382 threads... here is test code. Later tod...
- Lars Ivar Igesund (9/25) May 12 2007 For what it's worth, the Tango version of Thread has several fixes and
- kenny (6/27) May 12 2007 well, it looks like I'll be doing one of two things this weekend :)
- Lars Ivar Igesund (9/13) May 12 2007 I can also suggest
- kenny (2/16) May 12 2007 awesome, thanks for the link. Yep, I meant phobos :)
- Regan Heath (15/27) May 12 2007 I've no idea how to do a patch as such but I believe all you need is a c...
- Regan Heath (27/27) May 13 2007 Thought you might want some test code to demonstrate the problem.
- Regan Heath (2/8) May 13 2007 Regan Heath
- Vladimir Panteleev (5/6) May 13 2007 I don't see why this wouldn't work, with the condition that you don't ac...
- Daniel Keep (19/24) May 13 2007 The problem I can see with this is that *something* must still have a
- Vladimir Panteleev (6/19) May 13 2007 Returning pointers to local variables is comparable to this situation on...
- Daniel Keep (35/54) May 13 2007 I *know* how classes work; I wrote assembler code a while back to figure
Hi all, I am a newbie with D but experienced C++ developer.. this is my first post, so I MUST say that D ROCKS! Really impressive language. Walter, you are definitely the Lord of Compilers. Back to the subject, I am writing a massively multithreaded piece of code, and after some time I get a "failed to start" error. I digged in the forums and found someone else with same problem, but no answer. So was time to dig into Thread sources... ...and I see a static destructor cleaning a single handle (the main thread, I suppose), but no CloseHandle on any other handle created via _beginthreadex (I am talking about Windows, I should have specified before, sorry). I believe there should be an explicit Thread destructor able to free the handle via CloseHandle; also, because thread handles under Windows are a limited resource, probably a RIAA scheme (or explicit "delete" call) should be used, instead of waiting for the GC to pass by.. So, I added a CloseHandle(mythread.hdl) call and now my handles count (in the Task Manager) is much more under control... ciao Bab
May 11 2007
I suspect you have found a bug, in which case it should be reported in the bug tracker... Now, as I've been away for 6 months and forgotten, can someone tell us both how that's done. Regan
May 11 2007
Regan Heath schrieb:I suspect you have found a bug, in which case it should be reported in the bug tracker... Now, as I've been away for 6 months and forgotten, can someone tell us both how that's done. Reganlook here: http://d.puremagic.com/issues/index.cgi david
May 11 2007
On Fri, 11 May 2007 18:47:04 -0400, Regan Heath <regan netmail.co.nz> = wrote:I suspect you have found a bug, in which case it should be reported in==the bug tracker... Now, as I've been away for 6 months and forgotten,==can someone tell us both how that's done. ReganIt has been reported and confirmed several times: http://d.puremagic.com/issues/show_bug.cgi?id=3D318
May 11 2007
Babele Dunnit wrote:I am a newbie with D but experienced C++ developer.. this is my first post, so I MUST say that D ROCKS! Really impressive language. Walter, you are definitely the Lord of Compilers. Back to the subject, I am writing a massively multithreaded piece of code, and after some time I get a "failed to start" error. I digged in the forums and found someone else with same problem, but no answer. So was time to dig into Thread sources... ...and I see a static destructor cleaning a single handle (the main thread, I suppose), but no CloseHandle on any other handle created via _beginthreadex (I am talking about Windows, I should have specified before, sorry). I believe there should be an explicit Thread destructor able to free the handle via CloseHandle; also, because thread handles under Windows are a limited resource, probably a RIAA scheme (or explicit "delete" call) should be used, instead of waiting for the GC to pass by.. So, I added a CloseHandle(mythread.hdl) call and now my handles count (in the Task Manager) is much more under control...I'm not too experienced with threading, so if you could post a patch that would be welcome.
May 11 2007
Walter Bright wrote:Babele Dunnit wrote:I also experience the same issue in linux. It happens after I create/delete over 8000 threads. I dunno why 8000 is the magic number, but it is for my machine. I'll try and look into it as well on linux. Just wanted to let you know that the problem also occurs in linuxI am a newbie with D but experienced C++ developer.. this is my first post, so I MUST say that D ROCKS! Really impressive language. Walter, you are definitely the Lord of Compilers. Back to the subject, I am writing a massively multithreaded piece of code, and after some time I get a "failed to start" error. I digged in the forums and found someone else with same problem, but no answer. So was time to dig into Thread sources... ...and I see a static destructor cleaning a single handle (the main thread, I suppose), but no CloseHandle on any other handle created via _beginthreadex (I am talking about Windows, I should have specified before, sorry). I believe there should be an explicit Thread destructor able to free the handle via CloseHandle; also, because thread handles under Windows are a limited resource, probably a RIAA scheme (or explicit "delete" call) should be used, instead of waiting for the GC to pass by.. So, I added a CloseHandle(mythread.hdl) call and now my handles count (in the Task Manager) is much more under control...I'm not too experienced with threading, so if you could post a patch that would be welcome.
May 12 2007
kenny Wrote:I also experience the same issue in linux. It happens after I create/delete over 8000 threads. I dunno why 8000 is the magic number, but it is for my machine. I'll try and look into it as well on linux. Just wanted to let you know that the problem also occurs in linuxEach operating system has a maximum number of handles, it differs for operating system, each version of each operating system, and can be defined for each process upon execution as well. Or so I recall from my travels. Regan Heath
May 12 2007
Regan Heath wrote:kenny Wrote:wow, now it's only giving me 382 threads... here is test code. Later today, I will dip into phobos and try and find the problem.I also experience the same issue in linux. It happens after I create/delete over 8000 threads. I dunno why 8000 is the magic number, but it is for my machine. I'll try and look into it as well on linux. Just wanted to let you know that the problem also occurs in linuxEach operating system has a maximum number of handles, it differs for operating system, each version of each operating system, and can be defined for each process upon execution as well. Or so I recall from my travels. Regan Heath
May 12 2007
kenny wrote:Regan Heath wrote:For what it's worth, the Tango version of Thread has several fixes and changes compared to the Phobos version, and also provides other features like thread local storage and fibers/co-routines. -- Lars Ivar Igesund blog at http://larsivi.net DSource, #d.tango & #D: larsivi Dancing the Tangokenny Wrote:wow, now it's only giving me 382 threads... here is test code. Later today, I will dip into phobos and try and find the problem.I also experience the same issue in linux. It happens after I create/delete over 8000 threads. I dunno why 8000 is the magic number, but it is for my machine. I'll try and look into it as well on linux. Just wanted to let you know that the problem also occurs in linuxEach operating system has a maximum number of handles, it differs for operating system, each version of each operating system, and can be defined for each process upon execution as well. Or so I recall from my travels. Regan Heath
May 12 2007
Lars Ivar Igesund wrote:kenny wrote:well, it looks like I'll be doing one of two things this weekend :) 1. read http://yolinux.com/TUTORIALS/LinuxTutorialPosixThreads.html 2. install tango / tangbos because I have to make a program by wednesday that needs to create unlimited threads over unlimited time (eg a daemon of sorts). I'll check out tango, as it sounds a lot easier than bugfixing tango. Thanks for the tip :) KennyRegan Heath wrote:For what it's worth, the Tango version of Thread has several fixes and changes compared to the Phobos version, and also provides other features like thread local storage and fibers/co-routines.kenny Wrote:wow, now it's only giving me 382 threads... here is test code. Later today, I will dip into phobos and try and find the problem.I also experience the same issue in linux. It happens after I create/delete over 8000 threads. I dunno why 8000 is the magic number, but it is for my machine. I'll try and look into it as well on linux. Just wanted to let you know that the problem also occurs in linuxEach operating system has a maximum number of handles, it differs for operating system, each version of each operating system, and can be defined for each process upon execution as well. Or so I recall from my travels. Regan Heath
May 12 2007
kenny wrote:well, it looks like I'll be doing one of two things this weekend :) 1. read http://yolinux.com/TUTORIALS/LinuxTutorialPosixThreads.htmlI can also suggest http://www.dsource.org/projects/tango/wiki/ChapterThreading... as it sounds a lot easier than bugfixing tango.I hope you meant Phobos ;) -- Lars Ivar Igesund blog at http://larsivi.net DSource, #d.tango & #D: larsivi Dancing the Tango
May 12 2007
Lars Ivar Igesund wrote:kenny wrote:awesome, thanks for the link. Yep, I meant phobos :)well, it looks like I'll be doing one of two things this weekend :) 1. read http://yolinux.com/TUTORIALS/LinuxTutorialPosixThreads.htmlI can also suggest http://www.dsource.org/projects/tango/wiki/ChapterThreading... as it sounds a lot easier than bugfixing tango.I hope you meant Phobos ;)
May 12 2007
Walter Bright Wrote:Babele Dunnit wrote:I've no idea how to do a patch as such but I believe all you need is a call to CloseHandle in: extern (Windows) static uint threadstart(void *p) as this wrapper is used for all threads and the stuff at the end of the wrapper removes the thread from allThreads and does cleanup. eg debug (thread) printf("Ending thread %d\n", t.idx); version (Win32) { CloseHandle(t.hdl); t.hdl = cast(thread_hdl)0; } t.state = TS.TERMINATED; allThreads[t.idx] = null; I cannot recall (nor do I have old code I can refer to, sadly) whether there is a similar call required on Linux etc, I suspect not given that pthread_create returns a thread id, as opposed to a handle per-se. I have vague recollections of calling kill, but that may be to forcibly terminate a thread as opposed to cleaning up a handle. Regan HeathI am a newbie with D but experienced C++ developer.. this is my first post, so I MUST say that D ROCKS! Really impressive language. Walter, you are definitely the Lord of Compilers. Back to the subject, I am writing a massively multithreaded piece of code, and after some time I get a "failed to start" error. I digged in the forums and found someone else with same problem, but no answer. So was time to dig into Thread sources... ...and I see a static destructor cleaning a single handle (the main thread, I suppose), but no CloseHandle on any other handle created via _beginthreadex (I am talking about Windows, I should have specified before, sorry). I believe there should be an explicit Thread destructor able to free the handle via CloseHandle; also, because thread handles under Windows are a limited resource, probably a RIAA scheme (or explicit "delete" call) should be used, instead of waiting for the GC to pass by.. So, I added a CloseHandle(mythread.hdl) call and now my handles count (in the Task Manager) is much more under control...I'm not too experienced with threading, so if you could post a patch that would be welcome.
May 12 2007
Thought you might want some test code to demonstrate the problem. If you remove the call to CloseHandle and configure task manager to show handles and threads you should see the number of threads at a constant, around 17, and the number of handles steadily climbing. The addition of CloseHandle causes the number of handles to remain steady also, at approximately 2x the number of threads. The correct location for the call to CloseHandle, I believe, is at the end of threadstart, as shown in my other reply. Can someone run this same code on linux (without CloseHandle etc) and use 'top' or similar to watch the thread and handle counts, I suspect no fix is required for linux but I may be mistaken.
May 13 2007
One further question, is it actually valid to call delete this in a non-static method of a class as shown below. It seems to work but I would not have thought it possible...Regan Heath
May 13 2007
On Sun, 13 May 2007 18:23:13 +0300, Regan Heath <regan netmail.co.nz> wrote:One further question, is it actually valid to call delete this in a non-static method of a class as shown below. It seems to work but I would not have thought it possible...I don't see why this wouldn't work, with the condition that you don't access class fields or virtual methods after deleting the object (this includes invariants). The above might work in very specific circumstances, but you should never rely on accessing data marked as deallocated. -- Best regards, Vladimir mailto:thecybershadow gmail.com
May 13 2007
Vladimir Panteleev wrote:On Sun, 13 May 2007 18:23:13 +0300, Regan Heath <regan netmail.co.nz> wrote:The problem I can see with this is that *something* must still have a reference to the object, otherwise the method couldn't have been called. And if that's the case, that reference is now left dangling. (Plus, you can't use this on any class with invariants or they'll blow up.) I suppose, if you were careful, it would be OK. But frankly, it just seems wrong... a bit like returning pointers to local variables based on the assumption they get used before calling something else; technically feasible, but just *asking* for problems. -- Daniel -- int getRandomNumber() { return 4; // chosen by fair dice roll. // guaranteed to be random. } http://xkcd.com/ v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/One further question, is it actually valid to call delete this in a non-static method of a class as shown below. It seems to work but I would not have thought it possible...I don't see why this wouldn't work, with the condition that you don't access class fields or virtual methods after deleting the object (this includes invariants). The above might work in very specific circumstances, but you should never rely on accessing data marked as deallocated.
May 13 2007
On Mon, 14 May 2007 02:11:56 +0300, Daniel Keep <daniel.keep.lists gmail.com> wrote:Vladimir Panteleev wrote:Not really. The object context, aka "this", is nothing more than a "hidden" function parameter. Classes are just pointers to structs with VMTs. So "delete this" isn't different than deallocating something the reference to which was given to you as a function parameter. The same logic applies - the calling code shouldn't use the object reference after calling the function/method, just like you wouldn't use a reference after you delete it explicitely :)On Sun, 13 May 2007 18:23:13 +0300, Regan Heath <regan netmail.co.nz> wrote:The problem I can see with this is that *something* must still have a reference to the object, otherwise the method couldn't have been called. And if that's the case, that reference is now left dangling.One further question, is it actually valid to call delete this in a non-static method of a class as shown below. It seems to work but I would not have thought it possible...I don't see why this wouldn't work, with the condition that you don't access class fields or virtual methods after deleting the object (this includes invariants). The above might work in very specific circumstances, but you should never rely on accessing data marked as deallocated.I suppose, if you were careful, it would be OK. But frankly, it just seems wrong... a bit like returning pointers to local variables based on the assumption they get used before calling something else; technically feasible, but just *asking* for problems.Returning pointers to local variables is comparable to this situation only when you actually DO use the class's fields or virtual methods after you delete it. Just like with the stack, memory has been deallocated but usually not overwritten. In pre-multi-tasking OSes (MS-DOS specifically), using data left in the callees' stacks had a real risk of being overwritten by hardware interrupt handlers. -- Best regards, Vladimir mailto:thecybershadow gmail.com
May 13 2007
Vladimir Panteleev wrote:On Mon, 14 May 2007 02:11:56 +0300, Daniel Keep <daniel.keep.lists gmail.com> wrote:I *know* how classes work; I wrote assembler code a while back to figure out how the D ABI worked back before it was documented :PVladimir Panteleev wrote:Not really. The object context, aka "this", is nothing more than a "hidden" function parameter. Classes are just pointers to structs with VMTs. So "delete this" isn't different than deallocating something the reference to which was given to you as a function parameter. The same logic applies - the calling code shouldn't use the object reference after calling the function/method, just like you wouldn't use a reference after you delete it explicitely :)On Sun, 13 May 2007 18:23:13 +0300, Regan Heath <regan netmail.co.nz> wrote:The problem I can see with this is that *something* must still have a reference to the object, otherwise the method couldn't have been called. And if that's the case, that reference is now left dangling.One further question, is it actually valid to call delete this in a non-static method of a class as shown below. It seems to work but I would not have thought it possible...I don't see why this wouldn't work, with the condition that you don't access class fields or virtual methods after deleting the object (this includes invariants). The above might work in very specific circumstances, but you should never rely on accessing data marked as deallocated.My problem with this is basically the same reason Joel asserts that exceptions are evil: it's not that it necessarily introduces bugs or bad behaviour, but it sure as hell helps mask it! For instance: foo.run(); What about that line indicates that "foo" is no longer a valid reference? Absolutely nothing; you could argue that "the docs should state that the object deletes itself", but let's face it: reading the documentation is the *last* thing most people do. Programmers are lazy by their very definition (if we *weren't* lazy, we wouldn't be spending all this time and effort trying to get computers to do stuff for us.) Hell, I know I've looked at code before that did something sneaky and completely missed what it was doing, causing me no end of grief: and that's on code I wrote myself! On the other hand: foo.runThenDelete(); Is marginally better, but it's still obfuscating the deletion; what's it deleting? foo.run(); delete foo; Is not that much more typing, and is explicit about what's going on. Anyway, just my AU$0.02 :) -- Daniel -- int getRandomNumber() { return 4; // chosen by fair dice roll. // guaranteed to be random. } http://xkcd.com/ v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/I suppose, if you were careful, it would be OK. But frankly, it just seems wrong... a bit like returning pointers to local variables based on the assumption they get used before calling something else; technically feasible, but just *asking* for problems.Returning pointers to local variables is comparable to this situation only when you actually DO use the class's fields or virtual methods after you delete it. Just like with the stack, memory has been deallocated but usually not overwritten. In pre-multi-tasking OSes (MS-DOS specifically), using data left in the callees' stacks had a real risk of being overwritten by hardware interrupt handlers.
May 13 2007