www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Runtime code reloading in D, part 1

reply Benjamin Thaut <code benjamin-thaut.de> writes:
An article about runtime code reloading in the context of game 
developement. A topic I'm currently working on in my spare time. I hope 
it holds some valuable information for everyone working with D.

http://3d.benjamin-thaut.de/?p=25

Kind Regards
Benjamin Thaut
Dec 30 2012
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Benjamin Thaut:


 http://3d.benjamin-thaut.de/?p=25



The pointer to the RTTI information for each member is a 
workaround for a strange bug I encountered. For POD structs the 
rtInfo template will be instancated, but for some reason you can 
not query the results at runtime. It is possible that the linker 
stripps the information away but I could not yet investigate 
this issue further. If I can not query the RTTI information at 
runtime I fallback to the RTTI info pointer stored in the “next” 
member of thMemberInfo.<


This seems one of the situations where knowing "why" is almost 
more important than "doing": understanding the cause, and maybe 
even fixing it, will help future D programmers.

Bye,
bearophile
Dec 30 2012
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 30.12.2012 14:27, schrieb bearophile:

 This seems one of the situations where knowing "why" is almost more
 important than "doing": understanding the cause, and maybe even fixing
 it, will help future D programmers.

 Bye,
 bearophile


The problem only happens if you have a POD struct in a library file 
(.lib). If the POD struct is directly in the main executable everything 
works just fine. The only hint Walter gave me when I opened a thread in 
the D newsgroup was "look at the difference of the object files".

I don't really know how that should be done atm.

Kind Regards
Benjamin Thaut
Dec 30 2012
prev sibling next sibling parent "F i L" <witte2008 gmail.com> writes:
On Sunday, 30 December 2012 at 12:32:00 UTC, Benjamin Thaut wrote:

 An article about runtime code reloading in the context of game 
 developement. A topic I'm currently working on in my spare 
 time. I hope it holds some valuable information for everyone 
 working with D.

 http://3d.benjamin-thaut.de/?p=25

 Kind Regards
 Benjamin Thaut


Awesome, thanks for the post. I've been playing around with this 
idea for awhile, and that article is immensely helpful.

Proper .so/.dll support is very needed for this, and I hope 
something is happening on that front.
Dec 30 2012
prev sibling next sibling parent reply r_m_r <r_m_r mailinator.com> writes:
On 12/30/2012 06:02 PM, Benjamin Thaut wrote:

 An article about runtime code reloading in the context of game
 developement. A topic I'm currently working on in my spare time. I hope
 it holds some valuable information for everyone working with D.


Thanks for the article. It was very informative. Waiting for the next 
part :-)

regards,
r_m_r
Dec 30 2012
parent reply "DypthroposTheImposter" <mcbracket gmail.com> writes:
   Do you find that D without GC is more effective than C++? Seems 
like you would be stuck using structs which seems somewhat 
limiting, even compared to C++...

  UE4 has similar reloading capabilities(using DLLs), though they 
use C++ and rely more on the ability to serialize everything
Dec 31 2012
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 31.12.2012 15:02, schrieb DypthroposTheImposter:

    Do you find that D without GC is more effective than C++? Seems like
 you would be stuck using structs which seems somewhat limiting, even
 compared to C++...

   UE4 has similar reloading capabilities(using DLLs), though they use
 C++ and rely more on the ability to serialize everything


Why should I be stuck with structs? Its the exact same as in C++. I did 
build my own new and delete operators (as templates). It's not that hard 
and works pretty well, only the syntax is not ideal in some cases. I can 
use everything you can use with D+GC. Sometimes you have to be carefull 
with implict allocations (e.g. closures, array literals) but I have a 
leak detector that points me to these directly and usually its easy to 
free these manually.

And I'm quite a bit more productive then in C++. Module constructors 
with a defined order instead of random static initalizers,
code generation isnstead of huge amounts of boilerplate code and many 
other features are the cause of this.

Kind Regards
Benjamin Thaut
Dec 31 2012
parent reply "ixid" <nuaccount gmail.com> writes:
On Monday, 31 December 2012 at 14:40:48 UTC, Benjamin Thaut wrote:

 Am 31.12.2012 15:02, schrieb DypthroposTheImposter:

   Do you find that D without GC is more effective than C++? 
 Seems like
 you would be stuck using structs which seems somewhat 
 limiting, even
 compared to C++...

  UE4 has similar reloading capabilities(using DLLs), though 
 they use
 C++ and rely more on the ability to serialize everything


 Why should I be stuck with structs? Its the exact same as in 
 C++. I did build my own new and delete operators (as 
 templates). It's not that hard and works pretty well, only the 
 syntax is not ideal in some cases. I can use everything you can 
 use with D+GC. Sometimes you have to be carefull with implict 
 allocations (e.g. closures, array literals) but I have a leak 
 detector that points me to these directly and usually its easy 
 to free these manually.

 And I'm quite a bit more productive then in C++. Module 
 constructors with a defined order instead of random static 
 initalizers,
 code generation isnstead of huge amounts of boilerplate code 
 and many other features are the cause of this.

 Kind Regards
 Benjamin Thaut


Is D moving away from your sort of use? Games and bioinformatics
would seem to be the areas the language should be trying to get
people to start using it in. The features you're using seem very
much like they should be a part and mode of using the language.
Jan 04 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 05.01.2013 05:24, schrieb ixid:

 On Monday, 31 December 2012 at 14:40:48 UTC, Benjamin Thaut wrote:

 Am 31.12.2012 15:02, schrieb DypthroposTheImposter:

   Do you find that D without GC is more effective than C++? Seems like
 you would be stuck using structs which seems somewhat limiting, even
 compared to C++...

  UE4 has similar reloading capabilities(using DLLs), though they use
 C++ and rely more on the ability to serialize everything


 Why should I be stuck with structs? Its the exact same as in C++. I
 did build my own new and delete operators (as templates). It's not
 that hard and works pretty well, only the syntax is not ideal in some
 cases. I can use everything you can use with D+GC. Sometimes you have
 to be carefull with implict allocations (e.g. closures, array
 literals) but I have a leak detector that points me to these directly
 and usually its easy to free these manually.

 And I'm quite a bit more productive then in C++. Module constructors
 with a defined order instead of random static initalizers,
 code generation isnstead of huge amounts of boilerplate code and many
 other features are the cause of this.

 Kind Regards
 Benjamin Thaut


 Is D moving away from your sort of use? Games and bioinformatics
 would seem to be the areas the language should be trying to get
 people to start using it in. The features you're using seem very
 much like they should be a part and mode of using the language.


I wouldn't say its moving away from it. Some recent changes to druntime 
have made it significantly less leaking. But on the other hand a API 
design like toString() which pretty much does leak in almost all cases 
don't exactly help a GC free D. In Summary it feels to me that GC free D 
is not important to the community or the active contributors.

I also see D's biggest chances in becoming popular in the performance 
critical fields of programming. Which would be systems programming, 
Gaming and others. For programming fields in which a GC is applicable 

safe (see the recent "ref is unsafe" discussion in the newsgroup), they 
have nice productivity features like runtime code changing through the 
VM and have a way better GC because the language was designed from the 
beginning to support a advanced GC.

Kind Regards
Benjamin Thaut
Jan 06 2013
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Benjamin Thaut:


 In Summary it feels to me that GC free D is not important
 to the community or the active contributors.


I think it will become more important for them, in future. At the 
moment the work is mostly on finishing immutability, purity, 
shared, and other parts of the core language, and working on 
process characteristics (like GIT workflows) that are needed to 
make everything else work efficiently.



 they have nice productivity features like runtime code changing 
 through the VM


Some of those productivity features are important, and despite D 
doesn't run on a virtual machine I think some of them can be 
added to D, once enough attention and work is given on them.

Bye,
bearophile
Jan 06 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-06 11:37, Benjamin Thaut wrote:


 I wouldn't say its moving away from it. Some recent changes to druntime
 have made it significantly less leaking. But on the other hand a API
 design like toString() which pretty much does leak in almost all cases
 don't exactly help a GC free D. In Summary it feels to me that GC free D
 is not important to the community or the active contributors.


The design of toString() has been up for debate a couple of times. Many 
are not happy with the design.

-- 
/Jacob Carlborg
Jan 06 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 06 Jan 2013 12:10:38 +0100
schrieb Jacob Carlborg <doob me.com>:


 On 2013-01-06 11:37, Benjamin Thaut wrote:
 

 I wouldn't say its moving away from it. Some recent changes to
 druntime have made it significantly less leaking. But on the other
 hand a API design like toString() which pretty much does leak in
 almost all cases don't exactly help a GC free D. In Summary it
 feels to me that GC free D is not important to the community or the
 active contributors.


 
 The design of toString() has been up for debate a couple of times.
 Many are not happy with the design.
 


some modules already provide the (scope void delegate(const(char)[])
sink overload, it's also supported by to, format and writefln, afaik.
https://github.com/D-Programming-Language/phobos/blob/master/std/uuid.d#L806

I guess when we have custom allocators we can also provide toString
methods templated on allocators.
Jan 06 2013
parent reply David <d dav1d.de> writes:
 I guess when we have custom allocators we can also provide toString
 methods templated on allocators.
 


Custom allocators released together with HL3?
Jan 06 2013
parent Johannes Pfau <nospam example.com> writes:
Am Sun, 06 Jan 2013 13:14:39 +0100
schrieb David <d dav1d.de>:


 I guess when we have custom allocators we can also provide toString
 methods templated on allocators.
 


 
 Custom allocators released together with HL3?


A lot of things are blocked by custom allocators and this is indeed an
issue. It wouldn't make much sense to redefine toString without custom
allocators though.

BTW: I guess the Duke Nukem Forever joke doesn't work anyore ;-)
Jan 06 2013
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
06-Jan-2013 14:37, Benjamin Thaut пишет:

 Am 05.01.2013 05:24, schrieb ixid:

 On Monday, 31 December 2012 at 14:40:48 UTC, Benjamin Thaut wrote:

 Am 31.12.2012 15:02, schrieb DypthroposTheImposter:

   Do you find that D without GC is more effective than C++? Seems like
 you would be stuck using structs which seems somewhat limiting, even
 compared to C++...

  UE4 has similar reloading capabilities(using DLLs), though they use
 C++ and rely more on the ability to serialize everything


 Why should I be stuck with structs? Its the exact same as in C++. I
 did build my own new and delete operators (as templates). It's not
 that hard and works pretty well, only the syntax is not ideal in some
 cases. I can use everything you can use with D+GC. Sometimes you have
 to be carefull with implict allocations (e.g. closures, array
 literals) but I have a leak detector that points me to these directly
 and usually its easy to free these manually.

 And I'm quite a bit more productive then in C++. Module constructors
 with a defined order instead of random static initalizers,
 code generation isnstead of huge amounts of boilerplate code and many
 other features are the cause of this.

 Kind Regards
 Benjamin Thaut


 Is D moving away from your sort of use? Games and bioinformatics
 would seem to be the areas the language should be trying to get
 people to start using it in. The features you're using seem very
 much like they should be a part and mode of using the language.


 I wouldn't say its moving away from it. Some recent changes to druntime
 have made it significantly less leaking. But on the other hand a API
 design like toString() which pretty much does leak in almost all cases
 don't exactly help a GC free D. In Summary it feels to me that GC free D
 is not important to the community or the active contributors.


Regarding toString there is a better signature that avoids useless 
allocations:

void toString(scope void delegate(const (char)[]) sink);

It takes a delegate to output string representation directly to the 
destination via 'sink' delegate (that may e.g. write chars to file). 
Plus the 'scope' part of declaration avoids allocating the said delegate 
on the heap.

I'd even say that string toString(); is an artifact of the past, instead 
to!string should be used (if allocating a string is fine).


 I also see D's biggest chances in becoming popular in the performance
 critical fields of programming. Which would be systems programming,
 Gaming and others. For programming fields in which a GC is applicable

 safe (see the recent "ref is unsafe" discussion in the newsgroup), they
 have nice productivity features like runtime code changing through the
 VM and have a way better GC because the language was designed from the
 beginning to support a advanced GC.

 Kind Regards
 Benjamin Thaut



-- 
Dmitry Olshansky
Jan 06 2013
parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Sun, Jan 6, 2013 at 6:49 PM, Dmitry Olshansky <dmitry.olsh gmail.com>wrote:


 Regarding toString there is a better signature that avoids useless
 allocations:

 void toString(scope void delegate(const (char)[]) sink);

 It takes a delegate to output string representation directly to the
 destination via 'sink' delegate (that may e.g. write chars to file). Plus
 the 'scope' part of declaration avoids allocating the said delegate on the
 heap.

 I'd even say that string toString(); is an artifact of the past, instead
 to!string should be used (if allocating a string is fine).



I know you explained that already, but see, that again slipped my mind.
This toString(sink) thingy seems cool, but is there a documentation on it
somewhere? Without a related doc, I fear no one will know this exists.
Jan 06 2013
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
06-Jan-2013 23:55, Philippe Sigaud пишет:

 On Sun, Jan 6, 2013 at 6:49 PM, Dmitry Olshansky <dmitry.olsh gmail.com
 <mailto:dmitry.olsh gmail.com>> wrote:

     Regarding toString there is a better signature that avoids useless
     allocations:

     void toString(scope void delegate(const (char)[]) sink);

     It takes a delegate to output string representation directly to the
     destination via 'sink' delegate (that may e.g. write chars to file).
     Plus the 'scope' part of declaration avoids allocating the said
     delegate on the heap.

     I'd even say that string toString(); is an artifact of the past,
     instead to!string should be used (if allocating a string is fine).


 I know you explained that already, but see, that again slipped my mind.
 This toString(sink) thingy seems cool, but is there a documentation on
 it somewhere? Without a related doc, I fear no one will know this exists.


Guess I need to find the original on toString and bit-blit it over with 
this new sink thingy ;)

-- 
Dmitry Olshansky
Jan 06 2013
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
07-Jan-2013 00:02, Dmitry Olshansky пишет:

 06-Jan-2013 23:55, Philippe Sigaud пишет:

 On Sun, Jan 6, 2013 at 6:49 PM, Dmitry Olshansky <dmitry.olsh gmail.com
 <mailto:dmitry.olsh gmail.com>> wrote:

     Regarding toString there is a better signature that avoids useless
     allocations:

     void toString(scope void delegate(const (char)[]) sink);

     It takes a delegate to output string representation directly to the
     destination via 'sink' delegate (that may e.g. write chars to file).
     Plus the 'scope' part of declaration avoids allocating the said
     delegate on the heap.

     I'd even say that string toString(); is an artifact of the past,
     instead to!string should be used (if allocating a string is fine).


 I know you explained that already, but see, that again slipped my mind.
 This toString(sink) thingy seems cool, but is there a documentation on
 it somewhere? Without a related doc, I fear no one will know this exists.


 Guess I need to find the original on toString and bit-blit it over with
 this new sink thingy ;)


There is a short note in object.d:
...
string toString();
		Convert Object to a human readable string.
...

That is any class may override it.

However the real relevant golden bit of information is buried after a 
row of formatValue templates (that hardly makes any sense to newcomers) 
at the bottom of std.format page:
(http://dlang.org/phobos/std_format.html)

Aggregates (struct, union, class, and interface) are basically formatted 
by calling toString.

toString should have one of the following signatures:
const void toString(scope void delegate(const(char)[]) sink, FormatSpec 
fmt);
const void toString(scope void delegate(const(char)[]) sink, string fmt);
const void toString(scope void delegate(const(char)[]) sink);
const string toString();


So it's quite flexible. Even better then I've been advertising it.
Note that classes are covered. I bet it can call virtual functions 
through base reference if base defines the new variation of toString.


The problem is if it's going to be extended to wchar/dchar... At least 
wchar is useful on Windows.

But the real fun is that I fail to find any real description of toString 
in the docs for d-p-l website!

The only files in d-p-l site where it is found in are:

- errors.dd:
... class Error has a pure virtual a function called toString() which 
produces a char[] with a human readable description of the error.)

Nothing useful and it's out of date! It returns a string (immutable) 
these days.

- lazy-evaluation.dd uses as an example of expensive computation and has 
the form of toString(i) where i is an int (wtf?!).

Probably horribly out of date as it should use either format or to!string.

- tempalte.dd uses std.string.toString as an example (same as in lazy?) 
and it's again an out of date piece of crap.

And a couple more files call it here and there (e.g. windows.dd to 
display exception message box).


-- 
Dmitry Olshansky
Jan 06 2013
prev sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 06.01.2013 20:55, schrieb Philippe Sigaud:

 On Sun, Jan 6, 2013 at 6:49 PM, Dmitry Olshansky <dmitry.olsh gmail.com
 <mailto:dmitry.olsh gmail.com>> wrote:

     Regarding toString there is a better signature that avoids useless
     allocations:

     void toString(scope void delegate(const (char)[]) sink);

     It takes a delegate to output string representation directly to the
     destination via 'sink' delegate (that may e.g. write chars to file).
     Plus the 'scope' part of declaration avoids allocating the said
     delegate on the heap.

     I'd even say that string toString(); is an artifact of the past,
     instead to!string should be used (if allocating a string is fine).


 I know you explained that already, but see, that again slipped my mind.
 This toString(sink) thingy seems cool, but is there a documentation on
 it somewhere? Without a related doc, I fear no one will know this exists.


There was a language change request for it in the wiki, but I can't find 
it anymore.

Kind Regards
Benjamin Thaut
Jan 06 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-12-30 13:32, Benjamin Thaut wrote:

 An article about runtime code reloading in the context of game
 developement. A topic I'm currently working on in my spare time. I hope
 it holds some valuable information for everyone working with D.

 http://3d.benjamin-thaut.de/?p=25


This looks very cool. Question, are you manually triggering the code for 
generating the RTTI?

BTW, have you seen this old project implementing runtime reflection:

http://flectioned.kuehne.cn/

-- 
/Jacob Carlborg
Jan 05 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 05.01.2013 12:15, schrieb Jacob Carlborg:

 On 2012-12-30 13:32, Benjamin Thaut wrote:

 An article about runtime code reloading in the context of game
 developement. A topic I'm currently working on in my spare time. I hope
 it holds some valuable information for everyone working with D.

 http://3d.benjamin-thaut.de/?p=25


 This looks very cool. Question, are you manually triggering the code for
 generating the RTTI?

 BTW, have you seen this old project implementing runtime reflection:

 http://flectioned.kuehne.cn/


Yes I know flectioned. But I did not require that much RTTI information. 
No I'M not manually triggering the code for generating the RTTI. As 
mentioned in the article it is done via the RTInfo template inside 
object_.d / object.di which is automatically instanciated for each type 
used during compilation.

Kind Regards
Benjamin Thaut
Jan 06 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-06 11:30, Benjamin Thaut wrote:


 Yes I know flectioned. But I did not require that much RTTI information.
 No I'M not manually triggering the code for generating the RTTI. As
 mentioned in the article it is done via the RTInfo template inside
 object_.d / object.di which is automatically instanciated for each type
 used during compilation.


Ah, so you modified the existing RTInfo? I didn't know that existed.

-- 
/Jacob Carlborg
Jan 06 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 06.01.2013 12:17, schrieb Jacob Carlborg:

 On 2013-01-06 11:30, Benjamin Thaut wrote:


 Yes I know flectioned. But I did not require that much RTTI information.
 No I'M not manually triggering the code for generating the RTTI. As
 mentioned in the article it is done via the RTInfo template inside
 object_.d / object.di which is automatically instanciated for each type
 used during compilation.


 Ah, so you modified the existing RTInfo? I didn't know that existed.


It does not exist. The current RTInfo template just outputs a null 
pointer for every type. It is planned that the RTInfo template will be 
used for a percise GC in the future. (Did you read the article? ^^)

Kind Regards
Benjamin Thaut
Jan 06 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-06 12:28, Benjamin Thaut wrote:


 It does not exist. The current RTInfo template just outputs a null
 pointer for every type.


Yes, but it still exists.


 It is planned that the RTInfo template will be
 used for a percise GC in the future. (Did you read the article? ^^)


Yes, I've read the article. How do you think I otherwise could have come 
up with these questions :)

I can see now that you wrote:

"This template is instanciated for every type that is used during 
compilation and thus is ideal to generate RTTI information"

I get it now.

-- 
/Jacob Carlborg
Jan 06 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 06.01.2013 12:50, schrieb Jacob Carlborg:

 On 2013-01-06 12:28, Benjamin Thaut wrote:


 It does not exist. The current RTInfo template just outputs a null
 pointer for every type.


 Yes, but it still exists.


 It is planned that the RTInfo template will be
 used for a percise GC in the future. (Did you read the article? ^^)


 Yes, I've read the article. How do you think I otherwise could have come
 up with these questions :)

 I can see now that you wrote:

 "This template is instanciated for every type that is used during
 compilation and thus is ideal to generate RTTI information"

 I get it now.


Here is the full implementation if you are interested:
https://github.com/Ingrater/druntime/blob/master/src/rtti.d

Kind Regards
Benjamin Thaut
Jan 06 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-06 13:19, Benjamin Thaut wrote:


 Here is the full implementation if you are interested:
 https://github.com/Ingrater/druntime/blob/master/src/rtti.d


Thanks. Is that possible to do without having to modify the runtime?

-- 
/Jacob Carlborg
Jan 06 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 06.01.2013 13:23, schrieb Jacob Carlborg:

 On 2013-01-06 13:19, Benjamin Thaut wrote:


 Here is the full implementation if you are interested:
 https://github.com/Ingrater/druntime/blob/master/src/rtti.d


 Thanks. Is that possible to do without having to modify the runtime?


No. The template has to be known inside of object_.d/object.di so it is 
not possible without modifing the runtime. Also you usually want to have 
RTTI info for druntime/phobos too.
Jan 06 2013