www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP 45 - approval discussion

reply Benjamin Thaut <code benjamin-thaut.de> writes:
Attached you will find a e-mail conversation about the discussion of DIP 
45. I moved the discussion here so everyone can contribute.

Hi,

DIP 45 is waiting for some offical approval / comment so it can be 
implemented. It would be great if one of you could look into the topic 
and give official approval or improvement suggestions as the DIP has 
been in limbo for quite some time now. I'm not willing to start working 
on it before there is official approval, because I made bad experiences 
with this in the past.

In my opinion the DIP is in a state where further discussion will not 
lead to any improvements. The DIP should be implemented to see which 
problems arrise during implementation to then discuss how to solve them.

I've also seen a couple of questions to the state of Windows DLL support 
and shared libraries on the newsgroup. A working export keyword is 
required to actually implement that support on windows.

The DIP: http://wiki.dlang.org/DIP45
The latest discussion on the NG: 
http://forum.dlang.org/post/kvhu2c$2ikq$1 digitalmars.com
Bug associated with the problem: 
http://d.puremagic.com/issues/show_bug.cgi?id=9816

Kind Regards
Benjamin Thaut
Nov 10 2013
next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 03.11.2013 20:32, schrieb Andrei Alexandrescu:

 This looks good to me and solidly motivated and anchored in practical 


experience. I approve with the mention that I'm not an expert in the 
matters involved so for the most part I trust the authors with the details.

 Walter?


 Andrei
Nov 10 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 05.11.2013 00:32, schrieb Martin Nowak:

 On 11/03/2013 08:32 PM, Andrei Alexandrescu wrote:

 This looks good to me and solidly motivated and anchored in 




practical experience. I approve with the mention that I'm not an expert 
in the matters involved so for the most part I trust the authors with 
the details.

 Walter?


 I think most of the DIP is sane. We came up with some complications 


for Windows so that

 export can replace both __declspec(dllimport) and 


__declspec(dllexport). I'd like to hear

 what Walter thinks about this.

 Otherwise I agree with Benjamin that it's mainly an issue of 


implementation now.
Nov 10 2013
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 08.11.2013 20:32, schrieb Walter Bright:

 It looks pretty good, except I have serious reservations about the 


-lib switch proposed behavior:

 1. It's too blunt. A user could conceivably want to export some 


symbols and not others. This is all or nothing.

 2. It requires the compiler to rewrite the .obj files, even ones that 


dmd did not create. I would consider this very surprising behavior. The 
user will be "WTF the object files in the lib are different from on my 
disk! D sux!" -lib is a simple-minded creature that currently does 
exactly what one would expect it to.

 3. I've never heard anyone complain about the issue of a DLL 


referencing another DLL and the combined list of exports being the 
concatenation of both. It can be annoying, but it would be a minor 
issue. If it really is one, we can write a separate simple tool that 
strips exports from object files.

 And:

 Exporting of TLS variables is a minor issue. Frankly, I think people 


who export global variables should be burned at the stake anyway, why 
should we make it even easier? :-) Anyhow, if the user really, really 
wants to get himself immolated and export a TLS variable, he can always 
manually write the thunk (it is, after all, trivial).

 (Perhaps we could take this one step further - export doesn't apply 


to global variables, either, unless they are extern(C)?)

 I do like the idea that this proposal enables better code performance 


on Linux - better than C!
Nov 10 2013
next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 08.11.2013 20:32, schrieb Walter Bright:

 It looks pretty good, except I have serious reservations about the 


-lib switch proposed behavior:
I'm glad you like the proposal.


 1. It's too blunt. A user could conceivably want to export some 


symbols and not others. This is all or nothing.
A user is already able to control which symbols are exported and which 
are not by using the "export" attribute. Talking about expectations, 
when you come from C/C++ you usually expect that static libraries do 
_not_ contain any dll-exported symbols. At least I have never seen a 
single C or C++ static library that does contain dll-exported symbols. 
The -lib switch just deactivates dll-exporting symbols. I would also be 
fine with adding a additional compiler switch "--disable-export" or 
something like that. But I would strongly prefer a solution where I 
don't have to run additional tools on the generted object files to get 
rid of dll-exported symbols I didn't want in the first place, because I 
was compiling my code for a static library.


 2. It requires the compiler to rewrite the .obj files, even ones that 


dmd did not create. I would consider this very surprising behavior. The 
user will be "WTF the object files in the lib are different from on my 
disk! D sux!" -lib is a simple-minded creature that currently does 
exactly what one would expect it to.
I can agree with that. The consensus in the newsgroup dicussion was 
though, that this is better then adding another compiler switch for 
disabeling dll-exported symbols, because that compiler switch would need 
to be added manually, whereas the -lib solution would work "magically". 
So the -lib solution whas thought to have better usability.


 3. I've never heard anyone complain about the issue of a DLL 


referencing another DLL and the combined list of exports being the 
concatenation of both. It can be annoying, but it would be a minor 
issue. If it really is one, we can write a separate simple tool that 
strips exports from object files.

I don't think you understood the issue correctly. The issue is that you 
don't want to have dll-exported symbols from a static library inside a 
DLL. Static libraries are usually expected to not have any dll-exported 
smybols at all. For example when creating a DLL and you link against 
phobos you will always have all symbols from phobos & druntime inside 
your DLL, if we follow your suggested behaviour. This not only will lead 
to linker errors if you use two dlls that both link against phobos 
statically. But it will also make debugging and inspecting dll-exports 
harder as the dll will always have tons of exports you have to filter out.


 And:

 Exporting of TLS variables is a minor issue. Frankly, I think people 


who export global variables should be burned at the stake anyway, why 
should we make it even easier? Anyhow, if the user really, really wants 
to get himself immolated and export a TLS variable, he can always 
manually write the thunk (it is, after all, trivial).
I don't agree with that. It should be possible to create shared 
libraries which use all the capbailites of the language. And not 
restirct them to a arbitrary subset. In D TLS is a special issue because 
so many variables end up in TLS automatically. I'm especially thinking 
about inlining of small methods across dll boundaries. Also a shared 
library should really work the same way as a static one. Requiering a 
different design for a static versus a shared library seems inconsistent 
to me.

 (Perhaps we could take this one step further - export doesn't apply 


to global variables, either, unless they are extern(C)?)

 I do like the idea that this proposal enables better code performance 


on Linux - better than C!


I fully agree here, although this basically happened by accident. I was 
really astonished when I learned how much mostly useless indirections 
are added when compiling with -fPIC. Windows Dlls seem to have the 
better solution here regarding performance.

We should really continue this discussion on the newsgroup. If this is 
ok with you, just copy the e-mail conversation to a new newsgroup 
thread, or send me a message and I will do so.

Kind Regards
Benjamin Thaut



 Nov 10 2013



 next sibling parent  Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 09.11.2013 20:26, schrieb Martin Nowak:

 On 11/09/2013 08:11 PM, code benjamin-thaut.de wrote:

 2. It requires the compiler to rewrite the .obj files, even ones 
 that dmd did not create. I would consider this very surprising
 behavior. The user will be "WTF the object files in the lib are
 different from on my disk! D sux!" -lib is a simple-minded creature
 that currently does exactly what one would expect it to.


 I can agree with that. The consensus in the newsgroup dicussion was
 though, that this is better then adding another compiler switch for
 disabeling dll-exported symbols, because that compiler switch would
 need to be added manually, whereas the -lib solution would work
 "magically". So the -lib solution whas thought to have better
 usability.


 IIRC the main problem we're trying to solve is the following.
 If a static library contains object files with exported symbols those
 will
 be reexported from an exe/dll that links against the static library.
 Linking a static library into a dll is rare enough to justify a
 separate tool,
 but the exe case is problematic.





 Nov 10 2013



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/10/2013 12:42 PM, Benjamin Thaut wrote:

 Am 08.11.2013 20:32, schrieb Walter Bright:
  > It looks pretty good, except I have serious reservations about the
 -lib switch proposed behavior:
 I'm glad you like the proposal.

  >
  > 1. It's too blunt. A user could conceivably want to export some
 symbols and not others. This is all or nothing.
 A user is already able to control which symbols are exported and which
 are not by using the "export" attribute. ...


Using the export attribute on some member exports the entire module 
info. If I understand this right, the module info will contain 
information also about non-exported members of the module and therefore 
even non-exported members will be accessible from outside. Is this correct?



 Nov 10 2013



 next sibling parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 10.11.2013 13:03, schrieb Timon Gehr:

 On 11/10/2013 12:42 PM, Benjamin Thaut wrote:

 Am 08.11.2013 20:32, schrieb Walter Bright:
  > It looks pretty good, except I have serious reservations about the
 -lib switch proposed behavior:
 I'm glad you like the proposal.

  >
  > 1. It's too blunt. A user could conceivably want to export some
 symbols and not others. This is all or nothing.
 A user is already able to control which symbols are exported and which
 are not by using the "export" attribute. ...


 Using the export attribute on some member exports the entire module
 info. If I understand this right, the module info will contain
 information also about non-exported members of the module and therefore
 even non-exported members will be accessible from outside. Is this correct?


AFAIK the module info only contains information about the (shared) 
module constructors and destructors. And no, non exported symbols will 
not be acessible because the neccessary accessors are not generated for 
them (at least on windows).



 Nov 10 2013



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/10/2013 01:06 PM, Benjamin Thaut wrote:

 Am 10.11.2013 13:03, schrieb Timon Gehr:

 On 11/10/2013 12:42 PM, Benjamin Thaut wrote:

 Am 08.11.2013 20:32, schrieb Walter Bright:
  > It looks pretty good, except I have serious reservations about the
 -lib switch proposed behavior:
 I'm glad you like the proposal.

  >
  > 1. It's too blunt. A user could conceivably want to export some
 symbols and not others. This is all or nothing.
 A user is already able to control which symbols are exported and which
 are not by using the "export" attribute. ...


 Using the export attribute on some member exports the entire module
 info. If I understand this right, the module info will contain
 information also about non-exported members of the module and therefore
 even non-exported members will be accessible from outside. Is this
 correct?


 AFAIK the module info only contains information about the (shared)
 module constructors and destructors.  And no, non exported symbols will
 not be acessible because the neccessary accessors are not generated for
 them (at least on windows).


What about Object.factory?



 Nov 10 2013



 next sibling parent  Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 10.11.2013 13:39, schrieb Timon Gehr:

 On 11/10/2013 01:06 PM, Benjamin Thaut wrote:

 Am 10.11.2013 13:03, schrieb Timon Gehr:

 On 11/10/2013 12:42 PM, Benjamin Thaut wrote:

 Am 08.11.2013 20:32, schrieb Walter Bright:
  > It looks pretty good, except I have serious reservations about the
 -lib switch proposed behavior:
 I'm glad you like the proposal.

  >
  > 1. It's too blunt. A user could conceivably want to export some
 symbols and not others. This is all or nothing.
 A user is already able to control which symbols are exported and which
 are not by using the "export" attribute. ...


 Using the export attribute on some member exports the entire module
 info. If I understand this right, the module info will contain
 information also about non-exported members of the module and therefore
 even non-exported members will be accessible from outside. Is this
 correct?


 AFAIK the module info only contains information about the (shared)
 module constructors and destructors.  And no, non exported symbols will
 not be acessible because the neccessary accessors are not generated for
 them (at least on windows).


 What about Object.factory?


That would obviously work. But why would you want to restrict that for 
shared libraries? Its not restricted for static libraries either. And 
esentially a shared library should behave exactly the same as a static one.



 Nov 10 2013



prev sibling  parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Sunday, 10 November 2013 at 12:39:35 UTC, Timon Gehr wrote:

 What about Object.factory?


I always though of it as a misfeature.



 Nov 11 2013



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/11/2013 11:02 PM, deadalnix wrote:

 On Sunday, 10 November 2013 at 12:39:35 UTC, Timon Gehr wrote:

 What about Object.factory?


 I always though of it as a misfeature.


Me too.



 Nov 11 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/11/13 2:36 PM, Timon Gehr wrote:

 On 11/11/2013 11:02 PM, deadalnix wrote:

 On Sunday, 10 November 2013 at 12:39:35 UTC, Timon Gehr wrote:

 What about Object.factory?


 I always though of it as a misfeature.


 Me too.


... well time to substantiate.

Andrei



 Nov 11 2013



 next sibling parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Tuesday, 12 November 2013 at 05:46:19 UTC, Andrei Alexandrescu 
wrote:

 On 11/11/13 2:36 PM, Timon Gehr wrote:

 On 11/11/2013 11:02 PM, deadalnix wrote:

 On Sunday, 10 November 2013 at 12:39:35 UTC, Timon Gehr wrote:

 What about Object.factory?


 I always though of it as a misfeature.


 Me too.


 ... well time to substantiate.

 Andrei


OK, but first, it has to be noted that this is putting the burden 
of proof on the wrong side of the argument.

So I ran `grep -r * -e 'Object.factory'` on my d folder. I have 
various D project there. The only uses of Object.factory are the 
test suite of dmd, ldc and GDC.

I have bunch of D code in there, not simply mine. So first thing 
: it is unused.
Secondly, the only useful things you can do with that require to 
upcast (and break basic OOP principle, which is kind of ironic 
for a feature that produce objects). The design of the feature is 
unsound.

It is not that surprising as different possible uses case are 
better solved with compile time reflection and metaprograming.

Implementationwise, this feature require to pull into the 
application some unoptimizable bloat. It is also a tedious 
constraint for shared objects as we see in this thread.



 Nov 11 2013



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/11/13 10:59 PM, deadalnix wrote:

 On Tuesday, 12 November 2013 at 05:46:19 UTC, Andrei Alexandrescu wrote:

 On 11/11/13 2:36 PM, Timon Gehr wrote:

 On 11/11/2013 11:02 PM, deadalnix wrote:

 On Sunday, 10 November 2013 at 12:39:35 UTC, Timon Gehr wrote:

 What about Object.factory?


 I always though of it as a misfeature.


 Me too.


 ... well time to substantiate.

 Andrei


 OK, but first, it has to be noted that this is putting the burden of
 proof on the wrong side of the argument.


Well I'm not "burdening", just trying to figure how to do it better.


 So I ran `grep -r * -e 'Object.factory'` on my d folder. I have various
 D project there. The only uses of Object.factory are the test suite of
 dmd, ldc and GDC.


That really proves nothing.


 I have bunch of D code in there, not simply mine. So first thing : it is
 unused.
 Secondly, the only useful things you can do with that require to upcast
 (and break basic OOP principle, which is kind of ironic for a feature
 that produce objects). The design of the feature is unsound.


I think you're wrong here (even after one discounts for 
s/upcast/downcast/). There's no breakage of any basic OOP principle. 
There's no irony. Polymorphic creation aka the factory pattern is a 
classic in object-oriented design. You'd have a bad time arguing against 
it with anyone.


 It is not that surprising as different possible uses case are better
 solved with compile time reflection and metaprograming.


These don't take care of the dynamic case, which is sometimes needed 
(wherever factories are useful: serialization, scripting, ...).


 Implementationwise, this feature require to pull into the application
 some unoptimizable bloat.


This is a good point. But I presume the dedicated code is small in size.


 It is also a tedious constraint for shared objects as we see in this
 thread.


My reading of the thread is that there are concerns about DIP 45 beyond 
support of factories for shared objects.


Andrei



 Nov 11 2013



  parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Tuesday, 12 November 2013 at 07:16:03 UTC, Andrei Alexandrescu 
wrote:

 That really proves nothing.


That doesn't prove anything. But that doesn't smell good for the 
usefulness of the feature either.


 I have bunch of D code in there, not simply mine. So first 
 thing : it is
 unused.
 Secondly, the only useful things you can do with that require 
 to upcast
 (and break basic OOP principle, which is kind of ironic for a 
 feature
 that produce objects). The design of the feature is unsound.


 I think you're wrong here (even after one discounts for 
 s/upcast/downcast/). There's no breakage of any basic OOP 
 principle. There's no irony. Polymorphic creation aka the 
 factory pattern is a classic in object-oriented design. You'd 
 have a bad time arguing against it with anyone.


A Factory is always a factory of something. A factory fit nicely 
into the OOP paradigm as it allow the user of the object to not 
know what is the actual type of the object, but its interface.

A factory is also in charge of all the work necessary to put the 
object in a useful state. Object.factory do not allow any of this.

It doesn't make any sense to provide a factory of everything. The 
factory user will have to know what is built, at least partially 
to use it. This defeat the whole principle of a factory.


 It is not that surprising as different possible uses case are 
 better
 solved with compile time reflection and metaprograming.


 These don't take care of the dynamic case, which is sometimes 
 needed (wherever factories are useful: serialization, 
 scripting, ...).


It is actually quite easy to create runtime capabilities from 
these basic blocs. CTFE allow to create AA, and at this point, 
the user can get whatever is desired at runtime.

This feature has more to do with runtime reflection than OOP.


 Implementationwise, this feature require to pull into the 
 application
 some unoptimizable bloat.


 This is a good point. But I presume the dedicated code is small 
 in size.


The code in itself isn't really the problem. The compiler have to 
emit a bunch of wiring to make it work. Wiring which is almost 
never used. Basically, the compiler have to emit runtime 
reflection informations, but just for classes.

This seems to be the wrong tradeoff : we get the constraint of 
runtime reflection in term of design, but with very limited 
benefices.


 It is also a tedious constraint for shared objects as we see 
 in this
 thread.


 My reading of the thread is that there are concerns about DIP 
 45 beyond support of factories for shared objects.


This is A concern. Surely not the only one. That is why I didn't 
expanded myself on the subject in the first place. Simply that if 
we had to choose between this DIP and Object.factory, it is a 
braindead choice.



 Nov 11 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/11/13 11:58 PM, deadalnix wrote:

 On Tuesday, 12 November 2013 at 07:16:03 UTC, Andrei Alexandrescu wrote:

 That really proves nothing.


 That doesn't prove anything. But that doesn't smell good for the
 usefulness of the feature either.


My code doesn't use that, either. But it also doesn't use things like 
.toString for classes. It's just that I don't do much OOP in D - doesn't 
mean OOP and/or factory etc are bankrupt.


 I think you're wrong here (even after one discounts for
 s/upcast/downcast/). There's no breakage of any basic OOP principle.
 There's no irony. Polymorphic creation aka the factory pattern is a
 classic in object-oriented design. You'd have a bad time arguing
 against it with anyone.


 A Factory is always a factory of something. A factory fit nicely into
 the OOP paradigm as it allow the user of the object to not know what is
 the actual type of the object, but its interface.


Yes. Generalized, it stands to reason that Object (as the most inclusive 
class type) could plausibly have a factory.


 A factory is also in charge of all the work necessary to put the object
 in a useful state. Object.factory do not allow any of this.


But default class constructors can be elaborate, so one can assume the 
default constructor puts the object in a useful state. Granted, we could 
have added constructors with e.g. Variant parameters. But that would be 
just an enhancement.


 It doesn't make any sense to provide a factory of everything. The
 factory user will have to know what is built, at least partially to use
 it. This defeat the whole principle of a factory.


I don't see any defeating here. I do see a desire to specialize 
factories for specific subhierarchies, which is entirely fine. I don't 
see how that doesn't make a factory dealing in Object bad.


 It is not that surprising as different possible uses case are better
 solved with compile time reflection and metaprograming.


 These don't take care of the dynamic case, which is sometimes needed
 (wherever factories are useful: serialization, scripting, ...).


 It is actually quite easy to create runtime capabilities from these
 basic blocs. CTFE allow to create AA, and at this point, the user can
 get whatever is desired at runtime.


What's missing is a list of all class types in the system. This is a 
fundamental issue with factories.


 This feature has more to do with runtime reflection than OOP.


Implementation-wise, yes. Conceptually, people often see them as virtual 
constructors (i.e. an OOP notion extended to construction).


 Implementationwise, this feature require to pull into the application
 some unoptimizable bloat.


 This is a good point. But I presume the dedicated code is small in size.


 The code in itself isn't really the problem. The compiler have to emit a
 bunch of wiring to make it work. Wiring which is almost never used.
 Basically, the compiler have to emit runtime reflection informations,
 but just for classes.


One question would be whether the same work is needed for e.g. typeid() 
or toString. If not, maybe there is a case against factory.


 This seems to be the wrong tradeoff : we get the constraint of runtime
 reflection in term of design, but with very limited benefices.


I don't see how the existing system prevents you from rolling your own.


 It is also a tedious constraint for shared objects as we see in this
 thread.


 My reading of the thread is that there are concerns about DIP 45
 beyond support of factories for shared objects.


 This is A concern. Surely not the only one. That is why I didn't
 expanded myself on the subject in the first place. Simply that if we had
 to choose between this DIP and Object.factory, it is a braindead choice.


The time difference is crucial here. If this DIP came up in 2006 (when I 
estimate we introduced Object.factory) probably a lot of things would 
have come down differently. Proposals made in different epochs are, I 
think, very difficult to compare.


Andrei



 Nov 12 2013



 next sibling parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 12.11.2013 09:50, schrieb Andrei Alexandrescu:

 The time difference is crucial here. If this DIP came up in 2006 (when I
 estimate we introduced Object.factory) probably a lot of things would
 have come down differently. Proposals made in different epochs are, I
 think, very difficult to compare.


 Andrei


Wait. Whats the actual issue here? This DIP does not prevent 
object.factory from working across DLL boundaries. This DIP does also 
not conflict with object.factory. So what is the actual problem at hand?



 Nov 12 2013



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/12/13 1:30 AM, Benjamin Thaut wrote:

 Am 12.11.2013 09:50, schrieb Andrei Alexandrescu:

 The time difference is crucial here. If this DIP came up in 2006 (when I
 estimate we introduced Object.factory) probably a lot of things would
 have come down differently. Proposals made in different epochs are, I
 think, very difficult to compare.


 Andrei


 Wait. Whats the actual issue here? This DIP does not prevent
 object.factory from working across DLL boundaries. This DIP does also
 not conflict with object.factory. So what is the actual problem at hand?


You're right. I was meaning that in the abstract - generally feature 
creation is strongly conditioned by existing context.

Andrei



 Nov 12 2013



prev sibling  parent  "deadalnix" <deadalnix gmail.com>  writes:


On Tuesday, 12 November 2013 at 08:50:29 UTC, Andrei Alexandrescu 
wrote:

 One question would be whether the same work is needed for e.g. 
 typeid() or toString. If not, maybe there is a case against 
 factory.


That isn't the same. toString itself rely on typeid. And the 
typeid object is stored with the virtual function table. The code 
can retrieve that information for any object, even the ones that 
the code never heard about.

Object.factory require the inverse: finding the typeid from the 
name of the class. It require to build some structure that 
associate the name of a class and its typeid, and make it 
available for Object.factory.


 I don't see how the existing system prevents you from rolling 
 your own.


It doesn't, but I still need to pay for it.


 The time difference is crucial here. If this DIP came up in 
 2006 (when I estimate we introduced Object.factory) probably a 
 lot of things would have come down differently. Proposals made 
 in different epochs are, I think, very difficult to compare.


I understand some design has been made for historical reasons. 
That happen to every system. That doesn't mean much.



 Nov 12 2013



prev sibling next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 07:59, deadalnix wrote:


 OK, but first, it has to be noted that this is putting the burden of
 proof on the wrong side of the argument.

 So I ran `grep -r * -e 'Object.factory'` on my d folder. I have various
 D project there. The only uses of Object.factory are the test suite of
 dmd, ldc and GDC.

 I have bunch of D code in there, not simply mine. So first thing : it is
 unused.


I used it in my serialization library, Orange, and it will be used in 
std.serialization when I finish that project.

-- 
/Jacob Carlborg



 Nov 11 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 08:52, Jacob Carlborg wrote:


 I used it in my serialization library, Orange, and it will be used in
 std.serialization when I finish that project.


Well, technically I'm using ClassInfo.find.

-- 
/Jacob Carlborg



 Nov 12 2013



prev sibling  parent  Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 12.11.2013 07:59, schrieb deadalnix:

 On Tuesday, 12 November 2013 at 05:46:19 UTC, Andrei Alexandrescu wrote:

 ... well time to substantiate.

 Andrei


 OK, but first, it has to be noted that this is putting the burden of
 proof on the wrong side of the argument.


But what is the actual argument? What problem do you see regarding 
object.factory and this DIP?



 Nov 12 2013



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/12/2013 06:46 AM, Andrei Alexandrescu wrote:

 On 11/11/13 2:36 PM, Timon Gehr wrote:

 On 11/11/2013 11:02 PM, deadalnix wrote:

 On Sunday, 10 November 2013 at 12:39:35 UTC, Timon Gehr wrote:

 What about Object.factory?


 I always though of it as a misfeature.


 Me too.


 ... well time to substantiate.

 Andrei


- Every class in any imported module will need to show up in the 
generated code even if it is never used unless global static analysis is 
carried out on arguments to Object.factory.

- If I know the fully qualified name of a class, there are better ways 
to instantiate this class than passing a string containing that name to 
Object.factory in order to call the default constructor.

- The functionality provided by Object.factory is trivially replaced by 
a solution more specifically tailored to the problem at hand using 
compile-time reflection.



 Nov 12 2013



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/12/13 2:36 PM, Timon Gehr wrote:

 On 11/12/2013 06:46 AM, Andrei Alexandrescu wrote:

 On 11/11/13 2:36 PM, Timon Gehr wrote:

 On 11/11/2013 11:02 PM, deadalnix wrote:

 On Sunday, 10 November 2013 at 12:39:35 UTC, Timon Gehr wrote:

 What about Object.factory?


 I always though of it as a misfeature.


 Me too.


 ... well time to substantiate.

 Andrei


 - Every class in any imported module will need to show up in the
 generated code even if it is never used unless global static analysis is
 carried out on arguments to Object.factory.


Correct. On the other hand, a lot of unused classes in used modules 
seems to indicate a problem with the system's design.


 - If I know the fully qualified name of a class, there are better ways
 to instantiate this class than passing a string containing that name to
 Object.factory in order to call the default constructor.


What are the better ways? Note that most of the time you don't "know" 
the name of a class - you get it down the wire during some 
deserialization. So there must be some way to build an object from a 
token representation of the object.


 - The functionality provided by Object.factory is trivially replaced by
 a solution more specifically tailored to the problem at hand using
 compile-time reflection.


It's not quite trivial - somewhere there has to be a map and 
registration and lookup and whatnot. I don't see it why it's unbecoming 
for such functionality to be part of the standard library. I would 
agree, however, that it's a judgment call whether it should be wired in 
or provided on demand.


Andrei



 Nov 12 2013



 next sibling parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Tuesday, 12 November 2013 at 22:45:15 UTC, Andrei Alexandrescu 
wrote:

 What are the better ways? Note that most of the time you don't 
 "know" the name of a class - you get it down the wire during 
 some deserialization. So there must be some way to build an 
 object from a token representation of the object.


The serialization is a good example. You'll have to note that if 
the code has been able to serialize the data, it can generate at 
compile time the necessary scafolding to deserialize it.


 - The functionality provided by Object.factory is trivially 
 replaced by
 a solution more specifically tailored to the problem at hand 
 using
 compile-time reflection.


 It's not quite trivial - somewhere there has to be a map and 
 registration and lookup and whatnot. I don't see it why it's 
 unbecoming for such functionality to be part of the standard 
 library. I would agree, however, that it's a judgment call 
 whether it should be wired in or provided on demand.


That s why this is a problem to add all that is the core of the 
language when it isn't used often.



 Nov 12 2013



 next sibling parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/12/13 4:36 PM, deadalnix wrote:

 On Tuesday, 12 November 2013 at 22:45:15 UTC, Andrei Alexandrescu wrote:

 What are the better ways? Note that most of the time you don't "know"
 the name of a class - you get it down the wire during some
 deserialization. So there must be some way to build an object from a
 token representation of the object.


 The serialization is a good example. You'll have to note that if the
 code has been able to serialize the data, it can generate at compile
 time the necessary scafolding to deserialize it.


I think there is some confusion here. We're talking about polymorphic 
creation, i.e. an object comes down the wire and you don't know its type 
a priori. Again: grab MC++D and read the chapter on Factory, it 
clarifies all such stuff.


 It's not quite trivial - somewhere there has to be a map and
 registration and lookup and whatnot. I don't see it why it's
 unbecoming for such functionality to be part of the standard library.
 I would agree, however, that it's a judgment call whether it should be
 wired in or provided on demand.


 That s why this is a problem to add all that is the core of the language
 when it isn't used often.


That's a good point. I would tend to agree with that. But it doesn't 
make it a misfeature. It's a sensible feature.


Andrei



 Nov 12 2013



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 01:36, deadalnix wrote:


 The serialization is a good example. You'll have to note that if the
 code has been able to serialize the data, it can generate at compile
 time the necessary scafolding to deserialize it.


No, not necessarily. If you serialize an object through a base class 
reference you currently need to register that with the serializer. The 
static type information is lost and D doesn't provide enough runtime 
reflection to get the values of instance variables of an object at runtime.

-- 
/Jacob Carlborg



 Nov 13 2013



  parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Wednesday, 13 November 2013 at 08:48:09 UTC, Jacob Carlborg 
wrote:

 On 2013-11-13 01:36, deadalnix wrote:


 The serialization is a good example. You'll have to note that 
 if the
 code has been able to serialize the data, it can generate at 
 compile
 time the necessary scafolding to deserialize it.


 No, not necessarily. If you serialize an object through a base 
 class reference you currently need to register that with the 
 serializer. The static type information is lost and D doesn't 
 provide enough runtime reflection to get the values of instance 
 variables of an object at runtime.


OK, I see your problem here.

It is a much more general problem than the one solved by the 
factory. For instance, I wanted in the past to write some 
unittest that ensure Liskov's substitution principle (so run the 
unittest on subclasses).

This is fundamentally the same issue : you want to write some 
code that is aware of subclasses. That require somehow to be able 
to mixin something automatically in all subclasses.

Object.factory only solve poorly one instance of this problem.



 Nov 13 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 23:25, deadalnix wrote:


 OK, I see your problem here.

 It is a much more general problem than the one solved by the factory.
 For instance, I wanted in the past to write some unittest that ensure
 Liskov's substitution principle (so run the unittest on subclasses).

 This is fundamentally the same issue : you want to write some code that
 is aware of subclasses. That require somehow to be able to mixin
 something automatically in all subclasses.


Might be solveable with AST macros :). Although you can still have 
problems with subclasses you don't know of during compile time. Doing 
separate compilation, libraries and so on.


 Object.factory only solve poorly one instance of this problem.


-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/12/2013 11:45 PM, Andrei Alexandrescu wrote:

 - Every class in any imported module will need to show up in the
 generated code even if it is never used unless global static analysis is
 carried out on arguments to Object.factory.


 Correct. On the other hand, a lot of unused classes


It is not necessary that there are a lot. Anything transitively 
reachable from there will be pulled in.


 in used modules
 seems to indicate a problem with the system's design.


Those could be classes that are only used at compile time, like a 
builder for a grammar for a CTFE parser generator, for lack of a better 
example. Furthermore, library modules may be larger than their actively 
used interface. (Phobos?) It would be interesting to see how much 
Object.factory influences the size of hello world, even though Phobos 
does not use too many classes


 - If I know the fully qualified name of a class, there are better ways
 to instantiate this class than passing a string containing that name to
 Object.factory in order to call the default constructor.


 What are the better ways?


Call the default constructor. Call a delegate/virtual function that 
calls the default constructor.


 Note that most of the time you don't "know"
 the name of a class - you get it down the wire during some
 deserialization.


You don't want to call the default constructor during deserialization 
and you need actual information about the Object layout, so seems to be 
an instance of the next point.


 So there must be some way to build an object from a
 token representation of the object.


 - The functionality provided by Object.factory is trivially replaced by
 a solution more specifically tailored to the problem at hand using
 compile-time reflection.


 It's not quite trivial - somewhere there has to be a map and
 registration and lookup and whatnot. I don't see it why it's unbecoming
 for such functionality to be part of the standard library. I would
 agree, however, that it's a judgment call whether it should be wired in
 or provided on demand.


I think that if supported at all, it should be an UDA declared somewhere 
in druntime.



 Nov 12 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/12/13 6:03 PM, Timon Gehr wrote:

 On 11/12/2013 11:45 PM, Andrei Alexandrescu wrote:

 - Every class in any imported module will need to show up in the
 generated code even if it is never used unless global static analysis is
 carried out on arguments to Object.factory.


 Correct. On the other hand, a lot of unused classes


 It is not necessary that there are a lot. Anything transitively
 reachable from there will be pulled in.


 in used modules
 seems to indicate a problem with the system's design.


 Those could be classes that are only used at compile time, like a
 builder for a grammar for a CTFE parser generator, for lack of a better
 example. Furthermore, library modules may be larger than their actively
 used interface. (Phobos?) It would be interesting to see how much
 Object.factory influences the size of hello world, even though Phobos
 does not use too many classes


Fair point.


 - If I know the fully qualified name of a class, there are better ways
 to instantiate this class than passing a string containing that name to
 Object.factory in order to call the default constructor.


 What are the better ways?


 Call the default constructor. Call a delegate/virtual function that
 calls the default constructor.


Wait, doesn't Object.factory call the default constructor of the created 
object?


 Note that most of the time you don't "know"
 the name of a class - you get it down the wire during some
 deserialization.


 You don't want to call the default constructor during deserialization
 and you need actual information about the Object layout, so seems to be
 an instance of the next point.


If not the default constructor, which constructor? Didn't you say above 
that calling the default constructor is good? I feel we're talking past 
each other.


 So there must be some way to build an object from a
 token representation of the object.


 - The functionality provided by Object.factory is trivially replaced by
 a solution more specifically tailored to the problem at hand using
 compile-time reflection.


 It's not quite trivial - somewhere there has to be a map and
 registration and lookup and whatnot. I don't see it why it's unbecoming
 for such functionality to be part of the standard library. I would
 agree, however, that it's a judgment call whether it should be wired in
 or provided on demand.


 I think that if supported at all, it should be an UDA declared somewhere
 in druntime.


That would be indeed nice. But I have the feeling a significant part of 
this dialog is marred by confusion.


Andrei



 Nov 12 2013



 next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/13/2013 03:15 AM, Andrei Alexandrescu wrote:

 ...

 - If I know the fully qualified name of a class, there are better ways
 to instantiate this class than passing a string containing that name to
 Object.factory in order to call the default constructor.


 What are the better ways?


 Call the default constructor. Call a delegate/virtual function that
 calls the default constructor.


 Wait, doesn't Object.factory call the default constructor of the created
 object?
 ...


Yes, I was listing some ways to do this that I think are better than 
calling Object.factory.


 Note that most of the time you don't "know"
 the name of a class - you get it down the wire during some
 deserialization.


 You don't want to call the default constructor during deserialization
 and you need actual information about the Object layout, so seems to be
 an instance of the next point.


 If not the default constructor, which constructor?


None, I guess you'd want to create the right object state in memory.


 Didn't you say above
 that calling the default constructor is good?


If that is what you want to do, yes.


 I feel we're talking past
 each other.


Maybe.



 Nov 12 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/12/13 6:26 PM, Timon Gehr wrote:

 On 11/13/2013 03:15 AM, Andrei Alexandrescu wrote:

 ...

 - If I know the fully qualified name of a class, there are better ways
 to instantiate this class than passing a string containing that
 name to
 Object.factory in order to call the default constructor.


 What are the better ways?


 Call the default constructor. Call a delegate/virtual function that
 calls the default constructor.


 Wait, doesn't Object.factory call the default constructor of the created
 object?
 ...


 Yes, I was listing some ways to do this that I think are better than
 calling Object.factory.


OK I think I figure you are confused. You can't call the default 
constructor of an object you don't know the type of.

Andrei



 Nov 12 2013



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/13/2013 03:48 AM, Andrei Alexandrescu wrote:

 On 11/12/13 6:26 PM, Timon Gehr wrote:

 On 11/13/2013 03:15 AM, Andrei Alexandrescu wrote:

 ...

 - If I know the fully qualified name of a class, there are better
 ways
 to instantiate this class than passing a string containing that
 name to
 Object.factory in order to call the default constructor.


 What are the better ways?


 Call the default constructor. Call a delegate/virtual function that
 calls the default constructor.


 Wait, doesn't Object.factory call the default constructor of the created
 object?
 ...


 Yes, I was listing some ways to do this that I think are better than
 calling Object.factory.


 OK I think I figure you are confused.


I'm not.


 You can't call the default
 constructor of an object you don't know the type of.

 Andrei


Indeed. There was a premise stating that I do. :o)

In any case, the only use for string -> new object is when there is some 
kind of parsing going on (and then you should make sure in some way that 
only objects of types in some well-defined subset can be constructed), 
otherwise you can always pass around richer data structures, such as an 
actual factory object.



 Nov 12 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/12/13 7:05 PM, Timon Gehr wrote:

 On 11/13/2013 03:48 AM, Andrei Alexandrescu wrote:

 On 11/12/13 6:26 PM, Timon Gehr wrote:

 On 11/13/2013 03:15 AM, Andrei Alexandrescu wrote:

 ...

 - If I know the fully qualified name of a class, there are better
 ways
 to instantiate this class than passing a string containing that
 name to
 Object.factory in order to call the default constructor.


 What are the better ways?


 Call the default constructor. Call a delegate/virtual function that
 calls the default constructor.


 Wait, doesn't Object.factory call the default constructor of the
 created
 object?
 ...


 Yes, I was listing some ways to do this that I think are better than
 calling Object.factory.


 OK I think I figure you are confused.


 I'm not.


Then how do you figure doing this:

class Streamable { ... }
class Foo : Streamable { ... }
class Bar : Streamable { ... }
string className = stream.readln();
Streamable obj = ...;

How do you create obj from className, when className could be either 
"Foo" or "Bar"? In the general case there could be any number of 
classes, in different modules.


 You can't call the default
 constructor of an object you don't know the type of.

 Andrei


 Indeed. There was a premise stating that I do. :o)


So you say Object.factory is a misfeature based on a use case that 
solves a very different problem?


 In any case, the only use for string -> new object is when there is some
 kind of parsing going on (and then you should make sure in some way that
 only objects of types in some well-defined subset can be constructed),
 otherwise you can always pass around richer data structures, such as an
 actual factory object.


No parsing needs to be involved, as in e.g. binary formats. And I'm 
talking data not code, which excludes passing around a factory object.


Andrei



 Nov 12 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 05:07, Andrei Alexandrescu wrote:


 Then how do you figure doing this:

 class Streamable { ... }
 class Foo : Streamable { ... }
 class Bar : Streamable { ... }
 string className = stream.readln();
 Streamable obj = ...;

 How do you create obj from className, when className could be either
 "Foo" or "Bar"? In the general case there could be any number of
 classes, in different modules.


This requires Object.factory (or equivalent) and that all subclasses 
have been registered as well. Since you don't know which subclass 
"className" represents you're forced to deserialize by doing this:

Streamable obj = deserialize!(Streamable)(className);

-- 
/Jacob Carlborg



 Nov 13 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/13/13 12:55 AM, Jacob Carlborg wrote:

 On 2013-11-13 05:07, Andrei Alexandrescu wrote:


 Then how do you figure doing this:

 class Streamable { ... }
 class Foo : Streamable { ... }
 class Bar : Streamable { ... }
 string className = stream.readln();
 Streamable obj = ...;

 How do you create obj from className, when className could be either
 "Foo" or "Bar"? In the general case there could be any number of
 classes, in different modules.


 This requires Object.factory (or equivalent) and that all subclasses
 have been registered as well.


With Object.factory that's taken care of already.


 Since you don't know which subclass
 "className" represents you're forced to deserialize by doing this:

 Streamable obj = deserialize!(Streamable)(className);


Again there is a confusion here. The idea was to create an object with 
the dynamic type Foo if the class name was "Foo" and an object with the 
dynamic type Bar if the class name was "Bar". You are glossing over the 
gist of it all with a function call. Are you sure we are talking about 
the same thing?


Andrei



 Nov 13 2013



 next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 10:27, Andrei Alexandrescu wrote:


 With Object.factory that's taken care of already.


No, you need to register the subclasses as well. The static type 
information is lost for the subclasses.


 Again there is a confusion here. The idea was to create an object with
 the dynamic type Foo if the class name was "Foo" and an object with the
 dynamic type Bar if the class name was "Bar". You are glossing over the
 gist of it all with a function call. Are you sure we are talking about
 the same thing?


Here is an excerpt of my serialization library, with the relevant code 
for this discussion:

void function (in Object) [ClassInfo] registeredTypes;

void register (T : Object) ()
{
     registeredTypes[T.classinfo] = &downcastSerialize!(T)
}

void downcastDeserialize (U : Object) (in Object value)
{
     alias Unqual!(U) T;
     auto casted = cast(T) value;

     assert(casted);
     assert(casted.classinfo is T.classinfo);

     deserializeHelper(casted);
}

void deserializeHelper (T) (T value) { ... }

bool isBaseClass (T) (T value)
{
     return value.classinfo !is T.classinfo;
}

T deserialize (T) (string name)
{
     T t;

     if (auto classInfo = ClassInfo.find(name))
     {
          auto value = cast(T) _d_alloc(classInfo);

         if (isBaseClass(value))
         {
             if (auto deserializer = value.classinfo in registeredTypes)
                 (*serializer)(value);

             else
                 throw new Exception("Unregistered subclass");
         }

         else
             deserializeHelper(value);

         return value;
     }

     else
     {
         throw new Exception("The class " ~ name ~ " couldn't be found");
         return null;
     }
}

For the full code see: 
https://github.com/jacob-carlborg/orange/blob/master/orange/serialization/Serializer.d

-- 
/Jacob Carlborg



 Nov 13 2013



 next sibling parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/13/13 2:46 AM, Jacob Carlborg wrote:

 On 2013-11-13 10:27, Andrei Alexandrescu wrote:


 With Object.factory that's taken care of already.


 No, you need to register the subclasses as well. The static type
 information is lost for the subclasses.


We are definitely talking about different things.


      if (auto classInfo = ClassInfo.find(name))


You didn't have to register for find to work, which was my point.


Andrei



 Nov 13 2013



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/13/13 2:46 AM, Jacob Carlborg wrote:

 Here is an excerpt of my serialization library, with the relevant code
 for this discussion:


[snip]

To drive the point home: I looked once more over the code and I think 
it's too complicated for what it does. It could be improved as follows.

The key to simplification is using

Object create() const;

from ClassInfo (aka Typeinfo_Class) in addition to

static const(TypeInfo_Class) find(in char[] classname);

in the same type. The procedure would go as follows:

1. Retrieve the ClassInfo object for the class name.

2. Create a default-constructed object by calling create() against the 
object obtained at (1).

3. Cast that Object down to IDeserializable, which is an interface type 
that has methods such as deserialize(Archive).

4. If the cast failed, throw an exception - the object was not designed 
to be deserialized.

5. Otherwise, call deserialize against that object, passing it the archive.

Voila! No need for runtime registration - all user code has to do is 
implement the appropriate interface.

I hope this both clarifies my point about factory being a sensible 
feature, and helps you improve your library.


Andrei



 Nov 13 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 17:29, Andrei Alexandrescu wrote:


 To drive the point home: I looked once more over the code and I think
 it's too complicated for what it does. It could be improved as follows.


No, I have worked on this library for several years. I don't see how it 
could be less complicated for what it does. It seems you don't 
understand exactly what it does.


 The key to simplification is using

 Object create() const;


Requires a default constructor. Not good enough.


 static const(TypeInfo_Class) find(in char[] classname);

 in the same type. The procedure would go as follows:

 1. Retrieve the ClassInfo object for the class name.

 2. Create a default-constructed object by calling create() against the
 object obtained at (1).


As above, requires a default constructor. Not good enough.


 3. Cast that Object down to IDeserializable, which is an interface type
 that has methods such as deserialize(Archive).


* Requires a class to implement an interface. Not good enough.
* Requires the class to manually serialize itself. Not good enough.

A serialization library need to work with third party types you don't 
have any control over.


 4. If the cast failed, throw an exception - the object was not designed
 to be deserialized.

 5. Otherwise, call deserialize against that object, passing it the archive.


As above, requires the class to manually serialize itself. Not good enough.


 Voila! No need for runtime registration - all user code has to do is
 implement the appropriate interface.

 I hope this both clarifies my point about factory being a sensible
 feature, and helps you improve your library.


Unfortunately it doesn't help at all. You clearly missed a lot of 
features this library has.

To be able to automatically (de)serialize an object in D you need to 
know its static type. That requires registering the type. Note, this is 
only needed if you're serializing through a base class reference.

You'll have to do a lot better than this to convince me.

Note also that I'm not against Object.factory/create, I'm arguing for 
it. Serialization is one example of why it's needed.

The biggest reasons for why it's looking like it does are:

* Can (de)serialize arbitrary types, even third party types you don't 
have any control over

* It automatically (de)serializes the objects. No need to implement a 
method. You can do, if you want to customize the (de)serialization

-- 
/Jacob Carlborg



 Nov 13 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/13/13 11:20 AM, Jacob Carlborg wrote:

 A serialization library need to work with third party types you don't
 have any control over.


OK that convinced me. My post remains as an explanation of my point.

Andrei



 Nov 13 2013



  parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 13.11.2013 21:36, schrieb Andrei Alexandrescu:

 On 11/13/13 11:20 AM, Jacob Carlborg wrote:

 A serialization library need to work with third party types you don't
 have any control over.


 OK that convinced me. My post remains as an explanation of my point.

 Andrei


Would you mind moving this discussion to another thread? Everytime I see 
a update I think its something relevant and usually its just about 
object.factory.

Kind Regards
Benjamin Thaut



 Nov 13 2013



  parent  Martin Nowak <code dawg.eu>  writes:


On 11/13/2013 09:45 PM, Benjamin Thaut wrote:

 Would you mind moving this discussion to another thread? Everytime I see
 a update I think its something relevant and usually its just about
 object.factory.


Yes please, OT discussions ruin every focus.



 Nov 16 2013



prev sibling  parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


13-Nov-2013 13:27, Andrei Alexandrescu пишет:

 On 11/13/13 12:55 AM, Jacob Carlborg wrote:

 On 2013-11-13 05:07, Andrei Alexandrescu wrote:


 Then how do you figure doing this:

 class Streamable { ... }
 class Foo : Streamable { ... }
 class Bar : Streamable { ... }
 string className = stream.readln();
 Streamable obj = ...;

 How do you create obj from className, when className could be either
 "Foo" or "Bar"? In the general case there could be any number of
 classes, in different modules.


 This requires Object.factory (or equivalent) and that all subclasses
 have been registered as well.


 With Object.factory that's taken care of already.


I have to chime in.

Serialization != calling default constructor by name.
You have to work out fields somehow - Object.factory doesn't help with 
that. What serialization needs is some hook - be it delegete or whatnot:

Object deserialize(SomeStreamingAbstraction here);

Simply snatching a name is not serialization. Not to mention that far 
too many protocols are not text.

Keeping built-in map of class-name --> "create via default constructor"
is an awful example of ONE SIZE FITS ALL. It doesn't!

Simply on the ground of "solves problem in too narrow scope for a hefty 
coast to the user" I'd drop it.


-- 
Dmitry Olshansky



 Nov 13 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/13/13 10:41 AM, Dmitry Olshansky wrote:

 13-Nov-2013 13:27, Andrei Alexandrescu пишет:

 On 11/13/13 12:55 AM, Jacob Carlborg wrote:

 On 2013-11-13 05:07, Andrei Alexandrescu wrote:


 Then how do you figure doing this:

 class Streamable { ... }
 class Foo : Streamable { ... }
 class Bar : Streamable { ... }
 string className = stream.readln();
 Streamable obj = ...;

 How do you create obj from className, when className could be either
 "Foo" or "Bar"? In the general case there could be any number of
 classes, in different modules.


 This requires Object.factory (or equivalent) and that all subclasses
 have been registered as well.


 With Object.factory that's taken care of already.


 I have to chime in.

 Serialization != calling default constructor by name.


Of course not. It does give you access to an object with the correct 
dynamic type, and interfaces and virtual calls take care of the rest.


 You have to work out fields somehow - Object.factory doesn't help with
 that. What serialization needs is some hook - be it delegete or whatnot:

 Object deserialize(SomeStreamingAbstraction here);


Or a method.


 Simply snatching a name is not serialization. Not to mention that far
 too many protocols are not text.


Method.


 Keeping built-in map of class-name --> "create via default constructor"
 is an awful example of ONE SIZE FITS ALL. It doesn't!


It is: it achieves virtual construction, the only missing piece in a 
classic OOP scheme. Once you have the object, the rest is smooth sailing.


 Simply on the ground of "solves problem in too narrow scope for a hefty
 coast to the user" I'd drop it.


It's good. Let's keep it.



Andrei



 Nov 13 2013



  parent  Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


13-Nov-2013 22:46, Andrei Alexandrescu пишет:

 On 11/13/13 10:41 AM, Dmitry Olshansky wrote:

 13-Nov-2013 13:27, Andrei Alexandrescu пишет:

 On 11/13/13 12:55 AM, Jacob Carlborg wrote:

 On 2013-11-13 05:07, Andrei Alexandrescu wrote:


 Then how do you figure doing this:

 class Streamable { ... }
 class Foo : Streamable { ... }
 class Bar : Streamable { ... }
 string className = stream.readln();
 Streamable obj = ...;

 How do you create obj from className, when className could be either
 "Foo" or "Bar"? In the general case there could be any number of
 classes, in different modules.


 This requires Object.factory (or equivalent) and that all subclasses
 have been registered as well.


 With Object.factory that's taken care of already.


 I have to chime in.

 Serialization != calling default constructor by name.


 Of course not. It does give you access to an object with the correct
 dynamic type, and interfaces and virtual calls take care of the rest.


Allow me to retort, see below.
BTW the keywords here were:
_default_ and _by name_


 Simply on the ground of "solves problem in too narrow scope for a hefty
 coast to the user" I'd drop it.


 It's good. Let's keep it.


Oh, my... We've lost him :o)

Okay. Let me iterate on some facts:

1. Serialization may or may not be intrusive. There are many cases out 
there to externalize serialization (see e.g. boost serialization).
2. Want speed - avoid virtuals. Requiring that serialization be always 
done via specific virtual call harms performance.
3. Something you seem to dodge - AA lookup by fully qualified name is 
not fast nor convenient for many serialization backends.
4. Not everything needs to be serialized yet it's registered (and slows 
down lookups).
5. Forcing a pattern - every class object must be default constructible. 
Btw what factory does with these not defining this() ? Sadly the reality 
is the other way around - many interesting objects don't have valid 
default states.

Another point that is not a fact but rather a speculation.
I _think_ that the idea of having user pay beforehand for something 
simply because it may one day need it is going against D ethos. Unlike 
Java or .NET we have definite trend of catering for specific needs and 
extreme flexibility at no extra cost.

-- 
Dmitry Olshansky



 Nov 13 2013



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 03:15, Andrei Alexandrescu wrote:


 Wait, doesn't Object.factory call the default constructor of the created
 object?


Yes, but you use ClassInfo.find and _d_newclass, which is exactly the 
same thing, except it won't call the constructor. Object.factory uses 
these functions internally.

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 23:45, Andrei Alexandrescu wrote:


 It's not quite trivial - somewhere there has to be a map and
 registration and lookup and whatnot. I don't see it why it's unbecoming
 for such functionality to be part of the standard library. I would
 agree, however, that it's a judgment call whether it should be wired in
 or provided on demand.


 From a (de)serialization point of view, I like that it's not necessary 
to register every class that should be (de)serialized.

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling  parent reply "Jakob Ovrum" <jakobovrum gmail.com>  writes:


On Tuesday, 12 November 2013 at 22:36:22 UTC, Timon Gehr wrote:

 - Every class in any imported module will need to show up in 
 the generated code even if it is never used unless global 
 static analysis is carried out on arguments to Object.factory.

 - If I know the fully qualified name of a class, there are 
 better ways to instantiate this class than passing a string 
 containing that name to Object.factory in order to call the 
 default constructor.

 - The functionality provided by Object.factory is trivially 
 replaced by a solution more specifically tailored to the 
 problem at hand using compile-time reflection.


+ 1.

To test, I made a Factory template that works like this:

---
unittest
{
	import std.traits : fullyQualifiedName;

	interface I {}
	class A : I {}
	class B : I {}

	alias f = Factory!(I, A, B);

	static assert(is(f.Product == I));

	I a = f.create(fullyQualifiedName!A);
	assert(cast(A)a);

	I b = f.create(fullyQualifiedName!B);
	assert(cast(B)b);
}
---

Some notes:

  * The first parameter, the Product parameter, which can be any 
class or interface, is required because CommonType is always 
Object when given a heterogeneous list of classes. I assume this 
is because of the difficulty of choosing a common interface when 
multiple interfaces are present.

  * All subsequent parameters, the factory provider classes, are 
of course checked that they actually implement the Product, 
whether as a base class or an interface. Is a bit of a gotcha 
because AliasThis has to be "worked around" in the checking code.

  * Implementation uses a sorted array instead of an AA because 
AAs can't be transferred from CT to RT yet.

  * The array is immutable, so the same list is used for the same 
Factory across threads (but I suppose that has no real 
significance, just an observation).

  * If the Product member is deemed unnecessary, it could just be 
a function template instead of a template with two members 
(Product and create).

Advantages over Object.factory:

  * The opCall returns Product, not Object, so client code doesn't 
need to cast in most circumstances.

  * `fullyQualifiedName` is used in the example because it 
internally uses `TypeInfo_Class.create`, which requires the full, 
internal names of the A and B types, which of course, are nested 
in a unittest. However, since the range of implementations are 
known, it could be amended to statically figure out a less strict 
naming scheme. In this case, just "A" and "B" could be made valid 
without ambiguity. Object.factory cannot practically do this.

  * Interestingly, since it knows all the constructors, it could 
present an overload set of the constructors that all implementors 
support, with some implementation effort!

  * Lookup has the opportunity to be a lot faster than 
Object.factory due to the smaller set of providers.

  * The runtime list of providers does not contain types that 
aren't part of the set (duh), hence no bloat.

The disadvantage is that all implementing types must be passed at 
compile-time. That is the only advantage of Object.factory, 
AFAICS. As it seems like a very niche need that everyone has to 
pay for, I don't think Object.factory carries its weight.



 Nov 12 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/12/13 3:05 PM, Jakob Ovrum wrote:

 On Tuesday, 12 November 2013 at 22:36:22 UTC, Timon Gehr wrote:

 - Every class in any imported module will need to show up in the
 generated code even if it is never used unless global static analysis
 is carried out on arguments to Object.factory.

 - If I know the fully qualified name of a class, there are better ways
 to instantiate this class than passing a string containing that name
 to Object.factory in order to call the default constructor.

 - The functionality provided by Object.factory is trivially replaced
 by a solution more specifically tailored to the problem at hand using
 compile-time reflection.


 + 1.

 To test, I made a Factory template that works like this:

 ---
 unittest
 {
      import std.traits : fullyQualifiedName;

      interface I {}
      class A : I {}
      class B : I {}

      alias f = Factory!(I, A, B);


Stopped reading here, and everybody should. There is a thorough 
treatment of object factories in "Modern C++ Design" which I recommend 
not because I wrote it, but because it's thorough. Part of it is to show 
why the approach above is marred by dependency issues and should often 
be avoided.


Andrei



 Nov 12 2013



  parent reply "Jakob Ovrum" <jakobovrum gmail.com>  writes:


On Tuesday, 12 November 2013 at 23:47:10 UTC, Andrei Alexandrescu 
wrote:

 On 11/12/13 3:05 PM, Jakob Ovrum wrote:

 On Tuesday, 12 November 2013 at 22:36:22 UTC, Timon Gehr wrote:

 - Every class in any imported module will need to show up in 
 the
 generated code even if it is never used unless global static 
 analysis
 is carried out on arguments to Object.factory.

 - If I know the fully qualified name of a class, there are 
 better ways
 to instantiate this class than passing a string containing 
 that name
 to Object.factory in order to call the default constructor.

 - The functionality provided by Object.factory is trivially 
 replaced
 by a solution more specifically tailored to the problem at 
 hand using
 compile-time reflection.


 + 1.

 To test, I made a Factory template that works like this:

 ---
 unittest
 {
     import std.traits : fullyQualifiedName;

     interface I {}
     class A : I {}
     class B : I {}

     alias f = Factory!(I, A, B);


 Stopped reading here, and everybody should. There is a thorough 
 treatment of object factories in "Modern C++ Design" which I 
 recommend not because I wrote it, but because it's thorough. 
 Part of it is to show why the approach above is marred by 
 dependency issues and should often be avoided.


Sure, it has dependency issues, it needs to know all the types 
then and there, with is the anti-thesis of many factories. So, 
call it something else, like TypeMap, and it's still a useful 
type.

But alright, it is not a good factory.



 Nov 12 2013



  parent reply "Dicebot" <public dicebot.lv>  writes:


On Wednesday, 13 November 2013 at 00:03:49 UTC, Jakob Ovrum wrote:

 But alright, it is not a good factory.


Well, D compile-time reflection allows to define factory that 
finds all classes that implement given interface and provides 
them for construction, with no explicit list.

I have used this in D1 though because of limited compile-time 
reflection capabilities there.



 Nov 12 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/12/13 4:18 PM, Dicebot wrote:

 On Wednesday, 13 November 2013 at 00:03:49 UTC, Jakob Ovrum wrote:

 But alright, it is not a good factory.


 Well, D compile-time reflection allows to define factory that finds all
 classes that implement given interface and provides them for
 construction, with no explicit list.


I don't think all classes in a system are discoverable during 
compilation. This is the part of Object.factory that is not easy to 
replace. There must be a global map somewhere and a per-module 
registration during initialization.

Andrei



 Nov 12 2013



  parent  "Dicebot" <public dicebot.lv>  writes:


On Wednesday, 13 November 2013 at 00:25:36 UTC, Andrei 
Alexandrescu wrote:

 I don't think all classes in a system are discoverable during 
 compilation. This is the part of Object.factory that is not 
 easy to replace. There must be a global map somewhere and a 
 per-module registration during initialization.

 Andrei


Sure, this topic was already discussed in __traits(getUnittest) 
context recently, similar problem. You can find all classes that 
transitively accessible from context via imports - this is a 
natural limitation of compile-time reflection. Sometimes it is 
enough, sometimes registration is necessary. But it is a nice 
tool to have.



 Nov 12 2013



prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 10.11.2013 13:03, schrieb Timon Gehr:

 Using the export attribute on some member exports the entire module
 info. If I understand this right, the module info will contain
 information also about non-exported members of the module and therefore
 even non-exported members will be accessible from outside. Is this correct?


Addition: It is necessary to export the module info. Because as soon as 
you import a module and use any part of it, the compiler will reference 
the module info.



 Nov 10 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/10/2013 4:13 AM, Benjamin Thaut wrote:

 Addition: It is necessary to export the module info. Because as soon as you
 import a module and use any part of it, the compiler will reference the module
 info.


Actually, it isn't entirely clear to me why the moduleinfo needs to be exported.

The moduleinfo is needed to run the static ctors/dtors, etc., but that is
needed 
by the code internal to the dll that initializes the dll.



 Nov 10 2013



 next sibling parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 10.11.2013 20:33, schrieb Walter Bright:

 On 11/10/2013 4:13 AM, Benjamin Thaut wrote:

 Addition: It is necessary to export the module info. Because as soon
 as you
 import a module and use any part of it, the compiler will reference
 the module
 info.


 Actually, it isn't entirely clear to me why the moduleinfo needs to be
 exported.

 The moduleinfo is needed to run the static ctors/dtors, etc., but that
 is needed by the code internal to the dll that initializes the dll.


Well if the dll initializes all the modules, it has to initialize _all_ 
the modules because it can't know which are used and which are not. This 
is most certenly the way to go when loading a D dll with the LoadLibrary 
windows api function(s).

But when you link against a D dll using a genereated import library 
there would be the option to make it behave exactly like static 
libraries where only modules get initiliazed, which are actually used.

Also lets assume you do not export module infos, how does the compiler 
know that the module is inside a dll so that the module info is not 
referenced? That goes against the basic design of this DIP. The entire 
idea was to make static / shared liraries more similar and require less 
effort from the user. And not the other way around. If this is not 
wanted we can go right back to the C/C++ way of defining both dllexport 
and dllimport.

Kind Regards
Benjamin Thaut



 Nov 10 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/10/2013 1:25 PM, Benjamin Thaut wrote:

 Am 10.11.2013 20:33, schrieb Walter Bright:

 On 11/10/2013 4:13 AM, Benjamin Thaut wrote:

 Addition: It is necessary to export the module info. Because as soon
 as you
 import a module and use any part of it, the compiler will reference
 the module
 info.


 Actually, it isn't entirely clear to me why the moduleinfo needs to be
 exported.

 The moduleinfo is needed to run the static ctors/dtors, etc., but that
 is needed by the code internal to the dll that initializes the dll.


 Well if the dll initializes all the modules, it has to initialize _all_ the
 modules because it can't know which are used and which are not. This is most
 certenly the way to go when loading a D dll with the LoadLibrary windows api
 function(s).


Yes, and that's also the way shared library support has been implemented in 
druntime.



 But when you link against a D dll using a genereated import library there would
 be the option to make it behave exactly like static libraries where only
modules
 get initiliazed, which are actually used.


I don't believe it is necessary to support this. I don't think the C or C++ 
standard shared runtime library does. The general idea with DLLs is they become 
an indivisible unit - if you want to load only part of a DLL, you should split 
it into multiple DLLs.



 Also lets assume you do not export module infos, how does the compiler know
that
 the module is inside a dll so that the module info is not referenced? That goes
 against the basic design of this DIP. The entire idea was to make static /
 shared liraries more similar and require less effort from the user. And not the
 other way around. If this is not wanted we can go right back to the C/C++ way
of
 defining both dllexport and dllimport.


The trouble with dllexport and dllimport is the same module is both depending
on 
how it is used - I don't think that is very viable (C/C++ deal with this using 
macros). The idea is to use export for both, and then use the context to figure 
out if it is dllimport or dllexport.



 Nov 10 2013



  parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 11.11.2013 05:04, schrieb Walter Bright:

 The trouble with dllexport and dllimport is the same module is both
 depending on how it is used - I don't think that is very viable (C/C++
 deal with this using macros). The idea is to use export for both, and
 then use the context to figure out if it is dllimport or dllexport.


Still, how do you want to stop the compiler from referencing the module 
infos without knowing the module is part of a dll?



 Nov 10 2013



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/10/2013 10:02 PM, Benjamin Thaut wrote:

 Am 11.11.2013 05:04, schrieb Walter Bright:

 The trouble with dllexport and dllimport is the same module is both
 depending on how it is used - I don't think that is very viable (C/C++
 deal with this using macros). The idea is to use export for both, and
 then use the context to figure out if it is dllimport or dllexport.


 Still, how do you want to stop the compiler from referencing the module infos
 without knowing the module is part of a dll?


The compiler needs to know which modules are coming from a dll.

The only reference is in the import dependency list, which is used for static 
constructor calling. A dll's initialization should be self-contained and fully 
performed upon loading of the dll. The compiler should skip adding it to the 
import dependencies.



 Nov 11 2013



prev sibling  parent reply "Martin Nowak" <code dawg.eu>  writes:


On Sunday, 10 November 2013 at 19:34:32 UTC, Walter Bright wrote:

 On 11/10/2013 4:13 AM, Benjamin Thaut wrote:
 Actually, it isn't entirely clear to me why the moduleinfo 
 needs to be exported.


That depends on whether using a module might require to link 
against the moduleinfo, currently it does. For example there is 
an importedModules property in ModuleInfo.


 The moduleinfo is needed to run the static ctors/dtors, etc., 
 but that is needed by the code internal to the dll that 
 initializes the dll.


Initialization order is not an issue here, the whole DLL (and all 
modules) is initialized before any DLL/exe that depends on it.



 Nov 12 2013



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/12/2013 1:46 PM, Martin Nowak wrote:

 On Sunday, 10 November 2013 at 19:34:32 UTC, Walter Bright wrote:

 On 11/10/2013 4:13 AM, Benjamin Thaut wrote:
 Actually, it isn't entirely clear to me why the moduleinfo needs to be
exported.


 That depends on whether using a module might require to link against the
 moduleinfo, currently it does. For example there is an importedModules property
 in ModuleInfo.


That's only used for initialization order.



 The moduleinfo is needed to run the static ctors/dtors, etc., but that is
 needed by the code internal to the dll that initializes the dll.


 Initialization order is not an issue here, the whole DLL (and all modules) is
 initialized before any DLL/exe that depends on it.


Right, so it shouldn't need to list it as an imported module.



 Nov 14 2013



prev sibling  parent  "Martin Nowak" <code dawg.eu>  writes:


On Sunday, 10 November 2013 at 12:03:53 UTC, Timon Gehr wrote:

 Using the export attribute on some member exports the entire 
 module info. If I understand this right, the module info will 
 contain information also about non-exported members of the 
 module and therefore even non-exported members will be 
 accessible from outside. Is this correct?


Yes, the module infos provide basic runtime reflection which can 
be used to access non-exported members.



 Nov 12 2013



prev sibling  parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 08.11.2013 20:32, schrieb Walter Bright:

 And:

 Exporting of TLS variables is a minor issue. Frankly, I think people 


who export global variables should be burned at the stake anyway, why 
should we make it even easier? Anyhow, if the user really, really wants 
to get himself immolated and export a TLS variable, he can always 
manually write the thunk (it is, after all, trivial).

In my opinion it should be possible to add "export:" to the top of every 
source file and then compiler a dynamic library which exatcly behaves 
like a static library. If you don't want TLS variables in interfaces 
this should also be forbidden for interfaces of static libraries, 
because this isn't really any difference. This really should be 
consistent. Either TLS variabels are not allowed at all within library 
interfaces or they are allowed and supported always.

Kind Regards
Benjamin Thaut



 Nov 10 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/10/2013 6:52 AM, Benjamin Thaut wrote:

 Am 08.11.2013 20:32, schrieb Walter Bright:
  >
  > And:
  >
  > Exporting of TLS variables is a minor issue. Frankly, I think people who
 export global variables should be burned at the stake anyway, why should we
make
 it even easier? Anyhow, if the user really, really wants to get himself
 immolated and export a TLS variable, he can always manually write the thunk (it
 is, after all, trivial).

 In my opinion it should be possible to add "export:" to the top of every source
 file and then compiler a dynamic library which exatcly behaves like a static
 library. If you don't want TLS variables in interfaces this should also be
 forbidden for interfaces of static libraries, because this isn't really any
 difference. This really should be consistent. Either TLS variabels are not
 allowed at all within library interfaces or they are allowed and supported
always.


I understand the consistency argument, but I'll also argue that if one raises 
consistency as an overriding requirement, everything else suffers. Everything
is 
a tradeoff.

It's long been recognized that using global variables to communicate between 
interfaces is a bad idea. And it isn't even supported for dlls, because the 
proposed solution to making them work is to wrap them with a function (much
like 
D properties).

We'd be going out of our way to support a recognized bad paradigm. There is 
currently no existing D code that requires this. If we add it, then we'd be 
stuck with supporting it for backwards compatibility. If we don't add it, and
it 
becomes some sort of crisis that it isn't supported, we can add it in later 
without breaking things.



 Nov 10 2013



 next sibling parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 10.11.2013 20:46, schrieb Walter Bright:

 On 11/10/2013 6:52 AM, Benjamin Thaut wrote:

 I understand the consistency argument, but I'll also argue that if one
 raises consistency as an overriding requirement, everything else
 suffers. Everything is a tradeoff.

 It's long been recognized that using global variables to communicate
 between interfaces is a bad idea. And it isn't even supported for dlls,
 because the proposed solution to making them work is to wrap them with a
 function (much like D properties).

 We'd be going out of our way to support a recognized bad paradigm. There
 is currently no existing D code that requires this. If we add it, then
 we'd be stuck with supporting it for backwards compatibility. If we
 don't add it, and it becomes some sort of crisis that it isn't
 supported, we can add it in later without breaking things.


You completely ignored my inlining argument. Lets assume there is some 
small function that qualifies for inlining but accesses a TLS variable 
marked with the export attribute. The compiler now wants to cross-module 
inline it. How does it do that, if there is no way to access TLS 
variables across DLL boundaries? The obvious solution would be, not do 
it at all. That means to disqualify all functions that do TLS access for 
cross-module inlining, because the compiler can't know if they are 
linked in from a static or shared library.

With D beeing a language that aims for performance, do we really want to 
do that?

Kind Regards
Benjamin Thaut



 Nov 10 2013



 next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/10/2013 1:50 PM, Benjamin Thaut wrote:

 You completely ignored my inlining argument. Lets assume there is some small
 function that qualifies for inlining but accesses a TLS variable marked with
the
 export attribute. The compiler now wants to cross-module inline it. How does it
 do that, if there is no way to access TLS variables across DLL boundaries? The
 obvious solution would be, not do it at all.


Right.


 That means to disqualify all
 functions that do TLS access for cross-module inlining, because the compiler
 can't know if they are linked in from a static or shared library.


The compiler would benefit from knowing which modules are from a shared
library, 
and it needs to anyway in order to distinguish between dllimport and dllexport.



 With D beeing a language that aims for performance, do we really want to do
that?


The trouble is, the proposed solution for exporting TLS variables is to wrap
the 
access in a function that is not inline-able anyway. For performance oriented 
code, the user would be better off to write his own wrapper, and then hoist the 
call out of the hot spot, and use a pointer to the TLS variable in the hot spot.

As a general rule, if code has a "hot spot" that crosses a DLL boundary, there 
is going to be performance problems with it regardless. I don't think this is a 
problem that the language can solve without some effort from the user.



 Nov 10 2013



 next sibling parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 11.11.2013 05:17, schrieb Walter Bright:

 On 11/10/2013 1:50 PM, Benjamin Thaut wrote:

 The compiler would benefit from knowing which modules are from a shared
 library, and it needs to anyway in order to distinguish between
 dllimport and dllexport.


Then you didn't understand the DIP correctly. One of the main points of 
this DIP is to provide a implementation of the export attribute that 
does _not_ require the compiler to know anything about, wether certain 
modules will end up in a DLL or not. With the proposed behaviour you 
could compile a module into an object file and then link it both into a 
DLL and static lirary. Both would just work. When using a library the 
compiler also doesn't have to know if the lirary is shared or static. 
Both will work and it will be decided at link time which path is coosen 
(static or shared). Generally this behaviour makes the export attribute 
easier to use and less error prone.

In an early version of this DIP did involve giving the compiler the 
information which modules end up in an shared library, but it was 
rejected in the discussion. A solution which does not require this 
information was preferred by almost all users taking part in the discussion.


 The trouble is, the proposed solution for exporting TLS variables is to
 wrap the access in a function that is not inline-able anyway. For
 performance oriented code, the user would be better off to write his own
 wrapper, and then hoist the call out of the hot spot, and use a pointer
 to the TLS variable in the hot spot.

 As a general rule, if code has a "hot spot" that crosses a DLL boundary,
 there is going to be performance problems with it regardless. I don't
 think this is a problem that the language can solve without some effort
 from the user.


Correct. But the compiler would still be able do other optimizations 
given the function is sligtly bigger then a simple getter.



 Nov 10 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/10/2013 10:19 PM, Benjamin Thaut wrote:

 Am 11.11.2013 05:17, schrieb Walter Bright:

 On 11/10/2013 1:50 PM, Benjamin Thaut wrote:

 The compiler would benefit from knowing which modules are from a shared
 library, and it needs to anyway in order to distinguish between
 dllimport and dllexport.


 Then you didn't understand the DIP correctly. One of the main points of this
DIP
 is to provide a implementation of the export attribute that does _not_ require
 the compiler to know anything about, wether certain modules will end up in a
DLL
 or not. With the proposed behaviour you could compile a module into an object
 file and then link it both into a DLL and static lirary. Both would just work.
 When using a library the compiler also doesn't have to know if the lirary is
 shared or static. Both will work and it will be decided at link time which path
 is coosen (static or shared). Generally this behaviour makes the export
 attribute easier to use and less error prone.


I do understand that. I just have strong doubts about whether that is the best 
approach or not.



 In an early version of this DIP did involve giving the compiler the information
 which modules end up in an shared library, but it was rejected in the
 discussion. A solution which does not require this information was preferred by
 almost all users taking part in the discussion.


 The trouble is, the proposed solution for exporting TLS variables is to
 wrap the access in a function that is not inline-able anyway. For
 performance oriented code, the user would be better off to write his own
 wrapper, and then hoist the call out of the hot spot, and use a pointer
 to the TLS variable in the hot spot.

 As a general rule, if code has a "hot spot" that crosses a DLL boundary,
 there is going to be performance problems with it regardless. I don't
 think this is a problem that the language can solve without some effort
 from the user.


 Correct. But the compiler would still be able do other optimizations given the
 function is sligtly bigger then a simple getter.


That's possible, but I suspect much of the potential gains would be lost.



 Nov 11 2013



 next sibling parent  Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 11.11.2013 19:21, schrieb Walter Bright:

 On 11/10/2013 10:19 PM, Benjamin Thaut wrote:

 I do understand that. I just have strong doubts about whether that is
 the best approach or not.


So we throw 90% of the DIP away and you do a counter proposal?



 Nov 11 2013



prev sibling  parent  "Martin Nowak" <code dawg.eu>  writes:


On Monday, 11 November 2013 at 18:21:21 UTC, Walter Bright wrote:

 I do understand that. I just have strong doubts about whether 
 that is the best approach or not.


We made a slight adjustment on how exported data symbols should 
be accessed, always through a pointer indirection. Calling 
exported function already works transparently and accessing 
exported TLS variables isn't the most urgent issue.
Given that exported data variables should be rare we're trading a 
tiny implementation change for a no worries export attribute.



 Nov 12 2013



prev sibling  parent reply "Martin Nowak" <code dawg.eu>  writes:


On Monday, 11 November 2013 at 04:18:09 UTC, Walter Bright wrote:

 The compiler would benefit from knowing which modules are from 
 a shared library, and it needs to anyway in order to 
 distinguish between dllimport and dllexport.


We didn't found any reasonably simple solution to pass this 
information to the compiler. You'd have to explicitly list ALL 
modules that are supposed to be linked into the same DLL when 
compiling an object file.



 Nov 12 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/12/2013 2:23 PM, Martin Nowak wrote:

 On Monday, 11 November 2013 at 04:18:09 UTC, Walter Bright wrote:

 The compiler would benefit from knowing which modules are from a shared
 library, and it needs to anyway in order to distinguish between dllimport and
 dllexport.


 We didn't found any reasonably simple solution to pass this information to the
 compiler. You'd have to explicitly list ALL modules that are supposed to be
 linked into the same DLL when compiling an object file.


One possibility is modules listed on the command line are regarded as 
export==dllexport, and other modules as export==dllimport.

This of course means that functions may wind up going through the dllimport 
indirection even for calling functions in the same dll, but it should work.



 Nov 14 2013



 next sibling parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 14.11.2013 11:28, schrieb Walter Bright:

 On 11/12/2013 2:23 PM, Martin Nowak wrote:

 One possibility is modules listed on the command line are regarded as
 export==dllexport, and other modules as export==dllimport.

 This of course means that functions may wind up going through the
 dllimport indirection even for calling functions in the same dll, but it
 should work.


That doesns't work for the case where a dll "A" uses a dll "B".
In that case export has to mean "dllexport" for all modules of A but 
"dllimport" for all modules of B.

-- 
Kind Regards
Benjamin Thaut



 Nov 14 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/14/2013 3:37 AM, Benjamin Thaut wrote:

 Am 14.11.2013 11:28, schrieb Walter Bright:

 On 11/12/2013 2:23 PM, Martin Nowak wrote:

 One possibility is modules listed on the command line are regarded as
 export==dllexport, and other modules as export==dllimport.

 This of course means that functions may wind up going through the
 dllimport indirection even for calling functions in the same dll, but it
 should work.


 That doesns't work for the case where a dll "A" uses a dll "B".
 In that case export has to mean "dllexport" for all modules of A but
"dllimport"
 for all modules of B.


I don't follow. If you're compiling A, you're specifying A modules on the 
command line, and those will regard the B modules as dllimport.



 Nov 14 2013



  parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 15/11/2013 08:27, schrieb Walter Bright:

 On 11/14/2013 3:37 AM, Benjamin Thaut wrote:

 Am 14.11.2013 11:28, schrieb Walter Bright:

 On 11/12/2013 2:23 PM, Martin Nowak wrote:

 One possibility is modules listed on the command line are regarded as
 export==dllexport, and other modules as export==dllimport.

 This of course means that functions may wind up going through the
 dllimport indirection even for calling functions in the same dll, but it
 should work.


 That doesns't work for the case where a dll "A" uses a dll "B".
 In that case export has to mean "dllexport" for all modules of A but
 "dllimport"
 for all modules of B.


 I don't follow. If you're compiling A, you're specifying A modules on
 the command line, and those will regard the B modules as dllimport.


Ok now I understand what you suggest. So basically you want to do the 
exact same as DIP 45 just giving the compiler parameter a different name.

But you still didn't give a solution for the problem that the compiler 
does not know which modules are part of a shared library, which are part 
of a static library and which are part of the exeuctable. And please 
also consider single file compilation.

Kind Regards
Benjamin Thaut



 Nov 15 2013



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/15/2013 7:31 AM, Benjamin Thaut wrote:

 Am 15/11/2013 08:27, schrieb Walter Bright:

 On 11/14/2013 3:37 AM, Benjamin Thaut wrote:

 Am 14.11.2013 11:28, schrieb Walter Bright:

 On 11/12/2013 2:23 PM, Martin Nowak wrote:

 One possibility is modules listed on the command line are regarded as
 export==dllexport, and other modules as export==dllimport.

 This of course means that functions may wind up going through the
 dllimport indirection even for calling functions in the same dll, but it
 should work.


 That doesns't work for the case where a dll "A" uses a dll "B".
 In that case export has to mean "dllexport" for all modules of A but
 "dllimport"
 for all modules of B.


 I don't follow. If you're compiling A, you're specifying A modules on
 the command line, and those will regard the B modules as dllimport.


 Ok now I understand what you suggest. So basically you want to do the exact
same
 as DIP 45 just giving the compiler parameter a different name.

 But you still didn't give a solution for the problem that the compiler does not
 know which modules are part of a shared library, which are part of a static
 library and which are part of the exeuctable.


I thought this did cover it. "export" in imported modules that are not listed
on 
the compiler command line is regarded as "dllimport". "export" in modules
listed 
on the commmand line is regarded as "dllexport".


 And please also consider single file compilation.


And yes, this would mean that a function may wind up calling a function in its 
own dll through the dll import thunk -- however -- this thunk is often provided 
by the import library, and so wouldn't be there for intra-dll calls.



 Nov 17 2013



prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 14.11.2013 11:28, schrieb Walter Bright:

 This of course means that functions may wind up going through the
 dllimport indirection even for calling functions in the same dll, but it
 should work.


Also our suggested approach would not have this downside.

Think about phobos. All of phobos and druntime will be littered with 
export attributes. Do you really want all phobos and druntime from 
within the same dll to go through dll indirections? And what would 
happen when building phobos as static library? With your suggested 
behavior (export=dllimport if not specified otherwise) this would mean 
that even the phobos static library would have those indirections.

-- 
Kind Regards
Benjamin Thaut



 Nov 14 2013



 next sibling parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Thursday, 14 November 2013 at 13:31:30 UTC, Benjamin Thaut 
wrote:

 Am 14.11.2013 11:28, schrieb Walter Bright:

 This of course means that functions may wind up going through 
 the
 dllimport indirection even for calling functions in the same 
 dll, but it
 should work.


 Also our suggested approach would not have this downside.

 Think about phobos. All of phobos and druntime will be littered 
 with export attributes. Do you really want all phobos and 
 druntime from within the same dll to go through dll 
 indirections? And what would happen when building phobos as 
 static library? With your suggested behavior (export=dllimport 
 if not specified otherwise) this would mean that even the 
 phobos static library would have those indirections.


This thread have reached some end.

It has been shown in the beginning that linkers know how to get 
rid of the indirection here. If we are going to discuss the same 
thing again and again without even doing some experiment for the 
reality check, then we can stop right here.



 Nov 14 2013



  parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 14.11.2013 20:54, schrieb deadalnix:

 On Thursday, 14 November 2013 at 13:31:30 UTC, Benjamin Thaut wrote:

 Am 14.11.2013 11:28, schrieb Walter Bright:

 This of course means that functions may wind up going through the
 dllimport indirection even for calling functions in the same dll, but it
 should work.


 Also our suggested approach would not have this downside.

 Think about phobos. All of phobos and druntime will be littered with
 export attributes. Do you really want all phobos and druntime from
 within the same dll to go through dll indirections? And what would
 happen when building phobos as static library? With your suggested
 behavior (export=dllimport if not specified otherwise) this would mean
 that even the phobos static library would have those indirections.


 This thread have reached some end.

 It has been shown in the beginning that linkers know how to get rid of
 the indirection here. If we are going to discuss the same thing again
 and again without even doing some experiment for the reality check, then
 we can stop right here.


No, linkers can not get of this indirection. Only whole program 
optimization can, and that won't happen in D (at least dmd) for quite 
some time. We had exactly that discussion in the old thread about the 
export attribute. See here: 
http://forum.dlang.org/thread/kvhu2c$2ikq$1 digitalmars.com#post-kvhu2c:242ikq:241:40digitalmars.com

But because Walter does not agree with the way we solved the problem in 
DIP 45 we have to go through everything again from the beginning.
And I'm certenly not going to accept a solution where we rely on some 
non implemented feature to remove indirections which will most likely 
not happen in the near future. The same already happend for virtual 
functions. The documentation states that the compiler might optimize 
them away, but in pratice this still does not happen, and most likely 
never will.

With DIP 45 we setteled for additional indirections to achive better 
usability and I'm ok with that. But what Walter suggests esentially 
means even more indirections and worse usability. And I'm not ok with that.

I also don't like that we have to go through everything again we already 
discussed in the old thread, but if Walter doesn't agree it has to be 
that way. So please try to contribute in a constructive manner, or leave 
out comments entierly.

Kind Regards
Benjamin Thaut



 Nov 14 2013



  parent  "deadalnix" <deadalnix gmail.com>  writes:


OK, I missed the point where everything started back again. This 
is being ridiculous.



 Nov 14 2013



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/14/2013 5:31 AM, Benjamin Thaut wrote:

 Am 14.11.2013 11:28, schrieb Walter Bright:

 This of course means that functions may wind up going through the
 dllimport indirection even for calling functions in the same dll, but it
 should work.


 Also our suggested approach would not have this downside.

 Think about phobos. All of phobos and druntime will be littered with export
 attributes. Do you really want all phobos and druntime from within the same dll
 to go through dll indirections? And what would happen when building phobos as
 static library? With your suggested behavior (export=dllimport if not specified
 otherwise) this would mean that even the phobos static library would have those
 indirections.


It's not that bad. Phobos can be built by specifying all the files on the 
command line.

Also, at least on Windows, you can call functions in a DLL without saying 
dllimport on them and suffering a layer of indirection. The magic happens in
the 
import library, which provides the relevant thunk. It's about 15 years since I 
worked on this stuff, so I might be a bit fuzzy on the details.



 Nov 14 2013



 next sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com>  writes:


"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:l64imh$27na$1 digitalmars.com...

 Also, at least on Windows, you can call functions in a DLL without saying 
 dllimport on them and suffering a layer of indirection. The magic happens 
 in the import library, which provides the relevant thunk. It's about 15 
 years since I worked on this stuff, so I might be a bit fuzzy on the 
 details.


The symbol in the import library just translates to an import table 
indirection.

The trick you may have been thinking of is system dlls (kernel32.dll, 
user32.dll etc) are always loaded at the same address, so a little hackery 
will let you bypass the import table. 



 Nov 15 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/15/2013 12:00 AM, Daniel Murphy wrote:

 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:l64imh$27na$1 digitalmars.com...

 Also, at least on Windows, you can call functions in a DLL without saying
 dllimport on them and suffering a layer of indirection. The magic happens
 in the import library, which provides the relevant thunk. It's about 15
 years since I worked on this stuff, so I might be a bit fuzzy on the
 details.


 The symbol in the import library just translates to an import table
 indirection.


Yes, meaning the compiler doesn't have to do it if the import library is set up 
correctly. (implib.exe should do it.)



 Nov 15 2013



  parent reply "Daniel Murphy" <yebblies nospamgmail.com>  writes:


"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:l64lji$2buh$1 digitalmars.com...

 On 11/15/2013 12:00 AM, Daniel Murphy wrote:

 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:l64imh$27na$1 digitalmars.com...

 Also, at least on Windows, you can call functions in a DLL without 
 saying
 dllimport on them and suffering a layer of indirection. The magic 
 happens
 in the import library, which provides the relevant thunk. It's about 15
 years since I worked on this stuff, so I might be a bit fuzzy on the
 details.


 The symbol in the import library just translates to an import table
 indirection.


 Yes, meaning the compiler doesn't have to do it if the import library is 
 set up correctly. (implib.exe should do it.)


Right, I was saying the indirection still exists. 



 Nov 15 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/15/2013 8:08 AM, Daniel Murphy wrote:

 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:l64lji$2buh$1 digitalmars.com...

 On 11/15/2013 12:00 AM, Daniel Murphy wrote:

 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:l64imh$27na$1 digitalmars.com...

 Also, at least on Windows, you can call functions in a DLL without
 saying
 dllimport on them and suffering a layer of indirection. The magic
 happens
 in the import library, which provides the relevant thunk. It's about 15
 years since I worked on this stuff, so I might be a bit fuzzy on the
 details.


 The symbol in the import library just translates to an import table
 indirection.


 Yes, meaning the compiler doesn't have to do it if the import library is
 set up correctly. (implib.exe should do it.)


 Right, I was saying the indirection still exists.


No, not really, at least not on Windows. Try calling a function in the Windows 
API and disassemble it. You'll see a direct call.



 Nov 17 2013



  parent reply "Daniel Murphy" <yebblies nospamgmail.com>  writes:


"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:l6c7gt$27fc$1 digitalmars.com...

 Right, I was saying the indirection still exists.


 No, not really, at least not on Windows. Try calling a function in the 
 Windows API and disassemble it. You'll see a direct call.


You sure about that?

src:

import core.sys.windows.windows;

void main()
{
    auto x = GetModuleHandleA(null);
}

obj:

__Dmain PROC NEAR
;  COMDEF __Dmain
        push    0                                       ; 0000 _ 6A, 00
        call    dword ptr [__imp__GetModuleHandleA 4]   ; 0002 _ FF. 15, 
00000000(segrel)
        xor     eax, eax                                  ; 0008 _ 31. C0
        ret                                             ; 000A _ C3
__Dmain ENDP


exe:

?_0072  LABEL NEAR
        push    0                                       ; 00402010 _ 6A, 00
        call    dword ptr [?_0067]                      ; 00402012 _ FF. 15, 
00401794(d)
        xor     eax, eax                                ; 00402018 _ 31. C0
        ret                                             ; 0040201A _ C3



 Nov 17 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/17/2013 9:25 PM, Daniel Murphy wrote:

 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:l6c7gt$27fc$1 digitalmars.com...

 Right, I was saying the indirection still exists.


 No, not really, at least not on Windows. Try calling a function in the
 Windows API and disassemble it. You'll see a direct call.


 You sure about that?

 src:

 import core.sys.windows.windows;

 void main()
 {
      auto x = GetModuleHandleA(null);
 }

 obj:

 __Dmain PROC NEAR
 ;  COMDEF __Dmain
          push    0                                       ; 0000 _ 6A, 00
          call    dword ptr [__imp__GetModuleHandleA 4]   ; 0002 _ FF. 15,
 00000000(segrel)
          xor     eax, eax                                  ; 0008 _ 31. C0
          ret                                             ; 000A _ C3
 __Dmain ENDP


 exe:

 ?_0072  LABEL NEAR
          push    0                                       ; 00402010 _ 6A, 00
          call    dword ptr [?_0067]                      ; 00402012 _ FF. 15,
 00401794(d)
          xor     eax, eax                                ; 00402018 _ 31. C0
          ret                                             ; 0040201A _ C3


Try this:

extern (Windows) int GetModuleHandleA(char*);

void main()
{
     auto x = GetModuleHandleA(null);
}

and it compiles and links and runs. No indirection (in the object file, the 
indirection is supplied by linker, triggered by an impdef record in 
kernel32.lib). The "export" isn't actually needed.



 Nov 18 2013



  parent  "Daniel Murphy" <yebblies nospamgmail.com>  writes:


"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:l6cm9d$2uq8$1 digitalmars.com...

 On 11/17/2013 9:25 PM, Daniel Murphy wrote:

 "Walter Bright" <newshound2 digitalmars.com> wrote in message
 news:l6c7gt$27fc$1 digitalmars.com...

 Right, I was saying the indirection still exists.


 No, not really, at least not on Windows. Try calling a function in the
 Windows API and disassemble it. You'll see a direct call.


 You sure about that?

 src:

 import core.sys.windows.windows;

 void main()
 {
      auto x = GetModuleHandleA(null);
 }

 obj:

 __Dmain PROC NEAR
 ;  COMDEF __Dmain
          push    0                                       ; 0000 _ 6A, 00
          call    dword ptr [__imp__GetModuleHandleA 4]   ; 0002 _ FF. 15,
 00000000(segrel)
          xor     eax, eax                                  ; 0008 _ 31. 
 C0
          ret                                             ; 000A _ C3
 __Dmain ENDP


 exe:

 ?_0072  LABEL NEAR
          push    0                                       ; 00402010 _ 6A, 
 00
          call    dword ptr [?_0067]                      ; 00402012 _ FF. 
 15,
 00401794(d)
          xor     eax, eax                                ; 00402018 _ 31. 
 C0
          ret                                             ; 0040201A _ C3


 Try this:

 extern (Windows) int GetModuleHandleA(char*);

 void main()
 {
     auto x = GetModuleHandleA(null);
 }

 and it compiles and links and runs. No indirection (in the object file, 
 the indirection is supplied by linker, triggered by an impdef record in 
 kernel32.lib). The "export" isn't actually needed.


So now we get:

?_0072  LABEL NEAR
        push    0                                       ; 00402010 _ 6A, 00
        call    Unnamed_2_1A0DC                         ; 00402012 _ E8, 
0001A0C5
        xor     eax, eax                                ; 00402017 _ 31. C0
        ret                                             ; 00402019 _ C3

Where Unnamed_2_1A0DC is a thunk in the import table:

Unnamed_2_1A0DC LABEL NEAR
        jmp     dword ptr [?_0067]                      ; 0041C0DC _ FF. 25, 
00401794(d)

That still appears to be a layer of indirection to me...

So the difference is the compiler is unaware the function may be called 
through the import table?  Why would it ever _want_ to use the first 
version, are there any cases that can't simply be rewritten as the second? 



 Nov 18 2013



prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 15/11/2013 08:32, schrieb Walter Bright:

 It's not that bad. Phobos can be built by specifying all the files on
 the command line.

 Also, at least on Windows, you can call functions in a DLL without
 saying dllimport on them and suffering a layer of indirection. The magic
 happens in the import library, which provides the relevant thunk. It's
 about 15 years since I worked on this stuff, so I might be a bit fuzzy
 on the details.


I know that. And we are using that fact in DIP 45. For data symbols we 
did suggest a similar behaviour that has to be implemented by us first. 
And yes we need data symbols, because of all the implicit data symbols 
the D programming language generates. TypeInfos, vtables etc.

I suggest that you read all the links I gave for further reading in DIP 
45 and DIP 45 again, because you are pretty close to what we suggested 
in DIP 45 without actually realizing it.

Kind Regards
Benjamin Thaut



 Nov 15 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/15/2013 7:34 AM, Benjamin Thaut wrote:

 Am 15/11/2013 08:32, schrieb Walter Bright:

 It's not that bad. Phobos can be built by specifying all the files on
 the command line.

 Also, at least on Windows, you can call functions in a DLL without
 saying dllimport on them and suffering a layer of indirection. The magic
 happens in the import library, which provides the relevant thunk. It's
 about 15 years since I worked on this stuff, so I might be a bit fuzzy
 on the details.


 I know that. And we are using that fact in DIP 45. For data symbols we did
 suggest a similar behaviour that has to be implemented by us first. And yes we
 need data symbols, because of all the implicit data symbols the D programming
 language generates. TypeInfos, vtables etc.

 I suggest that you read all the links I gave for further reading in DIP 45 and
 DIP 45 again, because you are pretty close to what we suggested in DIP 45
 without actually realizing it.


I'm very much against the suggested rewriting of obj files to strip the export 
records :-)



 Nov 17 2013



 next sibling parent  Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 18.11.2013 06:12, schrieb Walter Bright:

 On 11/15/2013 7:34 AM, Benjamin Thaut wrote:

 Am 15/11/2013 08:32, schrieb Walter Bright:

 It's not that bad. Phobos can be built by specifying all the files on
 the command line.

 Also, at least on Windows, you can call functions in a DLL without
 saying dllimport on them and suffering a layer of indirection. The magic
 happens in the import library, which provides the relevant thunk. It's
 about 15 years since I worked on this stuff, so I might be a bit fuzzy
 on the details.


 I know that. And we are using that fact in DIP 45. For data symbols we
 did
 suggest a similar behaviour that has to be implemented by us first.
 And yes we
 need data symbols, because of all the implicit data symbols the D
 programming
 language generates. TypeInfos, vtables etc.

 I suggest that you read all the links I gave for further reading in
 DIP 45 and
 DIP 45 again, because you are pretty close to what we suggested in DIP 45
 without actually realizing it.


 I'm very much against the suggested rewriting of obj files to strip the
 export records :-)


That can easily be avoided by suppling a compiler switch which either 
actives exporting (if it is off by default) or deactivating export (if 
it is on by default).

-- 
Kind Regards
Benjamin Thaut



 Nov 20 2013



prev sibling next sibling parent reply Martin Nowak <code dawg.eu>  writes:


On 11/18/2013 06:12 AM, Walter Bright wrote:

 I'm very much against the suggested rewriting of obj files to strip the
 export records :-)


That is only one particular aspect of the DIP. It's not essential and 
maybe we'll find a different solution during implementation.
The need to statically link a library into a DLL is rare enough to 
delegate this to an extra tool.



 Dec 01 2013



  parent reply Manu <turkeyman gmail.com>  writes:


On 2 December 2013 06:05, Martin Nowak <code dawg.eu> wrote:


 On 11/18/2013 06:12 AM, Walter Bright wrote:


 I'm very much against the suggested rewriting of obj files to strip the
 export records :-)


 That is only one particular aspect of the DIP. It's not essential and
 maybe we'll find a different solution during implementation.
 The need to statically link a library into a DLL is rare enough to
 delegate this to an extra tool.


Sorry, I'm just casually following this thread. I'm very interested to have
this mess resolved, but don't have a great deal of input.
This comment surprised me a little though, you say it's rare to statically
link a lib into a DLL?

In my experience, I don't think that's rare at all, I'd say it's very
common, in fact, in my experience, I can't think of many instances where
I've built some code as a DLL which hasn't included other lib's (statically
linked).
Maybe it's just me, but if I'm building something as a DLL (rather than a
static lib), it's precisely because it is quite large, and offers generally
self-contained functionality (which often depends on other libs).
The only exception in my experience is plugin functionality.

Just a comment on other aspects of this thread...
There's all this debate around dllimport vs dllexport. And this is probably
my ignorance, but I have to wonder why dllimport even exists?
I've been using DLL's for a long time, and I've never used dllimport (that
I'm aware of).
I either statically link an import lib (which binds to the exports in a
nice efficient way), or I call LoadLibrary and find symbols manually...
I'm not really even sure what dllimport does. Why is it so important, and
why is it causing so much fuss?



 Dec 01 2013



  parent  Rainer Schuetze <r.sagitario gmx.de>  writes:


On 02.12.2013 05:59, Manu wrote:

 Just a comment on other aspects of this thread...
 There's all this debate around dllimport vs dllexport. And this is
 probably my ignorance, but I have to wonder why dllimport even exists?
 I've been using DLL's for a long time, and I've never used dllimport
 (that I'm aware of).
 I either statically link an import lib (which binds to the exports in a
 nice efficient way), or I call LoadLibrary and find symbols manually...
 I'm not really even sure what dllimport does. Why is it so important,
 and why is it causing so much fuss?


For function imports, not using dllimport magically works, but adding it 
to the declaration can avoid one jump when calling the imported function.

Imports to data variables don't have that magic, it can either be 
accessed directly, or it has to be accessed by an indirection through 
the import table. The dllimport specifies which code has to be generated.

I actually have an implementation, that patches the data accesses as a 
secondary relocation step, circumventing the indirection through the 
import table. Unfortunately it does not work on 64-bit. The problem here 
is that dmd generated code only uses 32-bit relative addressing, but 
that disallows accessing data in another shared library directly.

The respective discussion where this approach was rejected by Walter is 
supposed to be here: 
http://forum.dlang.org/thread/8bd42b4e-8c3b-42a6-9f00-2081b1329cd2 me.com , 
but it seems it is somehow reduced to a single post.



 Dec 02 2013



prev sibling  parent  Martin Nowak <code dawg.eu>  writes:


On 11/18/2013 06:12 AM, Walter Bright wrote:

 I'm very much against the suggested rewriting of obj files to strip the
 export records :-)


Changed
http://wiki.dlang.org/?title=DIP45&diff=3201&oldid=3094



 Dec 01 2013



prev sibling  parent reply Martin Nowak <code dawg.eu>  writes:


On 11/15/2013 08:32 AM, Walter Bright wrote:

 It's not that bad. Phobos can be built by specifying all the files on
 the command line.


That's the essential trade-off we have to make.
Either we come up with a clumsy way to list all DLL modules during 
compilation (separate compilation) or we add some compiler logic to 
auto-import exported data symbols. In the latter case compiled object 
files can be used for a DLL AND a static library.
We found the second approach to be more attractive.

The only issue is that most of the time one wants to strip exports when 
using objects for a static library.



 Nov 16 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/16/2013 7:22 AM, Martin Nowak wrote:

 On 11/15/2013 08:32 AM, Walter Bright wrote:

 It's not that bad. Phobos can be built by specifying all the files on
 the command line.


 That's the essential trade-off we have to make.
 Either we come up with a clumsy way to list all DLL modules during compilation
 (separate compilation) or we add some compiler logic to auto-import exported
 data symbols. In the latter case compiled object files can be used for a DLL
AND
 a static library.
 We found the second approach to be more attractive.


Why? dmd is very fast at compiling. I'm not sure what is being saved here.



 The only issue is that most of the time one wants to strip exports when using
 objects for a static library.


That's a pretty big issue (to me, anyway).



 Nov 17 2013



  parent  Martin Nowak <code dawg.eu>  writes:


On 11/18/2013 06:13 AM, Walter Bright wrote:

 Why? dmd is very fast at compiling. I'm not sure what is being saved here.


It's not the time argument, but correctly listing all modules that go 
into the same DLL as additional compilation argument.

You can see an earlier proposals of the DIP.

http://wiki.dlang.org/?title=DIP45&oldid=2921 - export(exampleLib)
     dmd -m64 -export libA -of"libA.dll" dllmain.d libA.d -L/DLL 
-L/IMPLIB:"libA.lib"
     dmd -m64 -export libB -import libA -of"libB.dll" dllmain.d libB.d 
-L/DLL -L/IMPLIB:"libB.lib" -LlibA.lib
     dmd -m64 -import libB -import libA main.d -LlibA.lib -LlibB.lib



 Dec 01 2013



prev sibling  parent  Martin Nowak <code dawg.eu>  writes:


On 11/14/2013 11:28 AM, Walter Bright wrote:

 One possibility is modules listed on the command line are regarded as
 export==dllexport, and other modules as export==dllimport.


We've considered this and it doesn't work for separate compilation.
This means I couldn't compile phobos dll with unittests.



 Nov 16 2013



prev sibling  parent reply "Martin Nowak" <code dawg.eu>  writes:


On Sunday, 10 November 2013 at 21:50:51 UTC, Benjamin Thaut wrote:

 You completely ignored my inlining argument. Lets assume there 
 is some small function that qualifies for inlining but accesses 
 a TLS variable marked with the export attribute.


That's not an issue, you can't inline exported functions.
This is because the actual implementation is only know at 
link-time (only at run-time on UNIX).



 Nov 12 2013



  parent  Martin Nowak <code dawg.eu>  writes:


On 11/12/2013 11:12 PM, Martin Nowak wrote:

 That's not an issue, you can't inline exported functions.
 This is because the actual implementation is only know at link-time
 (only at run-time on UNIX).


One might chose to ignore this possibly incorrect behaviour and still 
inline in favour of speed, but I don't think we need to worry about this 
now.



 Nov 16 2013



prev sibling  parent  "Martin Nowak" <code dawg.eu>  writes:


On Sunday, 10 November 2013 at 19:47:16 UTC, Walter Bright wrote:

 It's long been recognized that using global variables to 
 communicate between interfaces is a bad idea. And it isn't even 
 supported for dlls, because the proposed solution to making 
 them work is to wrap them with a function (much like D 
 properties).

 We'd be going out of our way to support a recognized bad 
 paradigm. There is currently no existing D code that requires 
 this. If we add it, then we'd be stuck with supporting it for 
 backwards compatibility. If we don't add it, and it becomes 
 some sort of crisis that it isn't supported, we can add it in 
 later without breaking things.


Yes, data interfaces (be it TLS or __gshared) are bad and make it 
more difficult to maintain a stable ABI. I also agree that TLS 
accesses shouldn't be a big problem.



 Nov 12 2013



prev sibling  parent reply Rainer Schuetze <r.sagitario gmx.de>  writes:


On 10.11.2013 12:38, Benjamin Thaut wrote:

 Attached you will find a e-mail conversation about the discussion of DIP
 45. I moved the discussion here so everyone can contribute.

 Hi,

 DIP 45 is waiting for some offical approval / comment so it can be
 implemented. It would be great if one of you could look into the topic
 and give official approval or improvement suggestions as the DIP has
 been in limbo for quite some time now. I'm not willing to start working
 on it before there is official approval, because I made bad experiences
 with this in the past.

 In my opinion the DIP is in a state where further discussion will not
 lead to any improvements. The DIP should be implemented to see which
 problems arrise during implementation to then discuss how to solve them.

 I've also seen a couple of questions to the state of Windows DLL support
 and shared libraries on the newsgroup. A working export keyword is
 required to actually implement that support on windows.

 The DIP: http://wiki.dlang.org/DIP45
 The latest discussion on the NG:
 http://forum.dlang.org/post/kvhu2c$2ikq$1 digitalmars.com
 Bug associated with the problem:
 http://d.puremagic.com/issues/show_bug.cgi?id=9816

 Kind Regards
 Benjamin Thaut


I think the DIP is very reasonable. Some random comments:

- when building a single object file or library, I would expect that it 
won't be torn apart later, so optimizations currently limited to 
accesses within the same module by the DIP could be extended to the 
modules included in current compilation target.

- a better example where cross-DLL data accesses to globals are needed 
is TypeInfo, e.g. when a class info references it's base class or 
interfaces.

- I'd be fine with reversing the default for stripping exports from a 
library, i.e. replacing "-libexports" with "-stripexports".

- I agree that cross DLL TLS accesses should be possible, though it 
might be implemented in a second step. Last time I tried it with C++, it 
wasn't supported, but it is probably not missed that much as thread 
local data is not as ubiquitous as in D.

- The DIP should state what happens to template instances. Assuming the 
definition is marked "export", I guess instances will be dllexport if 
they are compiled together with the module that contains the definition. 
What will happen if it is instantiated, but the defining module is 
merely imported?

Rainer



 Nov 12 2013



  parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 13.11.2013 08:31, schrieb Rainer Schuetze:

 - The DIP should state what happens to template instances. Assuming the
 definition is marked "export", I guess instances will be dllexport if
 they are compiled together with the module that contains the definition.
 What will happen if it is instantiated, but the defining module is
 merely imported?

 Rainer


That is a very good point I also thought about. But because I'm not 
familiar with how exactly template instanciation works I can't really 
provide a solution for that.

Does Walters recent template instanciation optimization also take into 
account template instaces of templates that are defined outside of the 
module that contains the template definition? I think the behaviour of 
exported template should mainly satisfy the needs of this optimization.

Kind Regards
Benjamin Thaut



 Nov 13 2013



  parent reply Rainer Schuetze <r.sagitario gmx.de>  writes:


On 13.11.2013 09:27, Benjamin Thaut wrote:

 Am 13.11.2013 08:31, schrieb Rainer Schuetze:

 - The DIP should state what happens to template instances. Assuming the
 definition is marked "export", I guess instances will be dllexport if
 they are compiled together with the module that contains the definition.
 What will happen if it is instantiated, but the defining module is
 merely imported?

 Rainer


 That is a very good point I also thought about. But because I'm not
 familiar with how exactly template instanciation works I can't really
 provide a solution for that.

 Does Walters recent template instanciation optimization also take into
 account template instaces of templates that are defined outside of the
 module that contains the template definition? I think the behaviour of
 exported template should mainly satisfy the needs of this optimization.



As far as I understand, the optimization avoids generating code for 
template instances only _created_ (not defined) by imported modules, 
e.g. when needed for semantic analysis inside the imported module, but 
never actually referenced by generated code by the root modules.

[OT: The problem of the current implementation is the detection of 
"only" in that definition, it fails too often leading to linker errors 
and the addition of that terrible -allinst switch. I had to disable the 
optimization for me before addition of that switch.]

With respect to export I guess the optimization doesn't change much as 
multiple instances of the same template instance still can be found in 
different object files (maybe even less predictable).

To avoid the trouble with templates, here is an idea for discussion:

- templates definitions by themselves are never dllexport/dllimport

- template instances are dllexport/dllimport with the proposed semantics 
of normal functions/variables if they are explicitely declared as 
"export" by an alias statement, e.g.

   export alias templ!int exported_tmpl_int;

- implicitely created template instances don't create dllexport 
versions, and use dllimport semantics if the declaration above is found 
in an import.

This would make it very explicit which module dllexports the template 
instance symbols. It is a bit like using typedef in C++ to ensure 
template instantiation.

One problem that might appear is that there are some conflicts if some 
modules import the "export alias" declaration, and some do not:

- In the dllexport case, it means that there are different COMDATs for 
the code and data of the template and it is undefined which one is 
chosen. (For Win64 COFF the exporting of the symbol is a separate 
directive, so it shouldn't be a problem, but I don't know for other 
object file formats.)

- It's worse for the dllimport case, as data accesses to template 
variables will sometimes use the additional indirection, sometimes not.



 Nov 13 2013



  parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 14.11.2013 08:36, schrieb Rainer Schuetze:

 As far as I understand, the optimization avoids generating code for
 template instances only _created_ (not defined) by imported modules,
 e.g. when needed for semantic analysis inside the imported module, but
 never actually referenced by generated code by the root modules.

 [OT: The problem of the current implementation is the detection of
 "only" in that definition, it fails too often leading to linker errors
 and the addition of that terrible -allinst switch. I had to disable the
 optimization for me before addition of that switch.]

 With respect to export I guess the optimization doesn't change much as
 multiple instances of the same template instance still can be found in
 different object files (maybe even less predictable).

 To avoid the trouble with templates, here is an idea for discussion:

 - templates definitions by themselves are never dllexport/dllimport

 - template instances are dllexport/dllimport with the proposed semantics
 of normal functions/variables if they are explicitely declared as
 "export" by an alias statement, e.g.

    export alias templ!int exported_tmpl_int;

 - implicitely created template instances don't create dllexport
 versions, and use dllimport semantics if the declaration above is found
 in an import.


I actually like this idea. Everytime I actually had to dllexport / 
dllimport templates in C++ I knew beforehand which types I need the 
template exported for.

But what would should in a case like this:

class WeakReferenced(T)
{
   export __gshared WeakReferenced!T[] m_weakTable;
}

Note that the export is not applied to the template directly but instead 
to a specific member inside the template.

Should this be an error?


 - It's worse for the dllimport case, as data accesses to template
 variables will sometimes use the additional indirection, sometimes not.


Why exactly is that? Would that also be the case with the system 
proposed by the DIP? The dip never actually uses dllimport, so shouldn't 
it just work (tm)?

Kind Regards
Benjamin Thaut



 Nov 14 2013



  parent reply Rainer Schuetze <r.sagitario gmx.de>  writes:


On 14.11.2013 18:25, Benjamin Thaut wrote:

 Am 14.11.2013 08:36, schrieb Rainer Schuetze:

 As far as I understand, the optimization avoids generating code for
 template instances only _created_ (not defined) by imported modules,
 e.g. when needed for semantic analysis inside the imported module, but
 never actually referenced by generated code by the root modules.

 [OT: The problem of the current implementation is the detection of
 "only" in that definition, it fails too often leading to linker errors
 and the addition of that terrible -allinst switch. I had to disable the
 optimization for me before addition of that switch.]

 With respect to export I guess the optimization doesn't change much as
 multiple instances of the same template instance still can be found in
 different object files (maybe even less predictable).

 To avoid the trouble with templates, here is an idea for discussion:

 - templates definitions by themselves are never dllexport/dllimport

 - template instances are dllexport/dllimport with the proposed semantics
 of normal functions/variables if they are explicitely declared as
 "export" by an alias statement, e.g.

    export alias templ!int exported_tmpl_int;

 - implicitely created template instances don't create dllexport
 versions, and use dllimport semantics if the declaration above is found
 in an import.


 I actually like this idea. Everytime I actually had to dllexport /
 dllimport templates in C++ I knew beforehand which types I need the
 template exported for.

 But what would should in a case like this:

 class WeakReferenced(T)
 {
    export __gshared WeakReferenced!T[] m_weakTable;
 }


According to the rule above "export" on the template definition has no 
effect, only when it is instantiated with

export alias WeakReferenced!int WeakRefIntArray;


 Note that the export is not applied to the template directly but instead
 to a specific member inside the template.

 Should this be an error?


Maybe. Another rule might be that only the declarations actually 
annotated with "export" gets exported with the instantiation, so you 
could add "export" to the whole class or only some declaraations.


 - It's worse for the dllimport case, as data accesses to template
 variables will sometimes use the additional indirection, sometimes not.


 Why exactly is that? Would that also be the case with the system
 proposed by the DIP? The dip never actually uses dllimport, so shouldn't
 it just work (tm)?


The problem is that the template instantiation can exist without the 
compiler seeing the alias:

-----------
module a;
class tmpl(T) { static shared int var; }

-----------
module b;
import a;
export alias tmpl!int tmpl_int;

-----------
module c;
import b;
int func() { return tmpl!int.var; } // uses dllimport access with 
indirection

-----------
module d;
import a;
int fund() { return tmpl!int.var; } // uses standard access without 
indirection

compiling c and d as single files will silently generate different code, 
because when compiling d, the export alias is never seen.

(this cannot happen with standard variables, only when declared multiple 
times, but differently, with extern(C/C++/System)).



 Nov 14 2013



  parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 15.11.2013 00:38, schrieb Rainer Schuetze:

 Maybe. Another rule might be that only the declarations actually
 annotated with "export" gets exported with the instantiation, so you
 could add "export" to the whole class or only some declaraations.


I don't think this is a good idea. It should be possible to put 
"export:" on top of a file and just export everything. If you limit it 
to decelerations the following would work:

export __gshared int g_var;

but the following wouldn't:

export __gshared int g_var = 0;

Although it would really produce equivalent code.


 compiling c and d as single files will silently generate different code,
 because when compiling d, the export alias is never seen.

 (this cannot happen with standard variables, only when declared multiple
 times, but differently, with extern(C/C++/System)).


And do you already have a idea how we could work around this problem?

Kind Regards
Benjamin Thaut



 Nov 14 2013



  parent reply Rainer Schuetze <r.sagitario gmx.de>  writes:


On 15.11.2013 08:02, Benjamin Thaut wrote:

 Am 15.11.2013 00:38, schrieb Rainer Schuetze:

 Maybe. Another rule might be that only the declarations actually
 annotated with "export" gets exported with the instantiation, so you
 could add "export" to the whole class or only some declaraations.


 I don't think this is a good idea. It should be possible to put
 "export:" on top of a file and just export everything. If you limit it
 to decelerations the following would work:

 export __gshared int g_var;

 but the following wouldn't:

 export __gshared int g_var = 0;

 Although it would really produce equivalent code.


I don't follow. What does this have to do with template instances?
I was referring to your example where you wanted to export just one 
symbol from a template class.


 compiling c and d as single files will silently generate different code,
 because when compiling d, the export alias is never seen.

 (this cannot happen with standard variables, only when declared multiple
 times, but differently, with extern(C/C++/System)).


 And do you already have a idea how we could work around this problem?


It might be possible to produce some linker errors: e.g. the dllimport 
version drags in the __imp_var symbol that also provides a _var 
definition record that produces some link error (e.g. by referring to a 
non-existing symbol). If it links to a non-dllimport version that 
actually refers to _var, it bails out.

Generated COMDAT records that define the _var symbol aswell might cause 
problems here, though, because the current COMDAT selection strategy is 
"pick any".



 Nov 15 2013



  parent reply Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 15/11/2013 18:04, schrieb Rainer Schuetze:

 I don't follow. What does this have to do with template instances?
 I was referring to your example where you wanted to export just one
 symbol from a template class.


Then I misunderstood your use of the word "declaration". With 
declarations you did mean template declarations only?



 Nov 15 2013



  parent reply Rainer Schuetze <r.sagitario gmx.de>  writes:


On 15.11.2013 18:18, Benjamin Thaut wrote:

 Am 15/11/2013 18:04, schrieb Rainer Schuetze:

 I don't follow. What does this have to do with template instances?
 I was referring to your example where you wanted to export just one
 symbol from a template class.


 Then I misunderstood your use of the word "declaration". With
 declarations you did mean template declarations only?


I actually meant declaration inside template definitions as in your example:

class WeakReferenced(T)
{
   export __gshared WeakReferenced!T[] m_weakTable;
}

The proposal was that

export alias WeakReferenced!int exported_weak_int_array;

would only export the symbol for m_weakTable, not any other class 
symbols like the class info.

I'm not sure if this is a good idea, I'm just exploring possibilities here.



 Nov 15 2013



  parent  Benjamin Thaut <code benjamin-thaut.de>  writes:


Am 15/11/2013 18:41, schrieb Rainer Schuetze:

 I actually meant declaration inside template definitions as in your
 example:

 class WeakReferenced(T)
 {
    export __gshared WeakReferenced!T[] m_weakTable;
 }

 The proposal was that

 export alias WeakReferenced!int exported_weak_int_array;

 would only export the symbol for m_weakTable, not any other class
 symbols like the class info.

 I'm not sure if this is a good idea, I'm just exploring possibilities here.


Sounds good to me. It gives the ability to percisely select what gets 
exported and what not. Basically also matches was C++ does.



 Nov 15 2013