www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Need clarification on dmd symbol generation

reply "Dicebot" <m.strashun gmail.com> writes:
I am currently investigating 
http://d.puremagic.com/issues/show_bug.cgi?id=9571 and after 
brief exploration of symbols emitted to object file as well as 
code that emits them I can't rid of an idea that I am missing 
something about module system.

Why does dmd emits horde of symbols from imported modules into 
object files, most of whose are not actually used in this one? No 
"-inline" used and I kind of expected to see there only directly 
called function and symbols from their signatures.

What does actually happen here?
Apr 04 2013
next sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Thursday, 4 April 2013 at 19:34:06 UTC, Dicebot wrote:

 I am currently investigating 
 http://d.puremagic.com/issues/show_bug.cgi?id=9571 and after 
 brief exploration of symbols emitted to object file as well as 
 code that emits them I can't rid of an idea that I am missing 
 something about module system.

 Why does dmd emits horde of symbols from imported modules into 
 object files, most of whose are not actually used in this one? 
 No "-inline" used and I kind of expected to see there only 
 directly called function and symbols from their signatures.

 What does actually happen here?


Okay, after some digging on the topic I am quite sure that DMD 
does add instantiated template symbols from imported modules to 
root module symbol list (template.c:5023). It does not care if 
root module was actually one to start template instantiation 
chain (and thus uses this symbol) and just bloats the symbol 
table.

Repeating initial question - was there rationale for this other 
than being reluctant to store additional context information 
about initial template chain instantiation scope?

I'd like to make a pull request for this and make one step closer 
to making separate compilation reality but don't want to waste 
time on dmd source code (argh) without real need, thus 
preliminary question.
Apr 05 2013
prev sibling next sibling parent "Dicebot" <m.strashun gmail.com> writes:
I am really gonna rise that up until I get some answer.
"GTFO" is acceptable answer and still better than no answer.
Apr 08 2013
prev sibling next sibling parent "Dicebot" <m.strashun gmail.com> writes:
.
Apr 10 2013
prev sibling parent reply kenji hara <k.hara.pg gmail.com> writes:
Sorry I have no answer for the issue. That is still unknown area to me...

Kenji Hara


2013/4/5 Dicebot <m.strashun gmail.com>


 I am currently investigating http://d.puremagic.com/issues/**
 show_bug.cgi?id=9571 <http://d.puremagic.com/issues/show_bug.cgi?id=9571>and
after brief exploration of symbols emitted to object file as well as
 code that emits them I can't rid of an idea that I am missing something
 about module system.

 Why does dmd emits horde of symbols from imported modules into object
 files, most of whose are not actually used in this one? No "-inline" used
 and I kind of expected to see there only directly called function and
 symbols from their signatures.

 What does actually happen here?
Apr 10 2013
parent reply "Dicebot" <m.strashun gmail.com> writes:
On Wednesday, 10 April 2013 at 08:41:35 UTC, kenji hara wrote:

 Sorry I have no answer for the issue. That is still unknown 
 area to me...

 Kenji Hara


Thanks for replying! Really appreciated.

May be you can give me a small insight on dmd internal data 
representation then? I think I have got the issue and know what 
needs to be done, but can't find reasonably short solution within 
existing code structure.
Apr 10 2013
parent reply kenji hara <k.hara.pg gmail.com> writes:
2013/4/10 Dicebot <m.strashun gmail.com>


 On Wednesday, 10 April 2013 at 08:41:35 UTC, kenji hara wrote:


 Sorry I have no answer for the issue. That is still unknown area to me...

 Kenji Hara


 Thanks for replying! Really appreciated.

 May be you can give me a small insight on dmd internal data representation
 then? I think I have got the issue and know what needs to be done, but
 can't find reasonably short solution within existing code structure.


OK. I don't understand the issue enough, but I'd like to help you.

- TemplateInstance::tinst may represent the enclosing template instance
that 'this' instance is instantiated. It is set in
TemplateInstance::semantic(|2|3) for its member's semantic process.

- TemplateInstance::enclosing may represent the parent of nested template
instance. If a template is instantiated in function scope with implicit
context, it points the function. It is set in
TemplateInstance::hasNestedArgs.

- Module::importedFrom may represents the "root module" of import chain. it
points one of the module that listed in command line.

Good luck.

Kenji Hara
Apr 10 2013
parent reply "Dicebot" <m.strashun gmail.com> writes:
On Wednesday, 10 April 2013 at 09:37:44 UTC, kenji hara wrote:

 OK. I don't understand the issue enough, but I'd like to help 
 you.

 - TemplateInstance::tinst may represent the enclosing template 
 instance
 that 'this' instance is instantiated. It is set in
 TemplateInstance::semantic(|2|3) for its member's semantic 
 process.

 - TemplateInstance::enclosing may represent the parent of 
 nested template
 instance. If a template is instantiated in function scope with 
 implicit
 context, it points the function. It is set in
 TemplateInstance::hasNestedArgs.

 - Module::importedFrom may represents the "root module" of 
 import chain. it
 points one of the module that listed in command line.

 Good luck.

 Kenji Hara


I see no TemplateInstance::enclosing, only Scope::enclosing and 
TemplateInstance is not Scope descendant. Essentially I need to 
emit symbols not for "importedFrom" module but for module of 
top-most tinst in chain. I supposed that iterating tinst can do 
the trick, but it is not. For example, when map is used, it call 
templated range functions somewhere down the line and iterating 
up from those don't get you initial map call scope. (Same issue 
in printInstantiationTrace as it uses this method).

Is this a problem with setting correct tinst initialization in 
other code, my misunderstanding or, probably, this info may be 
relied upon only in semantic3, not semantic1?
Apr 10 2013
next sibling parent reply "Robert" <jfanatiker gmx.at> writes:
TemplateInstance::enclosing, also does not get you the 
instantiating context, referring to the comment in the source 
code, I think for my tests it was null.

importedFrom was no solution either. The only thing that worked 
for me was walking up with tinst and then use loc. This way you 
get the instantiating file. (Got that from 
printInstantiationTrace) I think it should be possible to use 
this found file name to find the module by iterating 
Module::modules, but I have not even bothered trying, because I 
felt there has to be some better way, unfortunately I also got no 
replies.

At least it helps that I am not alone with this problem :-)

Maybe it is not entirely unlikely that there is in fact no better 
way, as apart from "where do I put the instantiated template" and 
error messages there is really no need to know where the template 
got instantiated.


Best regards,
Robert
Apr 10 2013
parent reply "Dicebot" <m.strashun gmail.com> writes:
On Wednesday, 10 April 2013 at 11:53:51 UTC, Robert wrote:

 TemplateInstance::enclosing, also does not get you the 
 instantiating context, referring to the comment in the source 
 code, I think for my tests it was null.

 importedFrom was no solution either. The only thing that worked 
 for me was walking up with tinst and then use loc. This way you 
 get the instantiating file. (Got that from 
 printInstantiationTrace) I think it should be possible to use 
 this found file name to find the module by iterating 
 Module::modules, but I have not even bothered trying, because I 
 felt there has to be some better way, unfortunately I also got 
 no replies.

 At least it helps that I am not alone with this problem :-)

 Maybe it is not entirely unlikely that there is in fact no 
 better way, as apart from "where do I put the instantiated 
 template" and error messages there is really no need to know 
 where the template got instantiated.


Excited to see I am not the only one caring about this! :) 
Module* can be found via tinst->scope->module, so it is pretty 
straightforward. I have tried to use this and it _almost_ worked 
but some symbols were still missing (I have mentioned example 
with map and range before, can reduce test case if you are 
interested).

importFrom is obvious hack instead of precisely tracking symbol 
source - "lets just emit weak symbols for every compiled file 
from all his imports and let linker clean up the mess".

Considering Kenji's explanation this seems to work as I have 
initially understood. Then I need to reduce that case and see why 
those std.range template symbols are not propagated to the top of 
the chain.

This is a major blocker for getting reliable and efficient 
separate compilation and bugzilla issue references in topic is 
just an observable side-effect.
Apr 10 2013
parent reply Robert <jfanatiker gmx.at> writes:
 Excited to see I am not the only one caring about this! :) 
 Module* can be found via tinst->scope->module, so it is pretty 
 straightforward. I have tried to use this and it _almost_ worked 
 but some symbols were still missing (I have mentioned example 
 with map and range before, can reduce test case if you are 
 interested).


For me tinst->scope->module resolved to the module where the template is
declared not the instantiating one, while tinst->loc.filename is the
instantiating file. I am interested!




 
 Considering Kenji's explanation this seems to work as I have 
 initially understood. Then I need to reduce that case and see why 
 those std.range template symbols are not propagated to the top of 
 the chain.
 
 This is a major blocker for getting reliable and efficient 
 separate compilation and bugzilla issue references in topic is 
 just an observable side-effect.


Getting reliable building is also quite important for dub, I don't want
to work around dmd bugs (sometimes not even feasible), I would rather
fix them :-)
Apr 10 2013
parent "Dicebot" <m.strashun gmail.com> writes:
On Wednesday, 10 April 2013 at 14:15:11 UTC, Robert wrote:

 ...


Yes, have found some issues with scope->module and finally 
resorted to getModule().
Please follow pull request : 
https://github.com/D-Programming-Language/dmd/pull/1882
Apr 10 2013
prev sibling parent reply kenji hara <k.hara.pg gmail.com> writes:
2013/4/10 Dicebot <m.strashun gmail.com>


 On Wednesday, 10 April 2013 at 09:37:44 UTC, kenji hara wrote:


 OK. I don't understand the issue enough, but I'd like to help you.

 - TemplateInstance::tinst may represent the enclosing template instance
 that 'this' instance is instantiated. It is set in
 TemplateInstance::semantic(|2|**3) for its member's semantic process.

 - TemplateInstance::enclosing may represent the parent of nested template
 instance. If a template is instantiated in function scope with implicit
 context, it points the function. It is set in
 TemplateInstance::**hasNestedArgs.

 - Module::importedFrom may represents the "root module" of import chain.
 it
 points one of the module that listed in command line.

 Good luck.

 Kenji Hara


 I see no TemplateInstance::enclosing, only Scope::enclosing and
 TemplateInstance is not Scope descendant.



Very recently TemplateInstance::isnested is renamed to
TemplateInstance::enclosing.



 Essentially I need to emit symbols not for "importedFrom" module but for
 module of top-most tinst in chain. I supposed that iterating tinst can do
 the trick, but it is not. For example, when map is used, it call templated
 range functions somewhere down the line and iterating up from those don't
 get you initial map call scope. (Same issue in printInstantiationTrace as
 it uses this method).


It sounds that using TemplateInstance::enclosing is correct.

Kenji Hara
Apr 10 2013
next sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Wednesday, 10 April 2013 at 12:36:35 UTC, kenji hara wrote:

 ...


Can't really express how big my gratitude is! :) Already trying 
"enclosing".
Apr 10 2013
prev sibling parent reply "Dicebot" <m.strashun gmail.com> writes:
You now what? It worked.

Before:
$ nm a.o | wc -l
358

After:
$ nm a.o | wc -l
68

I'll run a full test suite and do the pull request.

Kenji you are my hero ;)
Apr 10 2013
parent reply Robert <jfanatiker gmx.at> writes:
On Wed, 2013-04-10 at 15:50 +0200, Dicebot wrote:

 You now what? It worked.
 
 Before:
 $ nm a.o | wc -l
 358
 
 After:
 $ nm a.o | wc -l
 68
 
 I'll run a full test suite and do the pull request.
 
 Kenji you are my hero ;)


:-( Still wrong output for my code. I commented on your pull request
with more details.
Apr 10 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 10 April 2013 at 15:16:31 UTC, Robert wrote:

 On Wed, 2013-04-10 at 15:50 +0200, Dicebot wrote:

 You now what? It worked.
 
 Before:
 $ nm a.o | wc -l
 358
 
 After:
 $ nm a.o | wc -l
 68
 
 I'll run a full test suite and do the pull request.
 
 Kenji you are my hero ;)


 :-( Still wrong output for my code. I commented on your pull 
 request
 with more details.


Please add sample code to 
http://d.puremagic.com/issues/show_bug.cgi?id=9571
Apr 10 2013
parent "Dicebot" <m.strashun gmail.com> writes:
On Wednesday, 10 April 2013 at 16:01:08 UTC, deadalnix wrote:

 Please add sample code to 
 http://d.puremagic.com/issues/show_bug.cgi?id=9571


I know what Robert is speaking about, it is different issue 
though also related to template symbols. It does not belong to 
9571 but I try to address it too.
Apr 10 2013