www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 11284] New: add -allinst compiler switch

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11284

           Summary: add -allinst compiler switch
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: DMD
        AssignedTo: nobody puremagic.com
        ReportedBy: bugzilla digitalmars.com



16:36:37 PDT ---
Given these two files:

  ----- a.d ------
  int foo(T)(T i) { return i;)
  enum x = foo(3);
  enum f = foo(2.3);
  ---- test.d ----
  import a;
  int y = foo(4);
  ----------------

Compiling with 2.063 or earlier with:

    dmd test -c

would generate an instance of foo!int and foo!double into test.obj. In fact,
every file that imported a.d would have those instantiations inserted into its
resulting object file. This didn't affect the executable, as the linker would
remove duplicates (the magic of COMDAT segments).

However, template heavy imports can and did result in inordinately massive .obj
files, for example, try importing std.stdio.

Starting with 2.064, the compiler notices that foo!double is instantiated only
by import a, and so the code generated for foo!double would presumably be in
the object file a.obj, and so the compiler does not have to insert it into
test.obj. This can potentially result in large compiler speed increases, which
are a big feature of D.

If the foo!double is not in a.obj, then the user sees "undefined symbol
foo!double" coming from the linker in code that worked with 2.063, and files a
bug report. This circumstance can appear when:

1. a.obj is never linked in, i.e. a.d is regarded as a "header only" file.

2. a.d was modified but a.obj was never recompiled with the newer a.d.

3. a.d's template instantiations are behind VersionDeclaration's and the
versions in effect when a.d is compiled are different than the versions when
test.d is compiled.

4. Other forms of conditional compilation analogous to case (3).

I regard these as user build system irregularities, not bugs. However, user
confusion over this, and frustration over "it worked in 2.063 and now I get
this error message, what's wrong" is very real. Hence, I propose the -allinst
switch which will generate code for all instantiated templates, like 2.063
does.

This way, the user can throw the switch and continue working as before, or if
he wants to track down the source of the undefined symbol at least the switch
provides a starting point in diagnosing if it is one of cases 1..4 or something
else.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 16 2013
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11284


jfanatiker gmx.at changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jfanatiker gmx.at



It makes sense, but in this particular case: Why is foo!double put into a.obj
in the first place, when it is only used in CTFE? From reading the NG, it seems
there are a lot of unnecessary templates instantiated, for things like template
constraints. 

Best regards,
Robert

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 18 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11284


Dicebot <public dicebot.lv> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |public dicebot.lv





 It makes sense, but in this particular case: Why is foo!double put into a.obj
 in the first place, when it is only used in CTFE? From reading the NG, it seems
 there are a lot of unnecessary templates instantiated, for things like template
 constraints. 
 
 Best regards,
 Robert


There are 2 reasons for this.

First, related part of DMD source code is damn complicated. I have wasted
several months trying to figure this out and when Walter came out with his own
partial solution I was hardly even able to understand reasoning behind those
exact changes.

But probably I am just stupid. Second issue is much more important though:

Currently D spec does not allow any symbol visibility optimizations. The fact
that symbol X is used only during CTFE during compilation of module A does not
mean it won't be used in runtime from module B. Or even linked dynamically. Any
really clever stuff can be done here only after specification is restricted.

I have had a proposal for internal linkage attribute ages ago which is tightly
related but I am not skilled enough to implement it right now.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 18 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11284




15:58:07 PDT ---
https://github.com/D-Programming-Language/dmd/pull/2680

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 18 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11284


Max Samukha <samukha voliacable.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |samukha voliacable.com



PDT ---


 It makes sense, but in this particular case: Why is foo!double put into a.obj
 in the first place, when it is only used in CTFE? From reading the NG, it seems
 there are a lot of unnecessary templates instantiated, for things like template
 constraints. 
 
 Best regards,
 Robert


That is the problem. There should be a way to specify that a function is
intended for use only in CTFE. I tend to wrap the body of such functions in "if
(__ctfe) {...} else assert(false); but that still sucks.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 18 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11284






 One way to improve the situation a little that I think doesn't require compiler
 changes is to add a "private":
 
 private int foo(int x) { return x ^^ 2; }
 
 If this "foo" is used only at compile-time inside the module, then perhaps it
 should not appear in the module object.


Unfortunately, it is not that simple.

First, it may break existing code as currently it has external linkage and can
be used from C modules. Second, some private symbols may actually leak outside
of the module via public aliases and function parameters / return values. This
is partially explained here:

http://wiki.dlang.org/Access_specifiers_and_visibility
http://wiki.dlang.org/DIP22.1

What is more important, though, is that templates and CTFE functions are rarely
used in the same module they are declared. I have several ideas for more
efficient design that can fix this but all contain breaking changes when it
comes to separate compilation (in absence of whole program optimization at
least).

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 18 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11284





 Currently D spec does not allow any symbol visibility optimizations. The fact
 that symbol X is used only during CTFE during compilation of module A does not
 mean it won't be used in runtime from module B. Or even linked dynamically. Any
 really clever stuff can be done here only after specification is restricted.


I don't really get this. I am not talking about omitting regular functions
which are only used during CTFE in a module, because obviously the function
might be of use at runtime from within another module. But why do we put a
template instantiation in a.obj that never really got instantiated? If b.d uses
it, it should go to b.obj just as if a.d would not have used it at all.

I don't get why the use in CTFE triggers template instantiations.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 20 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11284


Denis Shelomovskij <verylonglogin.reg gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |verylonglogin.reg gmail.com



15:10:52 MSD ---


 I regard these as user build system irregularities, not bugs. However, user
 confusion over this, and frustration over "it worked in 2.063 and now I get
 this error message, what's wrong" is very real. Hence, I propose the -allinst
 switch which will generate code for all instantiated templates, like 2.063
 does.


Every release "accept-invalid" bugs are fixed and incorrect user code that used
to work become uncompileable (showing user his mistakes and making user happy
:) ).

If in this particular case invalid compilation actions are too common to be
broken immediately:
1. Links to user complains are needed to proof its huge count.
2. We have to find and resolve our mistake (e.g. lack of docs) that causes D
user to do such errors in building process.

Also IMHO points 2, 3, and 4 from the description are obviously user build
process bugs that have to be pointed out by a tool-chain like it do now to
prevent the worst case of buildable but incorrectly working application.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 20 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11284




15:17:22 MSD ---
Also it's too early to talk about such switch because symbol elimination

detect symbol instantiating module (see (already fixed) Issue 11069 and Issue
11114) and thus can't be included in releases at least until known regressions
are fixed.

[1] https://github.com/D-Programming-Language/dmd/pull/2550

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 20 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11284




Commits pushed to master at https://github.com/D-Programming-Language/dmd

https://github.com/D-Programming-Language/dmd/commit/a1f9f08b78f6a358e98ac7a4b0423b0ae779e943
fix Issue 11284 - add -allinst compiler switch

https://github.com/D-Programming-Language/dmd/commit/58355d7481b53e5a1e88c12f72075832c0c8c04a


fix Issue 11284 - add -allinst compiler switch

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 23 2013
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11284




Commit pushed to 2.064 at https://github.com/D-Programming-Language/dmd

https://github.com/D-Programming-Language/dmd/commit/452d8ad9672fbc0745b539b0ef6cc40e421cf06a


fix Issue 11284 - add -allinst compiler switch

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 23 2013