www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - are mixin string templates with functions removed?

reply "JS" <js.mdnq gmail.com> writes:
If I have a bunch of templates that are only used for code 
generation, are they removed in the binary(since they are not 
needed)?
Aug 01 2013
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:

 If I have a bunch of templates that are only used for code 
 generation, are they removed in the binary(since they are not 
 needed)?


templates don't exist in binaries.
Aug 01 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:

 On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:

If I have a bunch of templates that are only used for code
generation, are they removed in the binary(since they are not
needed)?


 
 templates don't exist in binaries.


Templates are like cookie molds, you use them to generate lots of
(almost) identical cookies, but you never serve the mold to the
customer. ;-)


T

-- 
He who does not appreciate the beauty of language is not worthy to bemoan its
flaws.
Aug 01 2013
next sibling parent reply "JS" <js.mdnq gmail.com> writes:
On Thursday, 1 August 2013 at 17:47:00 UTC, H. S. Teoh wrote:

 On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:

 On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:

If I have a bunch of templates that are only used for code
generation, are they removed in the binary(since they are not
needed)?


 
 templates don't exist in binaries.


 Templates are like cookie molds, you use them to generate lots 
 of
 (almost) identical cookies, but you never serve the mold to the
 customer. ;-)


 T


But what about the functions that exist in them?

e.g.,

template A()
{
     void A()
     {
         B();
     }
     void B() { }
}

is everything in the template removed 100% or is there junk that 
the compiler doesn't remove?


Oh... and I'm not talking about theoretical... I'm talking about 
what dmd actually does.
Aug 01 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Aug 01, 2013 at 07:52:28PM +0200, JS wrote:

 On Thursday, 1 August 2013 at 17:47:00 UTC, H. S. Teoh wrote:

On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:

On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:

If I have a bunch of templates that are only used for code
generation, are they removed in the binary(since they are not
needed)?


templates don't exist in binaries.


Templates are like cookie molds, you use them to generate lots of
(almost) identical cookies, but you never serve the mold to the
customer. ;-)


T


 
 But what about the functions that exist in them?


Like I said, cookie molds. You use the mold to press cookies, but only
the cookies are served, not the mold. The mold may be very complicated,
containing subcookies attached to bigger cookies, but whatever is
pressed (i.e., instantiated) is what's served on the dinner plate. The
mold always remains in the kitchen.



 template A()
 {
     void A()
     {
         B();
     }
     void B() { }
 }
 
 is everything in the template removed 100% or is there junk that the
 compiler doesn't remove?


There is nothing to remove. If you instantiated the template, then you
get a copy of everything in it. The number of copies equals the number
of distinct instantiations. The template itself is just a mold, an
abstract entity that doesn't exist in binary form. What it does is to
serve as a mold (hence, "template") to make code. So if you use to make
10 copies of the code, that's what you'll get in your executable. If
your template contains 5 functions, then each instantiation produces 5
copies of those functions. Simple as that.

Of course, not all code produces binary data -- enum and alias
definitions don't produce any binary code, for example -- they're just
logical entities that only exist at compile time. So if you have an enum
inside a template, it will get copied however many times you instantiate
the template, but none of those copies end up in the executable because
they're just declarations, not actual code or data.



 Oh... and I'm not talking about theoretical... I'm talking about
 what dmd actually does.


See for yourself: run a disassembler on the result (preferably filtered
through ddemangle so the symbols are actually readable) and see what's
included and what's not. On Posix, you can use objdump or nm (if you use
nm, though, keep in mind that not all symbols correspond with actual
code/data, but they do take up a little space in the executable -- at
least enough to store their names).


T

-- 
MACINTOSH: Most Applications Crash, If Not, The Operating System Hangs
Aug 01 2013
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 18:09:54 UTC, H. S. Teoh wrote:

 See for yourself: run a disassembler on the result (preferably 
 filtered
 through ddemangle so the symbols are actually readable) and see 
 what's
 included and what's not. On Posix, you can use objdump or nm 
 (if you use
 nm, though, keep in mind that not all symbols correspond with 
 actual
 code/data, but they do take up a little space in the executable 
 -- at
 least enough to store their names).


Done it, see my previous comment here. I do have an idea how to 
fix it, but
wasn't able to get though dmd code so far :(
Aug 01 2013
prev sibling parent reply "JS" <js.mdnq gmail.com> writes:
On Thursday, 1 August 2013 at 18:09:54 UTC, H. S. Teoh wrote:

 On Thu, Aug 01, 2013 at 07:52:28PM +0200, JS wrote:

 On Thursday, 1 August 2013 at 17:47:00 UTC, H. S. Teoh wrote:

On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:

On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:

If I have a bunch of templates that are only used for code
generation, are they removed in the binary(since they are 
not
needed)?


templates don't exist in binaries.


Templates are like cookie molds, you use them to generate 
lots of
(almost) identical cookies, but you never serve the mold to 
the
customer. ;-)


T


 
 But what about the functions that exist in them?


 Like I said, cookie molds. You use the mold to press cookies, 
 but only
 the cookies are served, not the mold. The mold may be very 
 complicated,
 containing subcookies attached to bigger cookies, but whatever 
 is
 pressed (i.e., instantiated) is what's served on the dinner 
 plate. The
 mold always remains in the kitchen.



 template A()
 {
     void A()
     {
         B();
     }
     void B() { }
 }
 
 is everything in the template removed 100% or is there junk 
 that the
 compiler doesn't remove?


 There is nothing to remove. If you instantiated the template, 
 then you
 get a copy of everything in it. The number of copies equals the 
 number
 of distinct instantiations. The template itself is just a mold, 
 an
 abstract entity that doesn't exist in binary form. What it does 
 is to
 serve as a mold (hence, "template") to make code. So if you use 
 to make
 10 copies of the code, that's what you'll get in your 
 executable. If
 your template contains 5 functions, then each instantiation 
 produces 5
 copies of those functions. Simple as that.

 Of course, not all code produces binary data -- enum and alias
 definitions don't produce any binary code, for example -- 
 they're just
 logical entities that only exist at compile time. So if you 
 have an enum
 inside a template, it will get copied however many times you 
 instantiate
 the template, but none of those copies end up in the executable 
 because
 they're just declarations, not actual code or data.


Ok, I'm not talking about the template itself but what is 
contained in the template. It is obvious that templates can 
"insert" stuff into the binary.

a mixin template can easily do that.


Now are you telling me that

template A()
{
     void foo() { writeln("asdf"); }
}
void main()
{
	A!().foo();
}

does not create a function foo in the binary? That it is 
equivalent to just calling writeln("asdf"); directly? (exact same 
code)

e.g., it is the same as

void main() { A!().foo(); }

cause when I actually debug I see a function call to foo.

So, saying that templates are like cookie cutter doesn't prove 
anything.

If I have

void main() { }

the binary file is is 150kB. If I add import std.stdio; It jumps 
to 300kB.

So the linker and compiler are not removing all untouched code.
Aug 01 2013
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Aug 01, 2013 at 10:06:54PM +0200, JS wrote:
[...]

 Now are you telling me that
 
 template A()
 {
     void foo() { writeln("asdf"); }
 }
 void main()
 {
 	A!().foo();
 }
 
 does not create a function foo in the binary? That it is equivalent
 to just calling writeln("asdf"); directly? (exact same code)


[...]

I said that every instantiation of a template creates a copy of
everything inside. Therefore, A!().foo() will create a copy of A.foo()
in the binary.

The template itself has no binary representation, in the sense that if
you write:

	template A(int x) {
		void foo() { writeln(x); }
	}

there is nothing in the binary corresponding with the template A, or the
uninstantiated function foo. But if you instantiate A with some value of
x, then you will get a copy of A.foo for every value of x that you
instantiate the template with. So if you write:

	A!1.foo();
	A!2.foo();
	A!3.foo();

Then you will get 3 copies of foo() in your executable, one for each
value of x.


T

-- 
What do you get if you drop a piano down a mineshaft? A flat minor.
Aug 01 2013
parent reply "JS" <js.mdnq gmail.com> writes:
On Thursday, 1 August 2013 at 21:17:34 UTC, H. S. Teoh wrote:

 On Thu, Aug 01, 2013 at 10:06:54PM +0200, JS wrote:
 [...]

 Now are you telling me that
 
 template A()
 {
     void foo() { writeln("asdf"); }
 }
 void main()
 {
 	A!().foo();
 }
 
 does not create a function foo in the binary? That it is 
 equivalent
 to just calling writeln("asdf"); directly? (exact same code)


 [...]

 I said that every instantiation of a template creates a copy of
 everything inside. Therefore, A!().foo() will create a copy of 
 A.foo()
 in the binary.

 The template itself has no binary representation, in the sense 
 that if
 you write:

 	template A(int x) {
 		void foo() { writeln(x); }
 	}

 there is nothing in the binary corresponding with the template 
 A, or the
 uninstantiated function foo. But if you instantiate A with some 
 value of
 x, then you will get a copy of A.foo for every value of x that 
 you
 instantiate the template with. So if you write:

 	A!1.foo();
 	A!2.foo();
 	A!3.foo();

 Then you will get 3 copies of foo() in your executable, one for 
 each
 value of x.



yes, I understand that... now use a template for a string 
mixin!!!!!

template A()
{
     string A() { ... }
}

...

mixin(A());

IS A GOING TO BE IN THE BINARY?!?!?! Yes, I'm yelling... just to 
get the point across about the question I'm trying to get 
answered.

the function A is never used at runtime SO it should technically 
not be in the binary UNLESS dmd treats it as a normal template 
function then it will(but shouldn't)!

e.g.,

if the compiler smart enough to realize that A(); is different 
from mixin(A());

(one being compile time and the other not)
Aug 01 2013
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 1 August 2013 at 22:04:57 UTC, JS wrote:

 IS A GOING TO BE IN THE BINARY?!?!?!


Why don't you try it and see? Disassemble the object file or get 
your linker of choice to print out some info.
Aug 01 2013
parent "JS" <js.mdnq gmail.com> writes:
On Thursday, 1 August 2013 at 22:13:25 UTC, John Colvin wrote:

 On Thursday, 1 August 2013 at 22:04:57 UTC, JS wrote:

 IS A GOING TO BE IN THE BINARY?!?!?!


 Why don't you try it and see? Disassemble the object file or 
 get your linker of choice to print out some info.


I don't have any of the tools to do so at the moment. The point 
was to see if anyone actually knew this specifically or was just 
guessing on what should be... I guess I have my answer(e.g., find 
out yourself = I don't know).
Aug 01 2013
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 22:04:57 UTC, JS wrote:

 IS A GOING TO BE IN THE BINARY?!?!?! Yes, I'm yelling... just 
 to get the point across about the question I'm trying to get 
 answered.


HOW MANY TIMES I NEED TO ANSWER YOUR QUESTION FOR YOU TO NOTICE?

Yes.
Aug 01 2013
parent reply "JS" <js.mdnq gmail.com> writes:
On Thursday, 1 August 2013 at 22:26:06 UTC, Dicebot wrote:

 On Thursday, 1 August 2013 at 22:04:57 UTC, JS wrote:

 IS A GOING TO BE IN THE BINARY?!?!?! Yes, I'm yelling... just 
 to get the point across about the question I'm trying to get 
 answered.


 HOW MANY TIMES I NEED TO ANSWER YOUR QUESTION FOR YOU TO NOTICE?

 Yes.



UM, NO, BECAUSE YOU ARE NOT ANSWERING MY QUESTION!

I don't know why it is so damn confusing...

I didn't say templates, I SAID string mixins of templates.

I did a test

http://dpaste.dzfl.pl/d8e6ca2a

http://dpaste.dzfl.pl/a43b923c

What is weird, on my comp, the file sizes are exactly the same, 
searching for the string only shows up in the one with the 
template call.

So It seems the dmd does not add code for string mixins(even 
though the function is called, it is smart enough not to add it 
to the binary)... unless it is somehow encrypting the 
template(again, I would expect the file size to be smaller... 
unless it does some type of alignment/padding).
Aug 01 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 22:42:11 UTC, JS wrote:

 I don't know why it is so damn confusing...

 I didn't say templates, I SAID string mixins of templates.


Symbol emitting does not go that way. Object files are old school 
entities from C times. They have function symbol and type 
symbols. Templates, mixins - does not matter. If function is used 
- it goes to object file. If struct definition gets generated - 
same story. In the world of linker there are no things such as 
templates or anything like that.
Aug 01 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 22:59:14 UTC, Dicebot wrote:

 ...


To further clarify this, I have used your simplified example:

-------------
// test.d
string generator(int x)
{
     if (x > 0)
     {
         asm
         {
             mov EAX, 0xDEAD;
         }
     }
     return "return 42;";
}

int main()
{
     mixin(generator(0));
}
-------------
$ dmd -release -inline -O test.d
-------------
$ objdump -d test | grep dead
   417705:	b8 ad de 00 00       	mov    $0xdead,%eax
Aug 01 2013
parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 23:15:47 UTC, Dicebot wrote:

 On Thursday, 1 August 2013 at 22:59:14 UTC, Dicebot wrote:

 ...


 To further clarify this, I have used your simplified example:


And in case someone thinks template functions are treated any 
differently, "string generator(T)(T a)" acts exactly the same.
Aug 01 2013
prev sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Thursday, 1 August 2013 at 20:06:56 UTC, JS wrote:


 the binary file is is 150kB. If I add import std.stdio; It 
 jumps to 300kB.


This scares me, it seems to me that the unused import should not 
be compiled in. And I frequently hear people talk about the 
linker removing unused code, but this proves that it doesn't or 
at least not all.

I wonder if this is a matter of something that is possible but 
just not implemented properly(or at all, i don't know) in the 
linker, or if their is some reason that makes this not possible.
Aug 01 2013
parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 21:38:15 UTC, Tofu Ninja wrote:

 This scares me, it seems to me that the unused import should 
 not be compiled in. And I frequently hear people talk about the 
 linker removing unused code, but this proves that it doesn't or 
 at least not all.


It does not. People talk about this because it seems reasonable. 
But reality differs a lot. No need to ask - just go an check.
Aug 01 2013
prev sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 08/01/2013 10:45 AM, H. S. Teoh wrote:

 On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:

 On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:

 If I have a bunch of templates that are only used for code
 generation, are they removed in the binary(since they are not
 needed)?


 templates don't exist in binaries.


 Templates are like cookie molds, you use them to generate lots of
 (almost) identical cookies, but you never serve the mold to the
 customer. ;-)


 T


I think the question is whether the instantiations of such templates are 
removed. If the instance is used only to initialize an enum, the 
function shouldn't stay in the binary.

Even if the function remains, I think the linker takes care of removing 
unused functions.

Ali
Aug 01 2013
next sibling parent reply "JS" <js.mdnq gmail.com> writes:
On Thursday, 1 August 2013 at 17:53:38 UTC, Ali Çehreli wrote:

 On 08/01/2013 10:45 AM, H. S. Teoh wrote:

 On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:

 On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:

 If I have a bunch of templates that are only used for code
 generation, are they removed in the binary(since they are not
 needed)?


 templates don't exist in binaries.


 Templates are like cookie molds, you use them to generate lots 
 of
 (almost) identical cookies, but you never serve the mold to the
 customer. ;-)


 T


 I think the question is whether the instantiations of such 
 templates are removed. If the instance is used only to 
 initialize an enum, the function shouldn't stay in the binary.

 Even if the function remains, I think the linker takes care of 
 removing unused functions.

 Ali


I have a lot of code generation templates(only for code). I want 
to make sure they don't hang around in the binary(even strings 
they use) as there is no point and it is potentially 
dangerous(security issues).
Aug 01 2013
parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 1 August 2013 at 17:57:28 UTC, JS wrote:

 On Thursday, 1 August 2013 at 17:53:38 UTC, Ali Çehreli wrote:

 On 08/01/2013 10:45 AM, H. S. Teoh wrote:

 On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:

 On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:

 If I have a bunch of templates that are only used for code
 generation, are they removed in the binary(since they are 
 not
 needed)?


 templates don't exist in binaries.


 Templates are like cookie molds, you use them to generate 
 lots of
 (almost) identical cookies, but you never serve the mold to 
 the
 customer. ;-)


 T


 I think the question is whether the instantiations of such 
 templates are removed. If the instance is used only to 
 initialize an enum, the function shouldn't stay in the binary.

 Even if the function remains, I think the linker takes care of 
 removing unused functions.

 Ali


 I have a lot of code generation templates(only for code). I 
 want to make sure they don't hang around in the binary(even 
 strings they use) as there is no point and it is potentially 
 dangerous(security issues).


The linker will get rid of anything not needed. Marking things as 
package (or more restrictive) will make sure they don't stay in 
librarys when not used.

Ultimately, if you're worried about it, get the linker to print a 
list of all functions, or check the assembly.
Aug 01 2013
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 17:53:38 UTC, Ali Çehreli wrote:

 I think the question is whether the instantiations of such 
 templates are removed. If the instance is used only to 
 initialize an enum, the function shouldn't stay in the binary.

 Even if the function remains, I think the linker takes care of 
 removing unused functions.

 Ali


Unfortunately, you are wrong. No symbol optimization is done by 
DMD is linker can't figure this out on its own, it requires 
specific object file layout to do reference counting for symbols.

It is really easy to check - make template function with some 
anchor value inside (OXDEADBEEF) and compile it with this 
template used exclusively in constraints or CTFE. Then strip and 
objdump - you'll find anchors in resulting binary for every 
instantiation.
Aug 01 2013