www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Re: Should binary sharing be done using Mixin Template?

reply bearophile <bearophileHUGS lycos.com> writes:
Matthew Ong:


 Should D be able to do binary sharing when the same template is being 
 used for different data type.



There is no a single silver bullet to reduce template bloat and its negative
effects. So this problem need to be faced from many different directions at the
same time. One of the directions is the one you talk about. I am thinking about
some class/struct/function annotation to ask the D compiler to use something
closer to generics with auto boxing-unboxing for that class/function. It's
useful in the spots where the full performance of templates is not needed (this
happens in programs).

Bye,
bearophile
May 21 2011
next sibling parent reply so <so so.so> writes:
On Sat, 21 May 2011 15:49:16 +0300, bearophile <bearophileHUGS lycos.com>  
wrote:


 It's useful in the spots where the full performance of templates is not  
 needed (this happens in programs).



What kind of applications are we talking about?
I can't imagine a situation where you sacrifice the performance (that you  
gain with templates) for a trivial issue like bloat,
when we finally get shared libraries, it will be even more trivial.
May 21 2011
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
so:


 What kind of applications are we talking about?



Let's say all applications longer than few thousands lines of code.



 I can't imagine a situation where you sacrifice the performance (that you  
 gain with templates) for a trivial issue like bloat,



Experience with profilers and with dynamic languages shows well that in all
programs longer than few thousand lines a certain (sometimes good) percentage
of the code doesn't influence much the performance of the whole program. So for
such code other factors like code succinctness, safety, maintainability, and
even binary compactness become more important than performance.



 when we finally get shared libraries, it will be even more trivial.



We will see. But I don't believe a single solution will solve this problem.

Bye,
bearophile
May 21 2011
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 21.05.2011 23:03, bearophile wrote:

 so:


 What kind of applications are we talking about?






 I can't imagine a situation where you sacrifice the performance (that you
 gain with templates) for a trivial issue like bloat,






 when we finally get shared libraries, it will be even more trivial.






I think one problem is mentality e.g. people do template things on wrong 
criteria.
As a small example, some containers (like lists and unlike sets) do not 
care about "what's in the box". It's only size in bytes + alignment that 
really matters (and if there are indirections in data). Then it's just 
simple wrapper on top of it that does interpretation of bytes.
So imagine that something like this:
struct List(T)
{
     Node!T head;
//    ... methods on List
}
struct Node(T)
{
     T data;
     Node* next;
}

should really be more like:

struct ListBase(T.sizeof)
{
     NodeBase!(T.sizeof) head;
//    ... methods on ListBase
}
struct NodeBase(size_t size)
{
     void[size] data=void;
     NodeBase* next;
}

List!T forwards to ListBase!(T.sizeof) almost no new code, 'front' and 
'back' aside. Also we have our type checking back.
All of the wrappers are doing very inlineable things like e.g.
struct ListRange(T)
{
     NodeBase!(T.sizeof)* current;
      property ref T front(){ return *cast(T*)(&current.data[0]); }
      property void popFront(){ current = current.next; }
      property bool empty(){ return !current; }
}


 Bye,
 bearophile




-- 
Dmitry Olshansky
May 22 2011
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On 2011-05-21 09:13, so wrote:

 On Sat, 21 May 2011 15:49:16 +0300, bearophile <bearophileHUGS lycos.com>
 
 wrote:

 It's useful in the spots where the full performance of templates is not
 needed (this happens in programs).



 What kind of applications are we talking about?
 I can't imagine a situation where you sacrifice the performance (that you
 gain with templates) for a trivial issue like bloat,
 when we finally get shared libraries, it will be even more trivial.



There are plenty of cases where programs don't care about effeciency much, so 
any gain you get from templates over generics doesn't really matter. There are 
also plenty of cases where it doesn't really matter that templates increase 
the size of the executable in comparison to generics. It depends on what 
you're doing. On honestly, I think that a lot of the arguments one way or the 
other stem from perception rather than reality. Some people are brought to 
believe that templates are bloated and that the way that generics do things is 
inherently better because of that. Others are brought to believe that generics 
are slow and that templates are better because of that. There is some truth to 
both sides, but unless you're creating a programming language and choosing 
whether you're going to implement templates or generics or if you're going to 
use that as a factor in choosing which programming langage that you're going 
to use, it doesn't really matter. Each language does it it's own way, and if 
you're using a particular language, you're pretty much stuck with how it does 
things. D happens to use templates, so if you're using D, you get templates 
not generics.

Really, I think that the key thing that needs to be understood here is the key 
nature of templates and how that differs from generics. If you're using Java, 
you need to understand what generics are and how they work so that you can 
best understand how to use them. Trying to make them act like templates 
doesn't work. The same goes for D, except in reverse. You need to understand 
templates an how they work so that you can best use them. Templates are 
heavily used in Phobos, and anyone who doesn't understand what templates 
really are is going to be at a disadvantage.

At their heart, templates are a tool for code generation. They allow you to 
create functions and types which use different types without having to type 
all of the different functions and types yourself, and they do it by 
generating code. This has far-reaching consequences. One of the coolest is 
eponymous templates (such as std.metastrings.Format). You couldn't do anything 
like eponymous templates with generics.

The fact that templated stuff can be specialized on type and entirely change 
what its implementation is based on its arguments is also incredibly powerful. 
One instance of that that I find very cool is core.time.convert. It takes two 
strings indicating the units of time that you want to convert to and from and 
the value that you want to convert, and it is able to generate the exact 
function which converts those specific units. With full optimizations on, it 
probably becomes an inlined expression which is very short and exactly what is 
needed for converting the value that you gave it to the units that it should 
be in. You don't have to have a bunch of different functions for doing 
conversions, and the function that you do use doesn't even need any if 
statements. It's all done at compile time with static ifs, and the result 
should be quite inlinable. You can't do anything like that with generics. You 
can do it with templates because what they are is code generation, not a way 
to inserts casts for you.

Ultimately, regardless of whether a programmer prefers generics or templates, 
when they use a language that uses them, they need to understand what they are 
and how they work so that they can best use them. Trying to pervert one into 
the other is only going to give you trouble. One would hope that the compiler 
writers would be able to optimize whichever the language uses in the best 
manner possible, and I'm sure that more can be done in that area in D, but you 
still need to understand the difference between generics and templates and why 
the way that they are if you want to use them efficiently and appropriately.

- Jonathan M Davis
May 21 2011
parent Matthew Ong <ongbp yahoo.com> writes:
On 5/22/2011 9:17 AM, Jonathan M Davis wrote:

 On 2011-05-21 09:13, so wrote:

 On Sat, 21 May 2011 15:49:16 +0300, bearophile<bearophileHUGS lycos.com>

 wrote:

 It's useful in the spots where the full performance of templates is not
 needed (this happens in programs).



 What kind of applications are we talking about?
 I can't imagine a situation where you sacrifice the performance (that you
 gain with templates) for a trivial issue like bloat,
 when we finally get shared libraries, it will be even more trivial.



 There are plenty of cases where programs don't care about effeciency much, so
 any gain you get from templates over generics doesn't really matter. There are
 also plenty of cases where it doesn't really matter that templates increase
 the size of the executable in comparison to generics. It depends on what
 you're doing. On honestly, I think that a lot of the arguments one way or the
 other stem from perception rather than reality. Some people are brought to
 believe that templates are bloated and that the way that generics do things is
 inherently better because of that. Others are brought to believe that generics
 are slow and that templates are better because of that. There is some truth to
 both sides, but unless you're creating a programming language and choosing
 whether you're going to implement templates or generics or if you're going to
 use that as a factor in choosing which programming langage that you're going
 to use, it doesn't really matter. Each language does it it's own way, and if
 you're using a particular language, you're pretty much stuck with how it does
 things. D happens to use templates, so if you're using D, you get templates
 not generics.

 Really, I think that the key thing that needs to be understood here is the key
 nature of templates and how that differs from generics. If you're using Java,
 you need to understand what generics are and how they work so that you can
 best understand how to use them. Trying to make them act like templates
 doesn't work. The same goes for D, except in reverse. You need to understand
 templates an how they work so that you can best use them. Templates are
 heavily used in Phobos, and anyone who doesn't understand what templates
 really are is going to be at a disadvantage.

 At their heart, templates are a tool for code generation. They allow you to
 create functions and types which use different types without having to type
 all of the different functions and types yourself, and they do it by
 generating code. This has far-reaching consequences. One of the coolest is
 eponymous templates (such as std.metastrings.Format). You couldn't do anything
 like eponymous templates with generics.

 The fact that templated stuff can be specialized on type and entirely change
 what its implementation is based on its arguments is also incredibly powerful.
 One instance of that that I find very cool is core.time.convert. It takes two
 strings indicating the units of time that you want to convert to and from and
 the value that you want to convert, and it is able to generate the exact
 function which converts those specific units. With full optimizations on, it
 probably becomes an inlined expression which is very short and exactly what is
 needed for converting the value that you gave it to the units that it should
 be in. You don't have to have a bunch of different functions for doing
 conversions, and the function that you do use doesn't even need any if
 statements. It's all done at compile time with static ifs, and the result
 should be quite inlinable. You can't do anything like that with generics. You
 can do it with templates because what they are is code generation, not a way
 to inserts casts for you.

 Ultimately, regardless of whether a programmer prefers generics or templates,
 when they use a language that uses them, they need to understand what they are
 and how they work so that they can best use them. Trying to pervert one into
 the other is only going to give you trouble. One would hope that the compiler
 writers would be able to optimize whichever the language uses in the best
 manner possible, and I'm sure that more can be done in that area in D, but you
 still need to understand the difference between generics and templates and why
 the way that they are if you want to use them efficiently and appropriately.

 - Jonathan M Davis




 From reading the threads and discussion. I think that most people are 
not talking about if generics or template syntax is better than the other.


 At their heart, templates are a tool for code generation.





Trying to make them act like templates doesn't work. The same goes for 



Not the case with using the example that was shown.

Form what I can see in D, at the syntax level, template with mixin in D 
is far better than generics in Java. It also allow primitive type such 
as MyTemplate!(int) but Java does not only uses wrapper class such as 
MyTemplate<Integer> from the java.lang.Integer class.

I the focus here is about the compiled output binary dll/exe. Could the 
D somehow generate shared when mixin created an instance for object 
type(since most object are refered by same sized pointer) and perhaps 
different binary for different sized(int/double) type.

Perhaps as one suggestion is to optimized that during compilation and 
NOT during linking phase. As Walter mentioned that during his own 
presentation in this URL, just watch part of it this morning.
http://video.google.com/videoplay?docid=-7073020265668105471

The compiler does has more information about typed being compiled like 
in the foreach loop compare to linker that has to do some guessing.

Just an suggestion.

-- 
Matthew Ong
email: ongbp yahoo.com
May 23 2011
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On 2011-05-23 00:04, Matthew Ong wrote:

 On 5/22/2011 9:17 AM, Jonathan M Davis wrote:

 On 2011-05-21 09:13, so wrote:

 On Sat, 21 May 2011 15:49:16 +0300, bearophile<bearophileHUGS lycos.com>
 
 wrote:

 It's useful in the spots where the full performance of templates is not
 needed (this happens in programs).



 What kind of applications are we talking about?
 I can't imagine a situation where you sacrifice the performance (that
 you gain with templates) for a trivial issue like bloat,
 when we finally get shared libraries, it will be even more trivial.



 There are plenty of cases where programs don't care about effeciency
 much, so any gain you get from templates over generics doesn't really
 matter. There are also plenty of cases where it doesn't really matter
 that templates increase the size of the executable in comparison to
 generics. It depends on what you're doing. On honestly, I think that a
 lot of the arguments one way or the other stem from perception rather
 than reality. Some people are brought to believe that templates are
 bloated and that the way that generics do things is inherently better
 because of that. Others are brought to believe that generics are slow
 and that templates are better because of that. There is some truth to
 both sides, but unless you're creating a programming language and
 choosing whether you're going to implement templates or generics or if
 you're going to use that as a factor in choosing which programming
 langage that you're going to use, it doesn't really matter. Each
 language does it it's own way, and if you're using a particular
 language, you're pretty much stuck with how it does things. D happens to
 use templates, so if you're using D, you get templates not generics.
 
 Really, I think that the key thing that needs to be understood here is
 the key nature of templates and how that differs from generics. If
 you're using Java, you need to understand what generics are and how they
 work so that you can best understand how to use them. Trying to make
 them act like templates doesn't work. The same goes for D, except in
 reverse. You need to understand templates an how they work so that you
 can best use them. Templates are heavily used in Phobos, and anyone who
 doesn't understand what templates really are is going to be at a
 disadvantage.
 
 At their heart, templates are a tool for code generation. They allow you
 to create functions and types which use different types without having
 to type all of the different functions and types yourself, and they do
 it by generating code. This has far-reaching consequences. One of the
 coolest is eponymous templates (such as std.metastrings.Format). You
 couldn't do anything like eponymous templates with generics.
 
 The fact that templated stuff can be specialized on type and entirely
 change what its implementation is based on its arguments is also
 incredibly powerful. One instance of that that I find very cool is
 core.time.convert. It takes two strings indicating the units of time
 that you want to convert to and from and the value that you want to
 convert, and it is able to generate the exact function which converts
 those specific units. With full optimizations on, it probably becomes an
 inlined expression which is very short and exactly what is needed for
 converting the value that you gave it to the units that it should be in.
 You don't have to have a bunch of different functions for doing
 conversions, and the function that you do use doesn't even need any if
 statements. It's all done at compile time with static ifs, and the
 result should be quite inlinable. You can't do anything like that with
 generics. You can do it with templates because what they are is code
 generation, not a way to inserts casts for you.
 
 Ultimately, regardless of whether a programmer prefers generics or
 templates, when they use a language that uses them, they need to
 understand what they are and how they work so that they can best use
 them. Trying to pervert one into the other is only going to give you
 trouble. One would hope that the compiler writers would be able to
 optimize whichever the language uses in the best manner possible, and
 I'm sure that more can be done in that area in D, but you still need to
 understand the difference between generics and templates and why the way
 that they are if you want to use them efficiently and appropriately.
 
 - Jonathan M Davis



 Hi Jonathan,
 
  From reading the threads and discussion. I think that most people are
 not talking about if generics or template syntax is better than the other.
 
  > At their heart, templates are a tool for code generation.
 
 Noted.
 
  >Trying to make them act like templates doesn't work. The same goes for
 
 D, except in reverse.
 Not the case with using the example that was shown.
 
 Form what I can see in D, at the syntax level, template with mixin in D
 is far better than generics in Java. It also allow primitive type such
 as MyTemplate!(int) but Java does not only uses wrapper class such as
 MyTemplate<Integer> from the java.lang.Integer class.
 
 I the focus here is about the compiled output binary dll/exe. Could the
 D somehow generate shared when mixin created an instance for object
 type(since most object are refered by same sized pointer) and perhaps
 different binary for different sized(int/double) type.
 
 Perhaps as one suggestion is to optimized that during compilation and
 NOT during linking phase. As Walter mentioned that during his own
 presentation in this URL, just watch part of it this morning.
 http://video.google.com/videoplay?docid=-7073020265668105471
 
 The compiler does has more information about typed being compiled like
 in the foreach loop compare to linker that has to do some guessing.
 
 Just an suggestion.



It is possible for the compiler and linker to further optimize templates and 
reduce the size of the resulting binary. But it's on a long list of possible 
optimizations which D could use. It may get them one day, but it's not going 
to get them any time soon. I believe that gcc only got that ability for C++ 
within the last few years. But regardless, it's an optimization detail. It has 
nothing to do with how you write code, unless you need a really small binary 
for some reason. D is _not_ going to change how templates work. At most, the 
compiler will be improved to better optimize them. But it won't happen any 
time soon. Personally, I think that if you're worrying much about this, you're 
either in a really abnormal situation, or you're worrying a lot more than you 
should.

- Jonathan M Davis
May 23 2011