www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Should binary sharing be done using Mixin Template?

reply Matthew Ong <ongbp yahoo.com> writes:
Hi D Developer/Walter Bright,

Coming from a VM environment.

Should D be able to do binary sharing when the same template is being 
used for different data type.

In java using template, the same LinkedList binary is shared for both 
String class type and also Integer class type.

LinkedList<String> list=new LinkedList<String>();
LinkedList<Integer> list=new LinkedList<Integer>();

// Can also apply for Account/Order/PO...
LinkedList<Account> list=new LinkedList<Account>();

But there is a single LinkedList Class File(single binary).

Perhaps that is possible via some sort of binary plumbing internal to 
the compiler? I believe the wrapper is just to ensure the Object type 
handling casting concern. I might be wrong.

Accoding to Jonathan Davis,

There is no linking involved in mixins. It's not shared.
This approach I believe allow the final output be smaller even and pushes the D to be closer even to the dynamic ability of VM but without the extra over head. Can someone really say why this is a bad bad idea for memory with some automated plumbing being done like in ActiveX/Com/DCOM. -- Matthew Ong email: ongbp yahoo.com
May 21 2011
next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sat, 21 May 2011 12:26:35 +0300, Matthew Ong <ongbp yahoo.com> wrote:

 Should D be able to do binary sharing when the same template is being  
 used for different data type.
A linker should be able to merge binary-identical objects in read-only segments, but this will cause ambiguity in stack traces (and possibly have other consequences I can't think of). -- Best regards, Vladimir mailto:vladimir thecybershadow.net
May 21 2011
prev sibling next sibling parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 21.05.2011 11:26, schrieb Matthew Ong:
 Hi D Developer/Walter Bright,

 Coming from a VM environment.

 Should D be able to do binary sharing when the same template is being
 used for different data type.

 In java using template, the same LinkedList binary is shared for both
 String class type and also Integer class type.

 LinkedList<String> list=new LinkedList<String>();
 LinkedList<Integer> list=new LinkedList<Integer>();

 // Can also apply for Account/Order/PO...
 LinkedList<Account> list=new LinkedList<Account>();

 But there is a single LinkedList Class File(single binary).
This is because Javas generics are gone when the code is compiled, i.e. List<Integer> und List<String> is the same type. I find them rather useless, you can't overload from Generics parameters: void foo(List<Integer> l) { ... } void foo(List<String> l) { ... } won't compile. Totally different from D or C++ where a new type is created that is actually specific to the template parameter.
 Perhaps that is possible via some sort of binary plumbing internal to
 the compiler? I believe the wrapper is just to ensure the Object type
 handling casting concern. I might be wrong.
In D you can have non class types as template parameters. And even with class types your code could handle different classes in a different way via "static if( is(T : MyType) ) { ... }" etc
 Accoding to Jonathan Davis,

  >There is no linking involved in mixins. It's not shared.

 This approach I believe allow the final output be smaller even and
 pushes the D to be closer even to the dynamic ability of VM but without
 the extra over head.

 Can someone really say why this is a bad bad idea for memory with some
 automated plumbing being done like in ActiveX/Com/DCOM.
May 21 2011
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On 2011-05-21 02:26, Matthew Ong wrote:
 Hi D Developer/Walter Bright,
 
 Coming from a VM environment.
 
 Should D be able to do binary sharing when the same template is being
 used for different data type.
 
 In java using template, the same LinkedList binary is shared for both
 String class type and also Integer class type.
 
 LinkedList<String> list=new LinkedList<String>();
 LinkedList<Integer> list=new LinkedList<Integer>();
 
 // Can also apply for Account/Order/PO...
 LinkedList<Account> list=new LinkedList<Account>();
 
 But there is a single LinkedList Class File(single binary).
 
 Perhaps that is possible via some sort of binary plumbing internal to
 the compiler? I believe the wrapper is just to ensure the Object type
 handling casting concern. I might be wrong.
 
 Accoding to Jonathan Davis,
 
  >There is no linking involved in mixins. It's not shared.
 
 This approach I believe allow the final output be smaller even and
 pushes the D to be closer even to the dynamic ability of VM but without
 the extra over head.
 
 Can someone really say why this is a bad bad idea for memory with some
 automated plumbing being done like in ActiveX/Com/DCOM.
If you want to have containers be shared, then just put Objects in them and cast them to the correct type when you get them out. That's what Java does. It's just that starting with Java 1.5, the generics do it for you. Templates are completely different from generics. Generics are compile-time artifacts only and have no effect on the final binary. Templates, on the other hand, are generated for each type that they're instantiated with. They are code generation, pure and simple. And since each and every instantiation could be very different depending on what was done with static if and other compile- time stuff which can affect code generation. So, if you had a LinkedList<Account> and a LinkedList<string>, their internal implementation could be completely different if the writer of LinkedList thought that it was appropriate to do so for whatever reason. In the general case, templates are not at all combinable. Even if two template instantiations generate effectively the same code, unless the're binary compatible, you can't possibly combine them. Now, as I understand it, gcc does do some template combining in C++ where it can, but that's an advanced optimization technique that not all C++ compilers do, and I don't believe that any D compilers do it at present. Templates and generics are completely different. They both have their advantages and disadvantages, but please don't try and make templates act like generics. It's just not going to work. They are inherently different regardless of how similar their syntax might be. - Jonathan M Davis
May 21 2011
next sibling parent reply "JimBob" <jim bob.com> writes:
"Jonathan M Davis" <jmdavisProg gmx.com> wrote in message 
news:mailman.306.1305975846.14074.digitalmars-d puremagic.com...
 Templates are completely different from generics. Generics are 
 compile-time
 artifacts only and have no effect on the final binary. Templates, on the 
 other
 hand, are generated for each type that they're instantiated with. They are
 code generation, pure and simple. And since each and every instantiation 
 could
 be very different depending on what was done with static if and other 
 compile-
 time stuff which can affect code generation. So, if you had a
 LinkedList<Account> and a LinkedList<string>, their internal 
 implementation
 could be completely different if the writer of LinkedList thought that it 
 was
 appropriate to do so for whatever reason.
Cant the compiler simply do a bitwise compare of all versions of a given template function/method, and remove any that have duplicate code? I mean actually compare the generated asm code.
May 22 2011
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 22 May 2011 17:12:17 +0300, JimBob <jim bob.com> wrote:

 Cant the compiler simply do a bitwise compare of all versions of a given
 template function/method, and remove any that have duplicate code? I mean
 actually compare the generated asm code.
That's what I suggested earlier, but I think this should be done by a linker, as it'll allow merging duplicate code objects in all modules. -- Best regards, Vladimir mailto:vladimir thecybershadow.net
May 22 2011
prev sibling parent Paulo Pinto <pjmlp progtools.org> writes:
On 21.05.2011 13:03, Jonathan M Davis wrote:
 Templates are completely different from generics. Generics are compile-time
 artifacts only and have no effect on the final binary. Templates, on the other
 hand, are generated for each type that they're instantiated with.
Only if you are speaking about Java. In Eiffel, Ada, Modula-3 Ocaml, Haskell, and .Net generics are also handled like C++ templates. They just don't allow the same type of metaprogramming tricks that C++ does, although Template Haskell does allow neat tricks as well. -- Paulo
May 24 2011
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Matthew Ong:

 Should D be able to do binary sharing when the same template is being 
 used for different data type.
There is no a single silver bullet to reduce template bloat and its negative effects. So this problem need to be faced from many different directions at the same time. One of the directions is the one you talk about. I am thinking about some class/struct/function annotation to ask the D compiler to use something closer to generics with auto boxing-unboxing for that class/function. It's useful in the spots where the full performance of templates is not needed (this happens in programs). Bye, bearophile
May 21 2011
parent reply so <so so.so> writes:
On Sat, 21 May 2011 15:49:16 +0300, bearophile <bearophileHUGS lycos.com>  
wrote:

 It's useful in the spots where the full performance of templates is not  
 needed (this happens in programs).
What kind of applications are we talking about? I can't imagine a situation where you sacrifice the performance (that you gain with templates) for a trivial issue like bloat, when we finally get shared libraries, it will be even more trivial.
May 21 2011
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
so:

 What kind of applications are we talking about?
Let's say all applications longer than few thousands lines of code.
 I can't imagine a situation where you sacrifice the performance (that you  
 gain with templates) for a trivial issue like bloat,
Experience with profilers and with dynamic languages shows well that in all programs longer than few thousand lines a certain (sometimes good) percentage of the code doesn't influence much the performance of the whole program. So for such code other factors like code succinctness, safety, maintainability, and even binary compactness become more important than performance.
 when we finally get shared libraries, it will be even more trivial.
We will see. But I don't believe a single solution will solve this problem. Bye, bearophile
May 21 2011
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 21.05.2011 23:03, bearophile wrote:
 so:

 What kind of applications are we talking about?
Let's say all applications longer than few thousands lines of code.
 I can't imagine a situation where you sacrifice the performance (that you
 gain with templates) for a trivial issue like bloat,
Experience with profilers and with dynamic languages shows well that in all programs longer than few thousand lines a certain (sometimes good) percentage of the code doesn't influence much the performance of the whole program. So for such code other factors like code succinctness, safety, maintainability, and even binary compactness become more important than performance.
 when we finally get shared libraries, it will be even more trivial.
We will see. But I don't believe a single solution will solve this problem.
I think one problem is mentality e.g. people do template things on wrong criteria. As a small example, some containers (like lists and unlike sets) do not care about "what's in the box". It's only size in bytes + alignment that really matters (and if there are indirections in data). Then it's just simple wrapper on top of it that does interpretation of bytes. So imagine that something like this: struct List(T) { Node!T head; // ... methods on List } struct Node(T) { T data; Node* next; } should really be more like: struct ListBase(T.sizeof) { NodeBase!(T.sizeof) head; // ... methods on ListBase } struct NodeBase(size_t size) { void[size] data=void; NodeBase* next; } List!T forwards to ListBase!(T.sizeof) almost no new code, 'front' and 'back' aside. Also we have our type checking back. All of the wrappers are doing very inlineable things like e.g. struct ListRange(T) { NodeBase!(T.sizeof)* current; property ref T front(){ return *cast(T*)(&current.data[0]); } property void popFront(){ current = current.next; } property bool empty(){ return !current; } }
 Bye,
 bearophile
-- Dmitry Olshansky
May 22 2011
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On 2011-05-21 09:13, so wrote:
 On Sat, 21 May 2011 15:49:16 +0300, bearophile <bearophileHUGS lycos.com>
 
 wrote:
 It's useful in the spots where the full performance of templates is not
 needed (this happens in programs).
What kind of applications are we talking about? I can't imagine a situation where you sacrifice the performance (that you gain with templates) for a trivial issue like bloat, when we finally get shared libraries, it will be even more trivial.
There are plenty of cases where programs don't care about effeciency much, so any gain you get from templates over generics doesn't really matter. There are also plenty of cases where it doesn't really matter that templates increase the size of the executable in comparison to generics. It depends on what you're doing. On honestly, I think that a lot of the arguments one way or the other stem from perception rather than reality. Some people are brought to believe that templates are bloated and that the way that generics do things is inherently better because of that. Others are brought to believe that generics are slow and that templates are better because of that. There is some truth to both sides, but unless you're creating a programming language and choosing whether you're going to implement templates or generics or if you're going to use that as a factor in choosing which programming langage that you're going to use, it doesn't really matter. Each language does it it's own way, and if you're using a particular language, you're pretty much stuck with how it does things. D happens to use templates, so if you're using D, you get templates not generics. Really, I think that the key thing that needs to be understood here is the key nature of templates and how that differs from generics. If you're using Java, you need to understand what generics are and how they work so that you can best understand how to use them. Trying to make them act like templates doesn't work. The same goes for D, except in reverse. You need to understand templates an how they work so that you can best use them. Templates are heavily used in Phobos, and anyone who doesn't understand what templates really are is going to be at a disadvantage. At their heart, templates are a tool for code generation. They allow you to create functions and types which use different types without having to type all of the different functions and types yourself, and they do it by generating code. This has far-reaching consequences. One of the coolest is eponymous templates (such as std.metastrings.Format). You couldn't do anything like eponymous templates with generics. The fact that templated stuff can be specialized on type and entirely change what its implementation is based on its arguments is also incredibly powerful. One instance of that that I find very cool is core.time.convert. It takes two strings indicating the units of time that you want to convert to and from and the value that you want to convert, and it is able to generate the exact function which converts those specific units. With full optimizations on, it probably becomes an inlined expression which is very short and exactly what is needed for converting the value that you gave it to the units that it should be in. You don't have to have a bunch of different functions for doing conversions, and the function that you do use doesn't even need any if statements. It's all done at compile time with static ifs, and the result should be quite inlinable. You can't do anything like that with generics. You can do it with templates because what they are is code generation, not a way to inserts casts for you. Ultimately, regardless of whether a programmer prefers generics or templates, when they use a language that uses them, they need to understand what they are and how they work so that they can best use them. Trying to pervert one into the other is only going to give you trouble. One would hope that the compiler writers would be able to optimize whichever the language uses in the best manner possible, and I'm sure that more can be done in that area in D, but you still need to understand the difference between generics and templates and why the way that they are if you want to use them efficiently and appropriately. - Jonathan M Davis
May 21 2011
parent reply Matthew Ong <ongbp yahoo.com> writes:
On 5/22/2011 9:17 AM, Jonathan M Davis wrote:
 On 2011-05-21 09:13, so wrote:
 On Sat, 21 May 2011 15:49:16 +0300, bearophile<bearophileHUGS lycos.com>

 wrote:
 It's useful in the spots where the full performance of templates is not
 needed (this happens in programs).
What kind of applications are we talking about? I can't imagine a situation where you sacrifice the performance (that you gain with templates) for a trivial issue like bloat, when we finally get shared libraries, it will be even more trivial.
There are plenty of cases where programs don't care about effeciency much, so any gain you get from templates over generics doesn't really matter. There are also plenty of cases where it doesn't really matter that templates increase the size of the executable in comparison to generics. It depends on what you're doing. On honestly, I think that a lot of the arguments one way or the other stem from perception rather than reality. Some people are brought to believe that templates are bloated and that the way that generics do things is inherently better because of that. Others are brought to believe that generics are slow and that templates are better because of that. There is some truth to both sides, but unless you're creating a programming language and choosing whether you're going to implement templates or generics or if you're going to use that as a factor in choosing which programming langage that you're going to use, it doesn't really matter. Each language does it it's own way, and if you're using a particular language, you're pretty much stuck with how it does things. D happens to use templates, so if you're using D, you get templates not generics. Really, I think that the key thing that needs to be understood here is the key nature of templates and how that differs from generics. If you're using Java, you need to understand what generics are and how they work so that you can best understand how to use them. Trying to make them act like templates doesn't work. The same goes for D, except in reverse. You need to understand templates an how they work so that you can best use them. Templates are heavily used in Phobos, and anyone who doesn't understand what templates really are is going to be at a disadvantage. At their heart, templates are a tool for code generation. They allow you to create functions and types which use different types without having to type all of the different functions and types yourself, and they do it by generating code. This has far-reaching consequences. One of the coolest is eponymous templates (such as std.metastrings.Format). You couldn't do anything like eponymous templates with generics. The fact that templated stuff can be specialized on type and entirely change what its implementation is based on its arguments is also incredibly powerful. One instance of that that I find very cool is core.time.convert. It takes two strings indicating the units of time that you want to convert to and from and the value that you want to convert, and it is able to generate the exact function which converts those specific units. With full optimizations on, it probably becomes an inlined expression which is very short and exactly what is needed for converting the value that you gave it to the units that it should be in. You don't have to have a bunch of different functions for doing conversions, and the function that you do use doesn't even need any if statements. It's all done at compile time with static ifs, and the result should be quite inlinable. You can't do anything like that with generics. You can do it with templates because what they are is code generation, not a way to inserts casts for you. Ultimately, regardless of whether a programmer prefers generics or templates, when they use a language that uses them, they need to understand what they are and how they work so that they can best use them. Trying to pervert one into the other is only going to give you trouble. One would hope that the compiler writers would be able to optimize whichever the language uses in the best manner possible, and I'm sure that more can be done in that area in D, but you still need to understand the difference between generics and templates and why the way that they are if you want to use them efficiently and appropriately. - Jonathan M Davis
Hi Jonathan, From reading the threads and discussion. I think that most people are not talking about if generics or template syntax is better than the other.
 At their heart, templates are a tool for code generation.
Noted.
Trying to make them act like templates doesn't work. The same goes for 
D, except in reverse. Not the case with using the example that was shown. Form what I can see in D, at the syntax level, template with mixin in D is far better than generics in Java. It also allow primitive type such as MyTemplate!(int) but Java does not only uses wrapper class such as MyTemplate<Integer> from the java.lang.Integer class. I the focus here is about the compiled output binary dll/exe. Could the D somehow generate shared when mixin created an instance for object type(since most object are refered by same sized pointer) and perhaps different binary for different sized(int/double) type. Perhaps as one suggestion is to optimized that during compilation and NOT during linking phase. As Walter mentioned that during his own presentation in this URL, just watch part of it this morning. http://video.google.com/videoplay?docid=-7073020265668105471 The compiler does has more information about typed being compiled like in the foreach loop compare to linker that has to do some guessing. Just an suggestion. -- Matthew Ong email: ongbp yahoo.com
May 23 2011
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On 2011-05-23 00:04, Matthew Ong wrote:
 On 5/22/2011 9:17 AM, Jonathan M Davis wrote:
 On 2011-05-21 09:13, so wrote:
 On Sat, 21 May 2011 15:49:16 +0300, bearophile<bearophileHUGS lycos.com>
 
 wrote:
 It's useful in the spots where the full performance of templates is not
 needed (this happens in programs).
What kind of applications are we talking about? I can't imagine a situation where you sacrifice the performance (that you gain with templates) for a trivial issue like bloat, when we finally get shared libraries, it will be even more trivial.
There are plenty of cases where programs don't care about effeciency much, so any gain you get from templates over generics doesn't really matter. There are also plenty of cases where it doesn't really matter that templates increase the size of the executable in comparison to generics. It depends on what you're doing. On honestly, I think that a lot of the arguments one way or the other stem from perception rather than reality. Some people are brought to believe that templates are bloated and that the way that generics do things is inherently better because of that. Others are brought to believe that generics are slow and that templates are better because of that. There is some truth to both sides, but unless you're creating a programming language and choosing whether you're going to implement templates or generics or if you're going to use that as a factor in choosing which programming langage that you're going to use, it doesn't really matter. Each language does it it's own way, and if you're using a particular language, you're pretty much stuck with how it does things. D happens to use templates, so if you're using D, you get templates not generics. Really, I think that the key thing that needs to be understood here is the key nature of templates and how that differs from generics. If you're using Java, you need to understand what generics are and how they work so that you can best understand how to use them. Trying to make them act like templates doesn't work. The same goes for D, except in reverse. You need to understand templates an how they work so that you can best use them. Templates are heavily used in Phobos, and anyone who doesn't understand what templates really are is going to be at a disadvantage. At their heart, templates are a tool for code generation. They allow you to create functions and types which use different types without having to type all of the different functions and types yourself, and they do it by generating code. This has far-reaching consequences. One of the coolest is eponymous templates (such as std.metastrings.Format). You couldn't do anything like eponymous templates with generics. The fact that templated stuff can be specialized on type and entirely change what its implementation is based on its arguments is also incredibly powerful. One instance of that that I find very cool is core.time.convert. It takes two strings indicating the units of time that you want to convert to and from and the value that you want to convert, and it is able to generate the exact function which converts those specific units. With full optimizations on, it probably becomes an inlined expression which is very short and exactly what is needed for converting the value that you gave it to the units that it should be in. You don't have to have a bunch of different functions for doing conversions, and the function that you do use doesn't even need any if statements. It's all done at compile time with static ifs, and the result should be quite inlinable. You can't do anything like that with generics. You can do it with templates because what they are is code generation, not a way to inserts casts for you. Ultimately, regardless of whether a programmer prefers generics or templates, when they use a language that uses them, they need to understand what they are and how they work so that they can best use them. Trying to pervert one into the other is only going to give you trouble. One would hope that the compiler writers would be able to optimize whichever the language uses in the best manner possible, and I'm sure that more can be done in that area in D, but you still need to understand the difference between generics and templates and why the way that they are if you want to use them efficiently and appropriately. - Jonathan M Davis
Hi Jonathan, From reading the threads and discussion. I think that most people are not talking about if generics or template syntax is better than the other. > At their heart, templates are a tool for code generation. Noted. >Trying to make them act like templates doesn't work. The same goes for D, except in reverse. Not the case with using the example that was shown. Form what I can see in D, at the syntax level, template with mixin in D is far better than generics in Java. It also allow primitive type such as MyTemplate!(int) but Java does not only uses wrapper class such as MyTemplate<Integer> from the java.lang.Integer class. I the focus here is about the compiled output binary dll/exe. Could the D somehow generate shared when mixin created an instance for object type(since most object are refered by same sized pointer) and perhaps different binary for different sized(int/double) type. Perhaps as one suggestion is to optimized that during compilation and NOT during linking phase. As Walter mentioned that during his own presentation in this URL, just watch part of it this morning. http://video.google.com/videoplay?docid=-7073020265668105471 The compiler does has more information about typed being compiled like in the foreach loop compare to linker that has to do some guessing. Just an suggestion.
It is possible for the compiler and linker to further optimize templates and reduce the size of the resulting binary. But it's on a long list of possible optimizations which D could use. It may get them one day, but it's not going to get them any time soon. I believe that gcc only got that ability for C++ within the last few years. But regardless, it's an optimization detail. It has nothing to do with how you write code, unless you need a really small binary for some reason. D is _not_ going to change how templates work. At most, the compiler will be improved to better optimize them. But it won't happen any time soon. Personally, I think that if you're worrying much about this, you're either in a really abnormal situation, or you're worrying a lot more than you should. - Jonathan M Davis
May 23 2011