www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - Identifier (...) is too long by X characters

reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
I just got such an error while trying to instantiate a rather longish 
template. This is pretty much a blocker for more complex template code. 
Couldn't symbol names be truncated at some predefined length ?

It also seems like long symbols are killing DMD performance-wise as they 
can eat lots of RAM while compiling. Then mem allocations + cache misses 
+ GC stalls make it crawl. I've tried instantiating a struct template 
(the Tuple from my 'bind' lib) with a nesting level of 50 and before I 
managed to kill DMD, it ate 700MB of virtual mem out of my 512MB RAM system.

Can anything be done with it ? It seems like compile times and memory 
use explode when dealing with more complex templates.


-- 
Tomasz Stachowiak  /+ a.k.a. h3r3tic +/
Jul 08 2006
parent reply John Reimer <John_member pathlink.com> writes:
In article <e8pf88$22ki$1 digitaldaemon.com>, Tom S says...

I just got such an error while trying to instantiate a rather longish 
template. This is pretty much a blocker for more complex template code. 
Couldn't symbol names be truncated at some predefined length ?

It also seems like long symbols are killing DMD performance-wise as they 
can eat lots of RAM while compiling. Then mem allocations + cache misses 
+ GC stalls make it crawl. I've tried instantiating a struct template 
(the Tuple from my 'bind' lib) with a nesting level of 50 and before I 
managed to kill DMD, it ate 700MB of virtual mem out of my 512MB RAM system.

Can anything be done with it ? It seems like compile times and memory 
use explode when dealing with more complex templates.


-- 
Tomasz Stachowiak  /+ a.k.a. h3r3tic +/



Tsk, tsk, tsk... what mad template experiments are you up to now, Tom?  Your
computer must be smoking at this point. lol!

-JJR
Jul 09 2006
parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
John Reimer wrote:

 Tsk, tsk, tsk... what mad template experiments are you up to now, Tom?  Your
 computer must be smoking at this point. lol!


You know... the usual stuff ;D
I'm just working with a *bit* more complex mixins, like this one:

	mixin Input!(`in`)
		.field!(DataStream!(vec3), `positions`)
		.field!(DataStream!(vec3), `normals`)
	.end input;

When I add too many fields, DMD eats many many rams and gives me that error.


-- 
Tomasz Stachowiak  /+ a.k.a. h3r3tic +/
Jul 09 2006
parent reply Chris Nicholson-Sauls <ibisbasenji gmail.com> writes:
Tom S wrote:

 ...DMD eats many many rams...


*Gasp!*  Those poor sheep/goats!

I would be interested in seeing exactly what your 'Input' mixin does.  The use
of 
self-referancing returns to initialize it is pretty nifty though.  Might be a
way to get 
rid of the '.end' at the end, though?

The error baffles me, though.

-- Chris Nicholson-Sauls
Jul 09 2006
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Chris Nicholson-Sauls wrote:

 Tom S wrote:

 ...DMD eats many many rams...


 
 *Gasp!*  Those poor sheep/goats!
 
 I would be interested in seeing exactly what your 'Input' mixin does.  
 The use of self-referancing returns to initialize it is pretty nifty 
 though.  Might be a way to get rid of the '.end' at the end, though?
 
 The error baffles me, though.


Symbols are limited to ~255 chars on Win32.  I think it's a limitation 
of the object file format.  "symbol too long" errors are painfully 
common in C++, though they're typically truncated to the max length to 
allow for debugging.


Sean
Jul 09 2006
parent reply Don Clugston <dac nospam.com.au> writes:
Sean Kelly wrote:

 Chris Nicholson-Sauls wrote:

 Tom S wrote:

 ...DMD eats many many rams...


 *Gasp!*  Those poor sheep/goats!

 I would be interested in seeing exactly what your 'Input' mixin does.  
 The use of self-referancing returns to initialize it is pretty nifty 
 though.  Might be a way to get rid of the '.end' at the end, though?

 The error baffles me, though.


 
 Symbols are limited to ~255 chars on Win32.  I think it's a limitation 
 of the object file format.  "symbol too long" errors are painfully 
 common in C++, though they're typically truncated to the max length to 
 allow for debugging.


I thought the OMF lib-imposed limit was about 4K. I've just made a 
template with a symbol name of 480 characters, and that was OK.
If Tuple!() works with a nesting level of 50, the limit would have to be 
pretty big.
Maybe in the future, we'll be able to add some kind of 'final' 
declaration to templates, which would indicate that they don't need to 
be stored in the object file or library, and can be discarded from the 
symbol table as soon as they are instantiated. This would be 
particularly useful for recursive templates, which can easily spew reams 
of garbage into the object file.
Jul 09 2006
next sibling parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Don Clugston wrote:

 If Tuple!() works with a nesting level of 50, the limit would have to be 
 pretty big.


Yeah, and it would cause DMD to eat RAMs like a medium-sized dragon.



 Maybe in the future, we'll be able to add some kind of 'final' 
 declaration to templates, which would indicate that they don't need to 
 be stored in the object file or library, and can be discarded from the 
 symbol table as soon as they are instantiated. This would be 
 particularly useful for recursive templates, which can easily spew reams 
 of garbage into the object file.


My thoughts exactly :) Right now I'd be happy with anything that worked 
though...


And while we're at recursive templates, don't you think that the 
limitation for recursive templates, which causes the error 'template 
instance recursive template expansion for template argument X' is pretty 
arbitary ? It can be hacked around, after all:


// tmp.d
import std.stdio;


struct Foo(T/*, int dummy=0*/) {
	Foo!(Foo!(T/*, dummy+1*/)) rec()() {    // line 5
		Foo!(Foo!(T/*, dummy+1*/)) res;
		return res;
	}
}


void main() {
	Foo!(int) a;
	writefln(typeid(typeof(a)));
	auto b = a.rec!()();
	writefln(typeid(typeof(b)));
}
// ----

tmp.d(5): template instance recursive template expansion for template 
argument Foo

But when you uncomment the 'dummies', it compiles and runs without any 
problems...

Yeah, I know, the error is an artifact of how the instantiation works, 
that it's not 'lazy' like in C++. But since it can be fooled into 
working, couldn't DMD apply the hack automagically ?


-- 
Tomasz Stachowiak  /+ a.k.a. h3r3tic +/
Jul 10 2006
parent reply Sean Kelly <sean f4.ca> writes:
Tom S wrote:

 Don Clugston wrote:

 If Tuple!() works with a nesting level of 50, the limit would have to 
 be pretty big.


 
 Yeah, and it would cause DMD to eat RAMs like a medium-sized dragon.
 
 

 Maybe in the future, we'll be able to add some kind of 'final' 
 declaration to templates, which would indicate that they don't need to 
 be stored in the object file or library, and can be discarded from the 
 symbol table as soon as they are instantiated. This would be 
 particularly useful for recursive templates, which can easily spew 
 reams of garbage into the object file.


 
 My thoughts exactly :) Right now I'd be happy with anything that worked 
 though...
 
 
 And while we're at recursive templates, don't you think that the 
 limitation for recursive templates, which causes the error 'template 
 instance recursive template expansion for template argument X' is pretty 
 arbitary ? It can be hacked around, after all:
 
 
 // tmp.d
 import std.stdio;
 
 
 struct Foo(T/*, int dummy=0*/) {
     Foo!(Foo!(T/*, dummy+1*/)) rec()() {    // line 5
         Foo!(Foo!(T/*, dummy+1*/)) res;
         return res;
     }
 }
 
 
 void main() {
     Foo!(int) a;
     writefln(typeid(typeof(a)));
     auto b = a.rec!()();
     writefln(typeid(typeof(b)));
 }
 // ----
 
 tmp.d(5): template instance recursive template expansion for template 
 argument Foo
 
 But when you uncomment the 'dummies', it compiles and runs without any 
 problems...
 
 Yeah, I know, the error is an artifact of how the instantiation works, 
 that it's not 'lazy' like in C++. But since it can be fooled into 
 working, couldn't DMD apply the hack automagically ?


I don't know.  To do so would be like allowing:

struct S
{
     S val;
}

I'm not sure I'd want the compiler to try and "fix" a mistake I'd made 
involving this.  Adding an integer is pretty easy anyway.


Sean
Jul 10 2006
parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Sean Kelly wrote:

 I don't know.  To do so would be like allowing:
 
 struct S
 {
     S val;
 }


Umm... not quite. In your case it's just a plain nested, recursive 
struct which is obviously impossible. In my case:

struct Foo(T) {
     Foo!(Foo!(T)) rec()() {
         Foo!(Foo!(T)) res;
         return res;
     }
}

When the function template is instantiated, one more nesting level is 
'added' and a non-recurring struct is returned. The compiler reports an 
error because somehow it doesn't instantiate the 'rec' template lazily. 
In the example you've given there's an infinite theoretical nesting, yet 
in the version with the function, infinite nesting isn't possible, 
because the nesting increases one level with each 'rec' instantiation. 
IIRC, that's perfectly legit in C++ as it uses a different method for 
template instantiation.



 I'm not sure I'd want the compiler to try and "fix" a mistake I'd made 
 involving this.  Adding an integer is pretty easy anyway.


Easy ? It's undocumented, tedious and generally weird. It might stop 
working in the next compiler version because I'm not even sure why DMD 
allows it.


-- 
Tomasz Stachowiak  /+ a.k.a. h3r3tic +/
Jul 10 2006
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Tom S wrote:

 Sean Kelly wrote:

 I don't know.  To do so would be like allowing:

 struct S
 {
     S val;
 }


 
 Umm... not quite. In your case it's just a plain nested, recursive 
 struct which is obviously impossible. In my case:
 
 struct Foo(T) {
     Foo!(Foo!(T)) rec()() {
         Foo!(Foo!(T)) res;
         return res;
     }
 }
 
 When the function template is instantiated, one more nesting level is 
 'added' and a non-recurring struct is returned.


Oops, you're right.  I should have looked at the example more closely.


 The compiler reports an
 error because somehow it doesn't instantiate the 'rec' template lazily. 
 In the example you've given there's an infinite theoretical nesting, yet 
 in the version with the function, infinite nesting isn't possible, 
 because the nesting increases one level with each 'rec' instantiation. 
 IIRC, that's perfectly legit in C++ as it uses a different method for 
 template instantiation.


Yeah.  I think the problem in D is that all template members are 
explicitly instantiated together instead of on an as-needed basis.  But 
as rec() is itself a template function I wouldn't have expected this 
behavior.  I agree that this should be changed, assuming it's possible 
to do so without entirely changing the way templates are instantiated in D.


 I'm not sure I'd want the compiler to try and "fix" a mistake I'd made 
 involving this.  Adding an integer is pretty easy anyway.


 
 Easy ? It's undocumented, tedious and generally weird. It might stop 
 working in the next compiler version because I'm not even sure why DMD 
 allows it.


You're right.  Sorry for the confusion.


Sean
Jul 10 2006
parent Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Sean Kelly wrote:

 Tom S wrote:

 Sean Kelly wrote:

 I don't know.  To do so would be like allowing:

 struct S
 {
     S val;
 }


 Umm... not quite. In your case it's just a plain nested, recursive 
 struct which is obviously impossible. In my case:

 struct Foo(T) {
     Foo!(Foo!(T)) rec()() {
         Foo!(Foo!(T)) res;
         return res;
     }
 }

 When the function template is instantiated, one more nesting level is 
 'added' and a non-recurring struct is returned.


 
 Oops, you're right.  I should have looked at the example more closely.


Ah, you haven't noticed the second pair of parentheses after 'rec' ?



 Yeah.  I think the problem in D is that all template members are 
 explicitly instantiated together instead of on an as-needed basis.  But 
 as rec() is itself a template function I wouldn't have expected this 
 behavior.  I agree that this should be changed, assuming it's possible 
 to do so without entirely changing the way templates are instantiated in D.


Thanks for support :)


-- 
Tomasz Stachowiak  /+ a.k.a. h3r3tic +/
Jul 10 2006
prev sibling parent Bruno Medeiros <brunodomedeirosATgmail SPAM.com> writes:
Tom S wrote:

 Sean Kelly wrote:

 I don't know.  To do so would be like allowing:

 struct S
 {
     S val;
 }


 
 Umm... not quite. In your case it's just a plain nested, recursive 
 struct which is obviously impossible. In my case:
 
 struct Foo(T) {
     Foo!(Foo!(T)) rec()() {
         Foo!(Foo!(T)) res;
         return res;
     }
 }
 
 When the function template is instantiated, one more nesting level is 
 'added' and a non-recurring struct is returned. The compiler reports an 
 error because somehow it doesn't instantiate the 'rec' template lazily. 
 In the example you've given there's an infinite theoretical nesting, yet 
 in the version with the function, infinite nesting isn't possible, 
 because the nesting increases one level with each 'rec' instantiation. 
 IIRC, that's perfectly legit in C++ as it uses a different method for 
 template instantiation.
 
 

 I'm not sure I'd want the compiler to try and "fix" a mistake I'd made 
 involving this.  Adding an integer is pretty easy anyway.


 
 Easy ? It's undocumented, tedious and generally weird. It might stop 
 working in the next compiler version because I'm not even sure why DMD 
 allows it.
 
 


The compiler does always evaluate "lazily", and that is documented, 
although it may not be immediately obvious:
The template spec states that "Semantic analysis is not done until 
instantiated." which in other words means a template is not evaluated 
until instantiated. This remains true for nested templates, and thus for 
your example as well.
Then why the error message? My guess is that the error message is simply 
a mistake on part of the compiler, where the recursive template 
detection system gives a false positive (it thinks it will expand 
indefinitely but it won't). Thus it is a bug.


-- 
Bruno Medeiros - CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Jul 12 2006
prev sibling next sibling parent reply Sean Kelly <sean f4.ca> writes:
Don Clugston wrote:

 Sean Kelly wrote:

 Chris Nicholson-Sauls wrote:

 Tom S wrote:

 ...DMD eats many many rams...


 *Gasp!*  Those poor sheep/goats!

 I would be interested in seeing exactly what your 'Input' mixin 
 does.  The use of self-referancing returns to initialize it is pretty 
 nifty though.  Might be a way to get rid of the '.end' at the end, 
 though?

 The error baffles me, though.


 Symbols are limited to ~255 chars on Win32.  I think it's a limitation 
 of the object file format.  "symbol too long" errors are painfully 
 common in C++, though they're typically truncated to the max length to 
 allow for debugging.


 
 I thought the OMF lib-imposed limit was about 4K. I've just made a 
 template with a symbol name of 480 characters, and that was OK.


Perhaps it is--it's been a while since I developed on Windows.  I only 
remember seeing the errors quite often.


 Maybe in the future, we'll be able to add some kind of 'final' 
 declaration to templates, which would indicate that they don't need to 
 be stored in the object file or library, and can be discarded from the 
 symbol table as soon as they are instantiated. This would be 
 particularly useful for recursive templates, which can easily spew reams 
 of garbage into the object file.


This would be nice.  I generally expect this to occur anyway, but it 
would be nice to have some sort of insurance.


Sean
Jul 10 2006
parent Don Clugston <dac nospam.com.au> writes:
Sean Kelly wrote:

 Don Clugston wrote:

 Sean Kelly wrote:

 Chris Nicholson-Sauls wrote:

 Tom S wrote:

 ...DMD eats many many rams...


 *Gasp!*  Those poor sheep/goats!

 I would be interested in seeing exactly what your 'Input' mixin 
 does.  The use of self-referancing returns to initialize it is 
 pretty nifty though.  Might be a way to get rid of the '.end' at the 
 end, though?

 The error baffles me, though.


 Symbols are limited to ~255 chars on Win32.  I think it's a 
 limitation of the object file format.  "symbol too long" errors are 
 painfully common in C++, though they're typically truncated to the 
 max length to allow for debugging.


 I thought the OMF lib-imposed limit was about 4K. I've just made a 
 template with a symbol name of 480 characters, and that was OK.


 
 Perhaps it is--it's been a while since I developed on Windows.  I only 
 remember seeing the errors quite often.
 

 Maybe in the future, we'll be able to add some kind of 'final' 
 declaration to templates, which would indicate that they don't need to 
 be stored in the object file or library, and can be discarded from the 
 symbol table as soon as they are instantiated. This would be 
 particularly useful for recursive templates, which can easily spew 
 reams of garbage into the object file.


 
 This would be nice.  I generally expect this to occur anyway, but it 
 would be nice to have some sort of insurance.


It definitely happens, but I don't understand why they're in the obj at 
all (if I understand correctly, it has to happen that way in C++ because 
it's possible for a seperate obj file to provide a specialisation; but 
that's not an issue for D).

It's a real problem for the use of metaprogramming with DDL. The .objs 
can a be a megabyte in size, but contribute only a hundred bytes to the 
final exe. It would make linking much faster if it didn't happen, and it 
would probably improve compilation time.
Jul 10 2006
prev sibling parent reply Bruno Medeiros <brunodomedeirosATgmail SPAM.com> writes:
Don Clugston wrote:

 Sean Kelly wrote:

 Chris Nicholson-Sauls wrote:

 Tom S wrote:

 ...DMD eats many many rams...


 *Gasp!*  Those poor sheep/goats!

 I would be interested in seeing exactly what your 'Input' mixin 
 does.  The use of self-referancing returns to initialize it is pretty 
 nifty though.  Might be a way to get rid of the '.end' at the end, 
 though?

 The error baffles me, though.


 Symbols are limited to ~255 chars on Win32.  I think it's a limitation 
 of the object file format.  "symbol too long" errors are painfully 
 common in C++, though they're typically truncated to the max length to 
 allow for debugging.


 
 I thought the OMF lib-imposed limit was about 4K. I've just made a 
 template with a symbol name of 480 characters, and that was OK.
 If Tuple!() works with a nesting level of 50, the limit would have to be 
 pretty big.
 Maybe in the future, we'll be able to add some kind of 'final' 
 declaration to templates, which would indicate that they don't need to 
 be stored in the object file or library, and can be discarded from the 
 symbol table as soon as they are instantiated. This would be 
 particularly useful for recursive templates, which can easily spew reams 
 of garbage into the object file.


What are these templates that can be discarded from the object file? Is 
it templates that only contain compile-time members? Do these even take 
space in an object file (since they have no run-time members) ?


-- 
Bruno Medeiros - CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Jul 14 2006
parent reply Don Clugston <dac nospam.com.au> writes:
Bruno Medeiros wrote:

 Don Clugston wrote:

 Sean Kelly wrote:

 Chris Nicholson-Sauls wrote:

 Tom S wrote:

 ...DMD eats many many rams...


 *Gasp!*  Those poor sheep/goats!

 I would be interested in seeing exactly what your 'Input' mixin 
 does.  The use of self-referancing returns to initialize it is 
 pretty nifty though.  Might be a way to get rid of the '.end' at the 
 end, though?

 The error baffles me, though.


 Symbols are limited to ~255 chars on Win32.  I think it's a 
 limitation of the object file format.  "symbol too long" errors are 
 painfully common in C++, though they're typically truncated to the 
 max length to allow for debugging.


 I thought the OMF lib-imposed limit was about 4K. I've just made a 
 template with a symbol name of 480 characters, and that was OK.
 If Tuple!() works with a nesting level of 50, the limit would have to 
 be pretty big.
 Maybe in the future, we'll be able to add some kind of 'final' 
 declaration to templates, which would indicate that they don't need to 
 be stored in the object file or library, and can be discarded from the 
 symbol table as soon as they are instantiated. This would be 
 particularly useful for recursive templates, which can easily spew 
 reams of garbage into the object file.


 
 What are these templates that can be discarded from the object file? Is 
 it templates that only contain compile-time members? Do these even take 
 space in an object file (since they have no run-time members) ?


Yes, they take up a lot of space in the object file (can be megabytes).

eg, if the source file contains
int f = factorial!(7),

then the object file will also contain the useless intermediates 
factorial!(6), factorial!(5),...factorial!(1). The unused ones don't get 
discarded until link time. I'm not sure why they're in the obj file at 
all, but they certainly get in there.
Jul 14 2006
parent Bruno Medeiros <brunodomedeirosATgmail SPAM.com> writes:
Don Clugston wrote:

 Bruno Medeiros wrote:

 Don Clugston wrote:

 Sean Kelly wrote:

 Chris Nicholson-Sauls wrote:

 Tom S wrote:

 ...DMD eats many many rams...


 *Gasp!*  Those poor sheep/goats!

 I would be interested in seeing exactly what your 'Input' mixin 
 does.  The use of self-referancing returns to initialize it is 
 pretty nifty though.  Might be a way to get rid of the '.end' at 
 the end, though?

 The error baffles me, though.


 Symbols are limited to ~255 chars on Win32.  I think it's a 
 limitation of the object file format.  "symbol too long" errors are 
 painfully common in C++, though they're typically truncated to the 
 max length to allow for debugging.


 I thought the OMF lib-imposed limit was about 4K. I've just made a 
 template with a symbol name of 480 characters, and that was OK.
 If Tuple!() works with a nesting level of 50, the limit would have to 
 be pretty big.
 Maybe in the future, we'll be able to add some kind of 'final' 
 declaration to templates, which would indicate that they don't need 
 to be stored in the object file or library, and can be discarded from 
 the symbol table as soon as they are instantiated. This would be 
 particularly useful for recursive templates, which can easily spew 
 reams of garbage into the object file.


 What are these templates that can be discarded from the object file? 
 Is it templates that only contain compile-time members? Do these even 
 take space in an object file (since they have no run-time members) ?


 
 Yes, they take up a lot of space in the object file (can be megabytes).
 
 eg, if the source file contains
 int f = factorial!(7),
 
 then the object file will also contain the useless intermediates 
 factorial!(6), factorial!(5),...factorial!(1). The unused ones don't get 
 discarded until link time. I'm not sure why they're in the obj file at 
 all, but they certainly get in there.


Then that means a language construct would not be necessary, as the 
compiler or linker should be able themselves to know not to generate 
empty template sections. And that seems easy to implement, at least at 
first sight (if it requires changing optlink we might not be so lucky).

-- 
Bruno Medeiros - CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Jul 15 2006
prev sibling parent Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Chris Nicholson-Sauls wrote:

 Tom S wrote:

 ...DMD eats many many rams...


 
 *Gasp!*  Those poor sheep/goats!
 
 I would be interested in seeing exactly what your 'Input' mixin does.


There you go: http://158.75.59.9/~h3/code/processing.rar
It's in processing/input.d



 The use of self-referancing returns to initialize it is pretty nifty 
 though.  Might be a way to get rid of the '.end' at the end, though?


Hmm... check the code and decide for yourself... I thought using the 
'end' was the most straightforward approach. Otherwise the implicit name 
promotion wouldn't work and I'd have to use field.field all over the 
place. 'field.add' could work, but I considered forcing the user to add 
one 'end' to be simpler.


-- 
Tomasz Stachowiak  /+ a.k.a. h3r3tic +/
Jul 10 2006