www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Everything on the Stack

reply Janthinidae <janthinidae gmail.com> writes:
Thanks to a reddit thread I stumbled over D and thought I should give
it a chance. I'm currently working on project where hard RT
requirements matter. In most places the GC is not suitable. I tried to
port some core functionaly of an existing application which is
currently written in C/C++. I tried to convert some things 1:1 to D,
well aware that a better approach may exist.

I think that only "struct" and "scope class" are suitable to create
objects. structs have the simple advantage that I don't have to type
"scope" in my whole program (can't alias it e.g.), but have the
disadvantage that I can't use an interface nor inheritance.

The application should have a dictionary which contains values of many
different objects. For this example, all objects should support a toFoo
method. To make it generic I've to rule out a struct, or to write
everything in a more functional style (a everythingToFoo method handles
all different types).

interface Entry {
	Foo toFoo();
}

scope class EntryNumeric(T) : Entry {
	T value = T.init;

	Foo toFoo() {
		...
	}
}

alias Entry!(int) EntryInt;
alias Entry!(float) EntryFloat;

setup(Entry[] entries) {
	scope auto e1 = new EntryInt();
	Foo = e1.toFoo();

	scope auto e2 = new EntryInt();
	Foo = e2.toFoo();

	entries[0] = e1;
	entries[1] = e2;
}

init() {
	Entry entries[2];
	setup(entries);
	entries[0].toFoo();
	entries[1].toFoo();
}

I wonder why I can compile the above piece, because e1/e2 shouldn't be
allowed to leave the scope. It obviously crashes.

The next thing I tried to solve is some kind of a templated Queue
class, which can store anything. In C++ it's easy to write a container
class that only works on the stack and uses some uninitialized bytes of
memory and placement new to copy construct objects into the Queue. With
"scope class" this seems to fail because it seems to be impossible to
return a class if it's type uses "scope" class. Missing copy
construction one could probably solve manually, by just carefully
create an own core class library (or just juse memcpy).

struct Queue(T, int N) {
}

scope class MyClass {
}

Queue!(MyClass) m;

Another problem with my whole, everything on the stack idea, is that D
limits my arrays to an internal limit (any paramter to change it?).
It's easy to create a stack overflow anyway, so why a limit? That
static arrays are stored in the .data section make things even more
complicated. That the new core.memory.GC somehow misses a function to
analyze if there is any hidden malloc in any used function would be
usefull as well, because it's not always obvious which construct had
what effect (forgetting scope on a delegate etc...)

Thanks for any insight, tips, hints, ...
Feb 19 2012
next sibling parent Trass3r <un known.com> writes:
 	scope auto e1 = new EntryInt();
 	Foo = e1.toFoo();


You don't need auto there.
And scope is deprecated, use std.typecons' scoped
Feb 20 2012
prev sibling next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Janthinidae:


 Thanks to a reddit thread I stumbled over D and thought I should give
 it a chance.


Welcome here then. Your post is quite interesting, your questions are complex.



 I'm currently working on project where hard RT
 requirements matter. In most places the GC is not suitable.


Please, show me the benchmarks you have seen or written, that rule out the D GC
for your purposes. There's always the possibility that you have written bad
benchmarks.



 I tried to
 port some core functionaly of an existing application which is
 currently written in C/C++. I tried to convert some things 1:1 to D,
 well aware that a better approach may exist.


It looks like an interesting project for D, also useful to spot possible
problems in D itself or its Phobos.



 I think that only "struct" and "scope class" are suitable to create
 objects. structs have the simple advantage that I don't have to type
 "scope" in my whole program (can't alias it e.g.), but have the
 disadvantage that I can't use an interface nor inheritance.


scope, used for your purpose, is deprecated (another usage of scope, for
function arguments, to avoid them to exit the function scope, or to avoid to
pass heap-allocated closures to functions, is still usable).
As Trass3r has said, there is a replacement for scoped classes (but it has some
problems still).



 I wonder why I can compile the above piece, because e1/e2 shouldn't be
 allowed to leave the scope. It obviously crashes.


scope was deprecated also because it was broken. The library defined "scoped"
is similarly broken, but better to have a broken library than a broken language
feature :-)



 The next thing I tried to solve is some kind of a templated Queue
 class, which can store anything. In C++ it's easy to write a container
 class that only works on the stack and uses some uninitialized bytes of
 memory and placement new to copy construct objects into the Queue. With
 "scope class" this seems to fail because it seems to be impossible to
 return a class if it's type uses "scope" class. Missing copy
 construction one could probably solve manually, by just carefully
 create an own core class library (or just juse memcpy).


Generally what's possible in C++ is possible in D too, even if maybe in a
different way. I think all you say here is doable in D too, with different
means.



 Another problem with my whole, everything on the stack idea, is that D
 limits my arrays to an internal limit (any paramter to change it?).
 It's easy to create a stack overflow anyway, so why a limit?


Walter says the limit is present because linkers have such limits and different
linkers have different limits. So I think to allow people to compile (link) D
code with different linkers, there is a built-in limit that is probably smaller
than the limit of most linkers.
Walter also says that past a certain size, it's not good to allocate an array
on the stack, and it's better to allocate it on the heap. If you really want
the stack there is alloca(), but I think allocating a single chunk of memory
from the D-GC or C heaps at the start of the program, doesn't damage the hard
RT qualities of your code.



 That static arrays are stored in the .data section make things
 even more complicated.


I am not expert on this, sorry. What problems is this giving you?



 That the new core.memory.GC somehow misses a function to
 analyze if there is any hidden malloc in any used function would be
 usefull as well, because it's not always obvious which construct had
 what effect (forgetting scope on a delegate etc...)


I have an enhancement request related to this, that usually people appreciate:
http://d.puremagic.com/issues/show_bug.cgi?id=5070
I think there is a similar enhancement request for generic allocations. Plus I
have suggested a  noheap annotation, but it was not appreciated by Don.
I suggest you to ask for this feature in the main D newsgroup. The more people
asks for something similar, the more probable/sooner that it will be
implemented.

Bye,
bearophile
Feb 20 2012
next sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
scope/scoped isn't broken, they're just not safe.  It's better to have an 
unsafe library feature than an unsafe language feature.
Feb 21 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 02/21/2012 11:27 AM, Daniel Murphy wrote:

 scope/scoped isn't broken, they're just not safe.  It's better to have an
 unsafe library feature than an unsafe language feature.


scope is broken because it is not enforced by the means of 
flow-analysis. As a result, it is not safe. Copying it to the library 
and temporarily disabling 'scope' for classes is a good move however, 
because this means we will be able to fix it at an arbitrary point in 
the future without additional code breakage.
Feb 21 2012
parent reply Don Clugston <dac nospam.com> writes:
On 21/02/12 12:12, Timon Gehr wrote:

 On 02/21/2012 11:27 AM, Daniel Murphy wrote:

 scope/scoped isn't broken, they're just not safe. It's better to have an
 unsafe library feature than an unsafe language feature.


 scope is broken because it is not enforced by the means of
 flow-analysis. As a result, it is not safe. Copying it to the library
 and temporarily disabling 'scope' for classes is a good move however,
 because this means we will be able to fix it at an arbitrary point in
 the future without additional code breakage.


Does the library solution actually work the same as the language solution?
Feb 21 2012
parent bearophile <bearophileHUGS lycos.com> writes:
Don Clugston:


 Does the library solution actually work the same as the language solution?


Some time ago scoped worked worse than the built-in scope:
http://d.puremagic.com/issues/show_bug.cgi?id=5115

But I don't know how well scoped works now, I have never used it any more.

Bye,
bearophile
Feb 21 2012
prev sibling parent reply Janthinidae <janthinidae gmail.com> writes:
First thanks for you long response,


 I'm currently working on project where hard RT
 requirements matter. In most places the GC is not suitable.


 
 Please, show me the benchmarks you have seen or written, that rule out the D
GC for your purposes. There's always the possibility that you have written bad
benchmarks.


There is no benchmark, it's just that the software has to have a
worstcase latency to some relevant below 1 ms. If the GC kicks in the
wrong time shit happens. With not using malloc at all or the GC I just
have some kind of proof that this will never be a problem.


 I think that only "struct" and "scope class" are suitable to create
 objects. structs have the simple advantage that I don't have to type
 "scope" in my whole program (can't alias it e.g.), but have the
 disadvantage that I can't use an interface nor inheritance.


 
 scope, used for your purpose, is deprecated (another usage of scope, for
function arguments, to avoid them to exit the function scope, or to avoid to
pass heap-allocated closures to functions, is still usable).
 As Trass3r has said, there is a replacement for scoped classes (but it has
some problems still).


I looked at this class and just have seen that it internally uses a
uint8 array to store the data. This means executable size gets larger
and larger like for any other array.


 scope was deprecated also because it was broken. The library defined "scoped"
is similarly broken, but better to have a broken library than a broken language
feature :-)


Ok didn't knew that it is deprecated.


 The next thing I tried to solve is some kind of a templated Queue
 class, which can store anything. In C++ it's easy to write a container
 class that only works on the stack and uses some uninitialized bytes of
 memory and placement new to copy construct objects into the Queue. With
 "scope class" this seems to fail because it seems to be impossible to
 return a class if it's type uses "scope" class. Missing copy
 construction one could probably solve manually, by just carefully
 create an own core class library (or just juse memcpy).


 
 Generally what's possible in C++ is possible in D too, even if maybe in a
different way. I think all you say here is doable in D too, with different
means.


Yes both are Turing complete :) But after all I hope that D makes
things easier and helps me to spot errors of certain aspects of my
program.


 Another problem with my whole, everything on the stack idea, is that D
 limits my arrays to an internal limit (any paramter to change it?).
 It's easy to create a stack overflow anyway, so why a limit?


 
 Walter says the limit is present because linkers have such limits and
different linkers have different limits. So I think to allow people to compile
(link) D code with different linkers, there is a built-in limit that is
probably smaller than the limit of most linkers.
 Walter also says that past a certain size, it's not good to allocate an array
on the stack, and it's better to allocate it on the heap. If you really want
the stack there is alloca(), but I think allocating a single chunk of memory
from the D-GC or C heaps at the start of the program, doesn't damage the hard
RT qualities of your code.


Ok I understand it, but if some linkers fail, then isn't this the
linkgers problem? I still think something like an option would be nice
until the compiler can ask the linker (which never may happen). Even if
we would say D has the correct approach here, then one can ask why it's
not possible to write

int[10000000] t10 = void;

but possible to write

int[1000000] t0 = void;
...
int[1000000] t0 = void;

But the problem for me here is that all static arrays are in the .data
section (that's why I mentioned it) and make the executable that big.
If I declare an array with "Type[N] = void" I somehow expect, that it
is not initialized and that there is no data allocated in the .data
section.

Trying everything with alloca is maybe a better idea (and rewriting the
Scope template).


 That the new core.memory.GC somehow misses a function to
 analyze if there is any hidden malloc in any used function would be
 usefull as well, because it's not always obvious which construct had
 what effect (forgetting scope on a delegate etc...)


 
 I have an enhancement request related to this, that usually people appreciate:
 http://d.puremagic.com/issues/show_bug.cgi?id=5070
 I think there is a similar enhancement request for generic allocations. Plus I
have suggested a  noheap annotation, but it was not appreciated by Don.
 I suggest you to ask for this feature in the main D newsgroup. The more people
asks for something similar, the more probable/sooner that it will be
implemented.


Ok I'll do that. Thanks once more.
Feb 22 2012
parent bearophile <bearophileHUGS lycos.com> writes:
Janthinidae:


 There is no benchmark, it's just that the software has to have a
 worstcase latency to some relevant below 1 ms. If the GC kicks in the
 wrong time shit happens. With not using malloc at all or the GC I just
 have some kind of proof that this will never be a problem.
 ...
 Trying everything with alloca is maybe a better idea (and rewriting the
 Scope template).


The GC gets active when you allocate memory. So if you allocate some GC-heap
memory at the start of your program, and then you never allocate some more of
it, the GC will be dormant. So it's not required to allocate everything on the
stack.



 Yes both are Turing complete :)


I meant to add "with similar efficiency of C++ code too" :-)



 Ok I understand it, but if some linkers fail, then isn't this the
 linkgers problem?


I don't know, but I presume lot of people lump compiler and linker in their
mind.



 then one can ask why it's
 not possible to write
 
 int[10000000] t10 = void;
 
 but possible to write
 
 int[1000000] t0 = void;
 ...
 int[1000000] t0 = void;


I don't know. I guess the total amount of memory and the largest chunk of
memory are two different parameters for a linker.



 But the problem for me here is that all static arrays are in the .data
 section (that's why I mentioned it) and make the executable that big.
 If I declare an array with "Type[N] = void" I somehow expect, that it
 is not initialized and that there is no data allocated in the .data
 section.


Try this too:

__gshared int[1_000_000] array;
void main() {}



 Ok I'll do that. Thanks once more.


Generally Walter listens to efficiency concerns. You are welcome.

Bye,
bearophile
Feb 22 2012
prev sibling parent bearophile <bearophileHUGS lycos.com> writes:
 Try this too:
 
 __gshared int[1_000_000] array;
 void main() {}


Sorry, I meant:

__gshared int[1_000_000] array = void;
void main() {}

Bye,
bearophile
Feb 22 2012