www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - druntime, templates and traits

reply "monarch_dodra" <monarchdodra gmail.com> writes:
One of the "issues" of druntime is that is that almost everything 
in it is resolved at runtime. This is good because it avoids 
bloat. However, it also means we run into 2 regular issues:

1. Performance (small issue). A lot of code that should be 
otherwise fast is slower than it could be, since druntime will 
check, at runtime, if postblit is needed, if destruction should 
be done. For example, if you "dup" a string by hand, you can do 
it 50% faster than what druntime does.

2. Inference. This is an odd one. Currently, the function that 
operate on arrays (reserve, dup, length) etc... may or may not be 
 safe or pure. Or nothrow. The problem is that we really can't do 
anything about it. duping a string is obviously nothrow, but if 
you dup something with a postblit, then it might not be (nor 
safe).

These issues kind of keep popping up in phobos, and we will have 
to address them sooner or later. The current state of affairs is 
a kind of status quo of "its wrong, inconsistent, but maybe OK 
most of the time".

For example, "dup" is not nothrow, yet it is safe and pure. 
Reserve is nothrow, yet assumeSafeAppend is not. Reserve may 
actually call postblit, but not assumeSafeAppend.

========

The solution (I think) would be to partially templatize 
d-runtime. We can keep most of what is in rt/lifetime.d, but 
statically split it into smaller chunks, which could be piloted 
with more accuracy via a templated "manager".

For now, if we could make a change as simple as "no postblit => 
safe pure nothrow, and fast guaranteed"/"postblit => not safe, 
not pure, may throw", then it would already be a big win (IMO).

========

I had started toying around with this, and the main and immediate 
challenge I ran into is the integration of templates. The 2 
issues are:
1. All templates must be in object.d, or the compiler won't see 
it.
2. A simple (and needed) trait like "hasElaborateCopyConstructor" 
requires a good third of both "typetuple.d" and "traits.d". 
"functionAttributes" is pretty simple though (I think).

So the two challenges are:
1. If we integrate a large amount of template code, where would 
it go?
2. How will we deal with not having massive code duplication?

========

I'm mostly looking to start a discussion on this, and for 
guidance, ideas, on where to start.
Sep 13 2013
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
monarch_dodra:


 So the two challenges are:
 1. If we integrate a large amount of template code, where would 
 it go?


How much large it is?



 2. How will we deal with not having massive code duplication?


Go developers accept a little amount of duplication.
But in your case a solution could be just to deprecate and later 
remove the Phobos code that you put in object.d.

Bye,
bearophile
Sep 13 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-09-13 12:26, monarch_dodra wrote:


 I had started toying around with this, and the main and immediate
 challenge I ran into is the integration of templates. The 2 issues are:
 1. All templates must be in object.d, or the compiler won't see it.


Do you know why that is? Can they be put in a separate module but 
imported by object.d?


 2. A simple (and needed) trait like "hasElaborateCopyConstructor"
 requires a good third of both "typetuple.d" and "traits.d".
 "functionAttributes" is pretty simple though (I think).


Can't we move some of the traits to druntime? std.traits would publicly 
import the ones which were moved.


 So the two challenges are:
 1. If we integrate a large amount of template code, where would it go?


It depends on where it needs to be used. Some suggestions:

* core.traits|typetuple
* rt.util.traits|typetuple


 2. How will we deal with not having massive code duplication?


Move to druntime? Phobos already depends on it.

-- 
/Jacob Carlborg
Sep 13 2013
parent Sean Kelly <sean invisibleduck.org> writes:
On Sep 13, 2013, at 5:35 AM, Jacob Carlborg <doob me.com> wrote:


 On 2013-09-13 12:26, monarch_dodra wrote:
=20

 I had started toying around with this, and the main and immediate
 challenge I ran into is the integration of templates. The 2 issues =




are:

 1. All templates must be in object.d, or the compiler won't see it.


=20
 Do you know why that is? Can they be put in a separate module but =


imported by object.d?

Yes.
Sep 13 2013
prev sibling parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Friday, 13 September 2013 at 10:26:35 UTC, monarch_dodra wrote:

 1. Performance (small issue). A lot of code that should be 
 otherwise fast is slower than it could be, since druntime will 
 check, at runtime, if postblit is needed, if destruction should 
 be done. For example, if you "dup" a string by hand, you can do 
 it 50% faster than what druntime does.


Yes, this is pitfall of D design.


 2. Inference. This is an odd one. Currently, the function that 
 operate on arrays (reserve, dup, length) etc... may or may not 
 be  safe or pure. Or nothrow. The problem is that we really 
 can't do anything about it. duping a string is obviously 
 nothrow, but if you dup something with a postblit, then it 
 might not be (nor safe).


Yes, this is pitfall of D design.


 These issues kind of keep popping up in phobos, and we will 
 have to address them sooner or later. The current state of 
 affairs is a kind of status quo of "its wrong, inconsistent, 
 but maybe OK most of the time".


They are systematically popping up because of systematical 
language features (D is statically typed language with little 
runtime supported).


 For example, "dup" is not nothrow, yet it is safe and pure. 
 Reserve is nothrow, yet assumeSafeAppend is not. Reserve may 
 actually call postblit, but not assumeSafeAppend.

 ========

 The solution (I think) would be to partially templatize 
 d-runtime. We can keep most of what is in rt/lifetime.d, but 
 statically split it into smaller chunks, which could be piloted 
 with more accuracy via a templated "manager".


It depends on how do you want to templatize it. Due to separate 
compilation model it is impossible to know beforehand (when you 
provide phobos library together with druntime) which types will 
be used.


 For now, if we could make a change as simple as "no postblit => 
 safe pure nothrow, and fast guaranteed"/"postblit => not safe, 
 not pure, may throw", then it would already be a big win (IMO).

 ========

 I had started toying around with this, and the main and 
 immediate challenge I ran into is the integration of templates. 
 The 2 issues are:
 1. All templates must be in object.d, or the compiler won't see 
 it.
 2. A simple (and needed) trait like 
 "hasElaborateCopyConstructor" requires a good third of both 
 "typetuple.d" and "traits.d". "functionAttributes" is pretty 
 simple though (I think).


Yes, this is pitfall of D design.


 I'm mostly looking to start a discussion on this, and for 
 guidance, ideas, on where to start.


I doubt that the problems you raised are solvable in general case 
with current runtime capabilities. While it still may be possible 
to tackle some of them, D design (separate compilation model, 
type system, templates, how pure,  safe, nothrow are defined) 
would prevent you from fixing these problems.
Sep 13 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Sep 13, 2013 at 03:20:10PM +0200, Maxim Fomin wrote:

 On Friday, 13 September 2013 at 10:26:35 UTC, monarch_dodra wrote:

1. Performance (small issue). A lot of code that should be
otherwise fast is slower than it could be, since druntime will
check, at runtime, if postblit is needed, if destruction should be
done. For example, if you "dup" a string by hand, you can do it
50% faster than what druntime does.


 
 Yes, this is pitfall of D design.


I don't see what this has to do with D design. It's just a quality of
implementation issue, isn't it?



2. Inference. This is an odd one. Currently, the function that
operate on arrays (reserve, dup, length) etc... may or may not be
 safe or pure. Or nothrow. The problem is that we really can't do
anything about it. duping a string is obviously nothrow, but if
you dup something with a postblit, then it might not be (nor
safe).


 
 Yes, this is pitfall of D design.


It's just a matter of taking care of the details. Dup'ing an array
involves allocating a new array (pure  safe nothrow) and copying
elements over (purity/safety/nothrowness depends on postblit). So there
should be two versions of dup, reserve, etc., one for arrays whose
elements can be copied in a pure/safe/nothrow way, one for arrays where
the safety/etc. of copying elements depends on safety/etc. of the
element postblits.

The latter can be taken care of with attribute inference. So for
example, we could do something like this:

	T[] dup(T)(T[] array) pure  safe nothrow
		if (!hasElaborateCopyConstructor!(T[]))
	{
		// To reduce template bloat, forward to static function
		// that does bitwise copying.
		return simpleDupImpl(array.ptr, array.size, T.sizeof);
	}

	T[] dup(T)(T[] array) /* attributes inferred at compile-time */
		if (hasElaborateCopyConstructor!(T[]))
	{
		// This part is pure  safe nothrow
		T[] copy = simpleDupImpl(array.ptr, array.size, T.sizeof);

		// This part's purity/safety/nothrowness inherits from
		// postblit's characteristics.
		callPostblits!T(copy);
	}



These issues kind of keep popping up in phobos, and we will have
to address them sooner or later. The current state of affairs is a
kind of status quo of "its wrong, inconsistent, but maybe OK most
of the time".




Yeah this is one thing that irks me about D. It works superbly -- for
the most part. But when you hit something outside "the most part", then
it's a cascade of troubles and roadblocks, one leading to the other. We
really need to dig into all these details and straighten them out,
otherwise it's going to be a big source of frustration to potential D
adopters.



 They are systematically popping up because of systematical language
 features (D is statically typed language with little runtime
 supported).


And how do you propose we address that?



For example, "dup" is not nothrow, yet it is safe and pure.
Reserve is nothrow, yet assumeSafeAppend is not. Reserve may
actually call postblit, but not assumeSafeAppend.




Why is assumeSafeAppend not nothrow? If it were up to me, I'd say that
if for whatever reason safe append can't be done (i.e. the assumption is
invalid) and we detect it, we should assert(0), since the code is
obviously not prepared to deal with this case by virtue of calling
assumeSafeAppend in the first place. Throwing an Exception makes no
sense in this case -- it's not something you can handle.



========

The solution (I think) would be to partially templatize d-runtime.
We can keep most of what is in rt/lifetime.d, but statically split
it into smaller chunks, which could be piloted with more accuracy
via a templated "manager".


 
 It depends on how do you want to templatize it. Due to separate
 compilation model it is impossible to know beforehand (when you
 provide phobos library together with druntime) which types will be
 used.


Right, so we use templates for the "front-end" that interface directly
with user types, but the real work is done by static backend functions.
There will be a different backend function to deal with each different
category of cases, and the front-end templates will just forward to the
most appropriate one.



For now, if we could make a change as simple as "no postblit =>
safe pure nothrow, and fast guaranteed"/"postblit => not safe, not
pure, may throw", then it would already be a big win (IMO).

========

I had started toying around with this, and the main and immediate
challenge I ran into is the integration of templates. The 2 issues
are:
1. All templates must be in object.d, or the compiler won't see
it.




public import is your friend. :)



2. A simple (and needed) trait like "hasElaborateCopyConstructor"
requires a good third of both "typetuple.d" and "traits.d".
"functionAttributes" is pretty simple though (I think).


 
 Yes, this is pitfall of D design.


How is this a problem with D design?

I'd argue that a large part of std.traits *should* be in druntime, not
Phobos, because they interface with compiler internals via __traits.
After "discovering" how parameter tuples work yesterday, I'm becoming
more convinced that __traits should not be used by user code at all, but
wrapped by more user-friendly functions in the runtime environment,
i.e., druntime.

One of the issues I ran into while trying to reimplement the built-in
AA's was that it needed some stuff from std.traits, but can't use it
because druntime isn't supposed to depend on Phobos. The duplication in
that case was relatively small, but if std.traits was in druntime in the
first place, this would never have been an issue.



I'm mostly looking to start a discussion on this, and for
guidance, ideas, on where to start.


 
 I doubt that the problems you raised are solvable in general case
 with current runtime capabilities. While it still may be possible to
 tackle some of them, D design (separate compilation model, type
 system, templates, how pure,  safe, nothrow are defined) would
 prevent you from fixing these problems.


The upcoming 2.064 has attribute inference for pure/ safe/nothrow. I'm
pretty sure this will address most, if not all, of these issues, if
applied correctly.


T

-- 
Laissez-faire is a French term commonly interpreted by Conservatives to
mean 'lazy fairy,' which is the belief that if governments are lazy
enough, the Good Fairy will come down from heaven and do all their work
for them.
Sep 13 2013
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 13 September 2013 at 17:02:24 UTC, H. S. Teoh wrote:

For example, "dup" is not nothrow, yet it is safe and pure.
Reserve is nothrow, yet assumeSafeAppend is not. Reserve may
actually call postblit, but not assumeSafeAppend.




 Why is assumeSafeAppend not nothrow? If it were up to me, I'd 
 say that
 if for whatever reason safe append can't be done (i.e. the 
 assumption is
 invalid) and we detect it, we should assert(0), since the code 
 is
 obviously not prepared to deal with this case by virtue of 
 calling
 assumeSafeAppend in the first place. Throwing an Exception 
 makes no
 sense in this case -- it's not something you can handle.


Well, by your own conclusion, it doesn't throw :D

It either just works, or it errors out. No exceptions.


I had started toying around with this, and the main and 
immediate
challenge I ran into is the integration of templates. The 2 
issues
are:
1. All templates must be in object.d, or the compiler won't 
see
it.




 public import is your friend. :)


What I'm getting so far from this conversation, is we should move 
select "foundation blocks" of typetuple/traits from phobos, into 
druntime. the phobos libraries would then publicly import their 
druntime counterparts.

Then object.d would import "core/traits.d" and "core/typetuple.d" 
(or whatever).

Why "public" import btw? Wouldn't that expose the functions in 
traits/typetuple to *all* D code? That seems like a bad idea...

One of the problems though is that we'd want to try to keep those 
"core/traits" libraries as light as possible, since they'd be 
pretty much imported 99% of the time.

Another issue to solve is that the above would work well for say, 
"reserve". However, "dup" (afaik) is hardwired by the compiler to 
call the druntime function "_adDupT". SO we'd have to change that 
if we even *hope* for it to infer anything.
Sep 13 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Sep 13, 2013 at 07:47:32PM +0200, monarch_dodra wrote:

 On Friday, 13 September 2013 at 17:02:24 UTC, H. S. Teoh wrote:

For example, "dup" is not nothrow, yet it is safe and pure.
Reserve is nothrow, yet assumeSafeAppend is not. Reserve may
actually call postblit, but not assumeSafeAppend.




Why is assumeSafeAppend not nothrow? If it were up to me, I'd say
that if for whatever reason safe append can't be done (i.e. the
assumption is invalid) and we detect it, we should assert(0), since
the code is obviously not prepared to deal with this case by virtue
of calling assumeSafeAppend in the first place. Throwing an Exception
makes no sense in this case -- it's not something you can handle.


 
 Well, by your own conclusion, it doesn't throw :D
 
 It either just works, or it errors out. No exceptions.


Right, so how come assumeSafeAppend isn't marked nothrow?



I had started toying around with this, and the main and immediate
challenge I ran into is the integration of templates. The 2 issues
are: 1. All templates must be in object.d, or the compiler won't
see it.




public import is your friend. :)


 
 What I'm getting so far from this conversation, is we should move
 select "foundation blocks" of typetuple/traits from phobos, into
 druntime. the phobos libraries would then publicly import their
 druntime counterparts.


Either that, or factor std.traits in terms of a small number of
fundamental primitives, and move those primitives into druntime.



 Then object.d would import "core/traits.d" and "core/typetuple.d"
 (or whatever).
 
 Why "public" import btw? Wouldn't that expose the functions in
 traits/typetuple to *all* D code? That seems like a bad idea...


This was in response to your question about why stuff must be in
object.d(i) otherwise it won't be seen. If it's only needed internally,
then a public import is not necessary, but if you want stuff to be
imported by default into all programs, but don't want to bloat
object.di, then public import is the solution.

I've thought before, about splitting up object_.d into more manageable
chunks. Public imports would allow us to do this in a transparent,
backwards-compatible way.



 One of the problems though is that we'd want to try to keep those
 "core/traits" libraries as light as possible, since they'd be pretty
 much imported 99% of the time.


Yes, that's why I said above that we should distill the stuff we need
from std.traits into their barest essentials, and put those in druntime.
Then std.traits itself can add more user conveniences around these
primitives.



 Another issue to solve is that the above would work well for say,
 "reserve". However, "dup" (afaik) is hardwired by the compiler to
 call the druntime function "_adDupT". SO we'd have to change that if
 we even *hope* for it to infer anything.


Oh joy, more compiler magic. Let me know when you figure out how to
extricate it from the compiler, since it will probably be useful in
extricating AA's from the compiler as well. ;)

OTOH, another approach, since the compiler already hardcodes this stuff,
is to make the *compiler* infer attributes for .dup. That might actually
be easier than trying to move compiler-hardcoded stuff into druntime.
(The downside is that you'll have to work with C++... :-P)


T

-- 
Klein bottle for rent ... inquire within. -- Stephen Mulraney
Sep 13 2013
parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 13 September 2013 at 18:01:30 UTC, H. S. Teoh wrote:

 On Fri, Sep 13, 2013 at 07:47:32PM +0200, monarch_dodra wrote:

 On Friday, 13 September 2013 at 17:02:24 UTC, H. S. Teoh wrote:

For example, "dup" is not nothrow, yet it is safe and pure.
Reserve is nothrow, yet assumeSafeAppend is not. Reserve may
actually call postblit, but not assumeSafeAppend.




Why is assumeSafeAppend not nothrow? If it were up to me, I'd 
say
that if for whatever reason safe append can't be done (i.e. 
the
assumption is invalid) and we detect it, we should assert(0), 
since
the code is obviously not prepared to deal with this case by 
virtue
of calling assumeSafeAppend in the first place. Throwing an 
Exception
makes no sense in this case -- it's not something you can 
handle.


 
 Well, by your own conclusion, it doesn't throw :D
 
 It either just works, or it errors out. No exceptions.


 Right, so how come assumeSafeAppend isn't marked nothrow?


Oh. Right. I understood you backwards. In that case, it's not 
nothrow simply because it hasn't been made that way yet. On topic:
https://github.com/D-Programming-Language/druntime/pull/553
  - Me trying to make assumeSafeAppend nothrow, and learn how 
druntime works.


 Another issue to solve is that the above would work well for 
 say,
 "reserve". However, "dup" (afaik) is hardwired by the compiler 
 to
 call the druntime function "_adDupT". SO we'd have to change 
 that if
 we even *hope* for it to infer anything.


 Oh joy, more compiler magic. Let me know when you figure out 
 how to
 extricate it from the compiler, since it will probably be 
 useful in
 extricating AA's from the compiler as well. ;)

 OTOH, another approach, since the compiler already hardcodes 
 this stuff,
 is to make the *compiler* infer attributes for .dup. That might 
 actually
 be easier than trying to move compiler-hardcoded stuff into 
 druntime.
 (The downside is that you'll have to work with C++... :-P)


 T


Hum. I wouldn't mind "starting" with reserve, and see from there 
what the best thing to do for dup/idup is from there (including 
just making it a free function in object ?)

An added plus about making it an actual UFCS in object.d, is that 
we'll be able to (correctly) write it as "myArray.dup()".

It's not a property god damn it!
Sep 13 2013