www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - CTFE bug causes null check to pass on null pointers (Issue 7602)

reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
While testing my AA implementation on existing AA-related bug, I came
across this issue:

	http://d.puremagic.com/issues/show_bug.cgi?id=7602
 
Upon playing around a bit with the sample code given in the bug, I
managed to find a code snippet that would cause this code:

	if (impl !is null) { ... }

to actually execute the code inside the if-statement body ***even when
impl is null***, while running in CTFE.

I've bumped the severity to critical because something is very very
wrong with this.


T

-- 
Study gravitation, it's a field with a lot of potential.
Mar 18 2012
parent reply Don Clugston <dac nospam.com> writes:
On 19/03/12 06:43, H. S. Teoh wrote:
 While testing my AA implementation on existing AA-related bug, I came
 across this issue:

 	http://d.puremagic.com/issues/show_bug.cgi?id=7602

 Upon playing around a bit with the sample code given in the bug, I
 managed to find a code snippet that would cause this code:

 	if (impl !is null) { ... }

 to actually execute the code inside the if-statement body ***even when
 impl is null***, while running in CTFE.

 I've bumped the severity to critical because something is very very
 wrong with this.


 T
Yes. The existing D2 AA implementation is hopelessly broken. You have to understand that the whole implementation of AAs in D2 is a HACK. It is extremely complicated and the slightest change to any code in the compiler or the runtime can break it. Basically CTFE has to reverse-engineer the druntime code in order to make it to work. It's not an implementation issue, it's a fundamental design flaw. I do not understand why it still part of the compiler after we agreed to roll back to the D1 version.
Mar 19 2012
next sibling parent "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Don Clugston" <dac nospam.com> wrote in message 
news:jk6ru4$1seu$1 digitalmars.com...
 I do not understand why it still part of the compiler after we agreed to 
 roll back to the D1 version.
Walter didn't agree, so it didn't happen.
Mar 19 2012
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Mar 19, 2012 at 09:49:07AM +0100, Don Clugston wrote:
[...]
 Yes. The existing D2 AA implementation is hopelessly broken.
 You have to understand that the whole implementation of AAs in D2 is
 a HACK. It is extremely complicated and the slightest change to any
 code in the compiler or the runtime can break it. Basically CTFE has
 to reverse-engineer the druntime code in order to make it to work.
 It's not an implementation issue, it's a fundamental design flaw.
I'm working on my AA implementation, hopefully to get it to the point it can replace the current mess. It already fixes a number of AA-related issues in the bug tracker. The main idea is to require a minimal number of lowerings from the compiler (effectively nothing more than syntactic sugar such as V[K] and AA literal syntax), and everything else will be done via existing operator overloading and templating mechanisms. Ideally, CTFE will "just work" with this implementation instead of requiring druntime-specific hacks in the compiler (but I'm not sure whether this will work, since it has to do memory allocations -- does CTFE support that?).
 I do not understand why it still part of the compiler after we
 agreed to roll back to the D1 version.
I'm late to the game; how was the D1 version implemented? T -- A mathematician is a device for turning coffee into theorems. -- P. Erdos
Mar 19 2012
parent reply Don <nospam nospam.com> writes:
On 19.03.2012 18:25, H. S. Teoh wrote:
 On Mon, Mar 19, 2012 at 09:49:07AM +0100, Don Clugston wrote:
 [...]
 Yes. The existing D2 AA implementation is hopelessly broken.
 You have to understand that the whole implementation of AAs in D2 is
 a HACK. It is extremely complicated and the slightest change to any
 code in the compiler or the runtime can break it. Basically CTFE has
 to reverse-engineer the druntime code in order to make it to work.
 It's not an implementation issue, it's a fundamental design flaw.
I'm working on my AA implementation, hopefully to get it to the point it can replace the current mess. It already fixes a number of AA-related issues in the bug tracker. The main idea is to require a minimal number of lowerings from the compiler (effectively nothing more than syntactic sugar such as V[K] and AA literal syntax), and everything else will be done via existing operator overloading and templating mechanisms. Ideally, CTFE will "just work" with this implementation instead of requiring druntime-specific hacks in the compiler (but I'm not sure whether this will work, since it has to do memory allocations -- does CTFE support that?).
Yes, CTFE supports 'new'. The big issue for the runtime is supporting AA literals. CTFE needs to be able to take the output of the runtime functions, and pass it as an AA literal to the rest of the compiler.
 I do not understand why it still part of the compiler after we
 agreed to roll back to the D1 version.
I'm late to the game; how was the D1 version implemented?
It was just extern(C) library functions. The D2 version is exactly the same thing (all of the D1 functions still exist in D2), except that it has an AssociativeArray!(Key, Value) wrapper around the extern(C) functions. Which sounds like a trivial intermediate step to a full library implementation, but it isn't. - it's a template, so it needs to be instantiated. What happens if it hasn't been instantiated yet? - what happens when AssociativeArray isn't a struct template? - what happens if there's an error while instantiating it? - what happens when all the functions are inlined away, and you're left with just void* pointers? - what happens when you something of type V[K] interacting with something of type AssociativeArray!(K, V)? This happens in things like template constraints, is() expressions, etc. - how is CTFE supposed to deal with this ruddy thing, that's fully of nasty casts to void *, and which may yet create AA literals themselves? - how are you supposed to get sensible error messages out of this beast? The answer to these questions is, hundreds of hours work, and the biggest implementation disaster in D's history. There can be no 'intermediate step'. The syntax sugar should be added last, not first.
Mar 19 2012
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Mar 19, 2012 at 09:46:04PM +0100, Don wrote:
 On 19.03.2012 18:25, H. S. Teoh wrote:
[...]
The main idea is to require a minimal number of lowerings from the
compiler (effectively nothing more than syntactic sugar such as V[K]
and AA literal syntax), and everything else will be done via existing
operator overloading and templating mechanisms. Ideally, CTFE will
"just work" with this implementation instead of requiring
druntime-specific hacks in the compiler (but I'm not sure whether
this will work, since it has to do memory allocations -- does CTFE
support that?).
Yes, CTFE supports 'new'. The big issue for the runtime is supporting AA literals. CTFE needs to be able to take the output of the runtime functions, and pass it as an AA literal to the rest of the compiler.
I've thought about supporting literals. Currently what I have in mind is to build the AA in CTFE, then use mixins to explicitly create Slot structs and link them all up into an Impl struct. The address of the Impl struct (that contains the array of Slot*) is then readable at compile-time, so it can be assigned to const AA variables, .dup'd into mutable AA variables, etc. This only works for literals that only have compile-time known contents, of course. Things like: int x; int[string] aa = [ "abc": x ]; won't work with this scheme. To support literals that reference variables, some kind of runtime mechanism would be needed, perhaps something similar to the current implementation where the compiler passes an array of keys and an array of values to an AA factory function. [...]
I do not understand why it still part of the compiler after we
agreed to roll back to the D1 version.
I'm late to the game; how was the D1 version implemented?
It was just extern(C) library functions. The D2 version is exactly the same thing (all of the D1 functions still exist in D2), except that it has an AssociativeArray!(Key, Value) wrapper around the extern(C) functions. Which sounds like a trivial intermediate step to a full library implementation, but it isn't.
Yeah, this "wrapper" is the source of a good number of issues currently on the bugtracker. It's also extremely ugly (the Range interface, for example, is essentially a copy-n-paste of the structs in aaA.d).
 - it's a template, so it needs to be instantiated. What happens if
 it hasn't been instantiated yet?
 - what happens when AssociativeArray isn't a struct template?
 - what happens if there's an error while instantiating it?
 - what happens when all the functions are inlined away, and you're
 left with just void* pointers?
 - what happens when you something of type V[K] interacting with
 something of type AssociativeArray!(K, V)? This happens in things
 like template constraints, is() expressions, etc.
 - how is CTFE supposed to deal with this ruddy thing, that's fully
 of nasty casts to void *, and which may yet create AA literals
 themselves?
 - how are you supposed to get sensible error messages out of this beast?
The source of most of these problems is the schizophrenic split between aaA.d and struct AssociativeArray. It has to be one or the other. There can be no intermediate. The compiler needs to treat V[K] as an alias for AssociativeArray!(K,V) (or vice versa, but regardless, the two must be *identical* in all respects). Having two different things for them (such as aaA.d returning void* and AssociativeArray!(K,V) being a struct wrapping a void*) only leads to pain and disaster.
 The answer to these questions is, hundreds of hours work, and the
 biggest implementation disaster in D's history. There can be no
 'intermediate step'. The syntax sugar should be added last, not first.
Agreed. T -- "Computer Science is no more about computers than astronomy is about telescopes." -- E.W. Dijkstra
Mar 19 2012
parent Don <nospam nospam.com> writes:
On 19.03.2012 22:14, H. S. Teoh wrote:
 On Mon, Mar 19, 2012 at 09:46:04PM +0100, Don wrote:
 On 19.03.2012 18:25, H. S. Teoh wrote:
[...]
 The main idea is to require a minimal number of lowerings from the
 compiler (effectively nothing more than syntactic sugar such as V[K]
 and AA literal syntax), and everything else will be done via existing
 operator overloading and templating mechanisms. Ideally, CTFE will
 "just work" with this implementation instead of requiring
 druntime-specific hacks in the compiler (but I'm not sure whether
 this will work, since it has to do memory allocations -- does CTFE
 support that?).
Yes, CTFE supports 'new'. The big issue for the runtime is supporting AA literals. CTFE needs to be able to take the output of the runtime functions, and pass it as an AA literal to the rest of the compiler.
I've thought about supporting literals. Currently what I have in mind is to build the AA in CTFE, then use mixins to explicitly create Slot structs and link them all up into an Impl struct. The address of the Impl struct (that contains the array of Slot*) is then readable at compile-time, so it can be assigned to const AA variables, .dup'd into mutable AA variables, etc. This only works for literals that only have compile-time known contents, of course. Things like: int x; int[string] aa = [ "abc": x ]; won't work with this scheme. To support literals that reference variables, some kind of runtime mechanism would be needed, perhaps something similar to the current implementation where the compiler passes an array of keys and an array of values to an AA factory function.
It may help you to know that inside the compiler, AA literals are implemented as two arrays, one of keys, one of values. Those non-constant literals never participate in CTFE, so I think it would be best to treat them completely separately.
 [...]
 I do not understand why it still part of the compiler after we
 agreed to roll back to the D1 version.
I'm late to the game; how was the D1 version implemented?
It was just extern(C) library functions. The D2 version is exactly the same thing (all of the D1 functions still exist in D2), except that it has an AssociativeArray!(Key, Value) wrapper around the extern(C) functions. Which sounds like a trivial intermediate step to a full library implementation, but it isn't.
Yeah, this "wrapper" is the source of a good number of issues currently on the bugtracker. It's also extremely ugly (the Range interface, for example, is essentially a copy-n-paste of the structs in aaA.d).
 - it's a template, so it needs to be instantiated. What happens if
 it hasn't been instantiated yet?
 - what happens when AssociativeArray isn't a struct template?
 - what happens if there's an error while instantiating it?
 - what happens when all the functions are inlined away, and you're
 left with just void* pointers?
 - what happens when you something of type V[K] interacting with
 something of type AssociativeArray!(K, V)? This happens in things
 like template constraints, is() expressions, etc.
 - how is CTFE supposed to deal with this ruddy thing, that's fully
 of nasty casts to void *, and which may yet create AA literals
 themselves?
 - how are you supposed to get sensible error messages out of this beast?
The source of most of these problems is the schizophrenic split between aaA.d and struct AssociativeArray. It has to be one or the other. There can be no intermediate. The compiler needs to treat V[K] as an alias for AssociativeArray!(K,V) (or vice versa, but regardless, the two must be *identical* in all respects). Having two different things for them (such as aaA.d returning void* and AssociativeArray!(K,V) being a struct wrapping a void*) only leads to pain and disaster.
 The answer to these questions is, hundreds of hours work, and the
 biggest implementation disaster in D's history. There can be no
 'intermediate step'. The syntax sugar should be added last, not first.
Agreed. T
Mar 19 2012
prev sibling parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Don wrote:
 On 19.03.2012 18:25, H. S. Teoh wrote:
On Mon, Mar 19, 2012 at 09:49:07AM +0100, Don Clugston wrote:
[...]
Yes. The existing D2 AA implementation is hopelessly broken.
You have to understand that the whole implementation of AAs in D2 is
a HACK. It is extremely complicated and the slightest change to any
code in the compiler or the runtime can break it. Basically CTFE has
to reverse-engineer the druntime code in order to make it to work.
It's not an implementation issue, it's a fundamental design flaw.
I'm working on my AA implementation, hopefully to get it to the point it can replace the current mess. It already fixes a number of AA-related issues in the bug tracker. The main idea is to require a minimal number of lowerings from the compiler (effectively nothing more than syntactic sugar such as V[K] and AA literal syntax), and everything else will be done via existing operator overloading and templating mechanisms. Ideally, CTFE will "just work" with this implementation instead of requiring druntime-specific hacks in the compiler (but I'm not sure whether this will work, since it has to do memory allocations -- does CTFE support that?).
Yes, CTFE supports 'new'. The big issue for the runtime is supporting AA literals. CTFE needs to be able to take the output of the runtime functions, and pass it as an AA literal to the rest of the compiler.
Interesting. How do I make use of this? struct Foo {} Foo* foo() { auto a = new Foo; return a; } unittest { //enum b = foo(); // fails } => Error: cannot use non-constant CTFE pointer in an initializer What's the trick to use memory allocated in a CTFE. Say e.g. I want to build a tree at compile time using CTFE. Jens
Mar 21 2012
parent reply "David Nadlinger" <see klickverbot.at> writes:
On Wednesday, 21 March 2012 at 09:51:43 UTC, Jens Mueller wrote:
 Interesting. How do I make use of this?

 […]

 What's the trick to use memory allocated in a CTFE. Say e.g. I 
 want to
 build a tree at compile time using CTFE.
You can't do that right now (i.e. converting CTFE-allocated memory to initializers for run-time values), but it enables you to use classes, etc. _during_ CTFE. David
Mar 21 2012
next sibling parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
David Nadlinger wrote:
 On Wednesday, 21 March 2012 at 09:51:43 UTC, Jens Mueller wrote:
Interesting. How do I make use of this?

[$B!D(B]

What's the trick to use memory allocated in a CTFE. Say e.g. I
want to
build a tree at compile time using CTFE.
You can't do that right now (i.e. converting CTFE-allocated memory to initializers for run-time values), but it enables you to use classes, etc. _during_ CTFE.
Just for clarification: Why do you say "initializers for _run-time_ values"? I believe I just want to allocate memory at compile-time. I just found out you are not allowed to return an instance. But you can use out parameters. That means I can achieve what I have in mind using classes. Even though it looks a bit clumsy and not like I do it in non-CTFE code. Many Thanks. Jens
Mar 21 2012
parent reply Don Clugston <dac nospam.com> writes:
On 21/03/12 11:22, Jens Mueller wrote:
 David Nadlinger wrote:
 On Wednesday, 21 March 2012 at 09:51:43 UTC, Jens Mueller wrote:
 Interesting. How do I make use of this?

 [$B!D(B]

 What's the trick to use memory allocated in a CTFE. Say e.g. I
 want to
 build a tree at compile time using CTFE.
You can't do that right now (i.e. converting CTFE-allocated memory to initializers for run-time values), but it enables you to use classes, etc. _during_ CTFE.
Just for clarification: Why do you say "initializers for _run-time_ values"? I believe I just want to allocate memory at compile-time.
Any heap-allocated object is on the CTFE heap. At runtime, it needs to be on the runtime heap. There is currently no mechanism in the back-end for transferring data from the CTFE heap to the runtime heap, so it currently generates an error message. You can return structs and arrays, because we have struct literals and array literals, but there is no such thing as a class literal. CTFE will currently allow you to return a class, but only if it is null.
 I just found out you are not allowed to return an instance. But you can
 use out parameters.
Oh dear. That sounds like a bug. There is just no way you can automatically instantiate a class at runtime, using CTFE data. There would need to be code in the runtime to do it, and it just doesn't exist! That means I can achieve what I have in mind using
 classes. Even though it looks a bit clumsy and not like I do it in
 non-CTFE code.
 
 Many Thanks.
 
 Jens
Mar 21 2012
next sibling parent Jens Mueller <jens.k.mueller gmx.de> writes:
Don Clugston wrote:
 On 21/03/12 11:22, Jens Mueller wrote:
 David Nadlinger wrote:
 On Wednesday, 21 March 2012 at 09:51:43 UTC, Jens Mueller wrote:
 Interesting. How do I make use of this?

 [$B!D(B]

 What's the trick to use memory allocated in a CTFE. Say e.g. I
 want to
 build a tree at compile time using CTFE.
You can't do that right now (i.e. converting CTFE-allocated memory to initializers for run-time values), but it enables you to use classes, etc. _during_ CTFE.
Just for clarification: Why do you say "initializers for _run-time_ values"? I believe I just want to allocate memory at compile-time.
Any heap-allocated object is on the CTFE heap. At runtime, it needs to be on the runtime heap. There is currently no mechanism in the back-end for transferring data from the CTFE heap to the runtime heap, so it currently generates an error message.
That's fine with me. Just want to use new at compile-time. But being able to move memory from CTFE heap to runtime heap would be nice and probably finds its uses.
 You can return structs and arrays, because we have struct literals and
 array literals, but there is no such thing as a class literal. CTFE will
 currently allow you to return a class, but only if it is null.
Aha.
 I just found out you are not allowed to return an instance. But you can
 use out parameters.
Oh dear. That sounds like a bug. There is just no way you can automatically instantiate a class at runtime, using CTFE data. There would need to be code in the runtime to do it, and it just doesn't exist!
I'm doing it all at compile-time. I need to build some data structure (using pointers) at compile time to generate a string that gets mixed in. I think there is no bug. It behaves perfectly as you say. Many Thanks Don for bringing run-time to compile-time. Making compile-time programming less awkward. Jens
Mar 21 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-03-21 11:53, Don Clugston wrote:
 You can return structs and arrays, because we have struct literals and
 array literals, but there is no such thing as a class literal. CTFE will
 currently allow you to return a class, but only if it is null.
What about anonymous nested classes, would that kind of like class literals? http://dlang.org/class.html#anonymous -- /Jacob Carlborg
Mar 21 2012
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Mar 21, 2012 at 10:55:05AM +0100, David Nadlinger wrote:
 On Wednesday, 21 March 2012 at 09:51:43 UTC, Jens Mueller wrote:
Interesting. How do I make use of this?

[…]

What's the trick to use memory allocated in a CTFE. Say e.g. I want
to build a tree at compile time using CTFE.
You can't do that right now (i.e. converting CTFE-allocated memory to initializers for run-time values), but it enables you to use classes, etc. _during_ CTFE.
[...] Actually you can... though it requires hacks using string mixins. Basically, you create the structure in CTFE, then walk the structure and construct a string containing declarations of the form: "struct node __node_1234 = { value1, value2, ... };" then use a string mixin to instantiate them. This will create the tree nodes as module globals, so they are compiled into the object file. They therefore also have runtime addresses, so the root of the tree can be assigned into a runtime tree pointer, for example. T -- Let X be the set not defined by this sentence...
Mar 21 2012