digitalmars.D - escaping pointer to scope local array: bug or not?
- HOSOKAWA Kenchi (22/22) Aug 16 2009 It seems dmd 2.031 forgets scope attribute for array.ptr in some cases, ...
- Robert Jacques (4/28) Aug 16 2009 I'd recommend checking to see if p* was allocated on the stack or on the...
- Robert Fraser (4/11) Aug 17 2009 Probably should be an error, but, FWIW, scope arrays are still
- Hosokawa Kenchi (12/26) Aug 17 2009 I'd tried to check where 'scoped' variables are on the Jacques's advice.
- Steven Schveighoffer (12/41) Aug 18 2009 remember, scope is a storage class, not a type modifier. "scopeness" is...
- bearophile (4/7) Aug 18 2009 Why is that bad?
- Steven Schveighoffer (5/10) Aug 18 2009 I don't think it's bad, but definitely a lot of work. I would be all fo...
- Robert Jacques (10/22) Aug 18 2009 Actually, it's really bad. Escape analysis requires whole program
- Steven Schveighoffer (11/37) Aug 18 2009 Admitting I didn't read any of that, I think incremental analysis is
- Robert Jacques (9/52) Aug 18 2009 *sigh* That doesn't work. From the wikipedia article:
- Steven Schveighoffer (7/14) Aug 19 2009 It also says that Java 6, a language compiled as I proposed D could be, ...
- Robert Fraser (5/7) Aug 19 2009 Java's escape analysis is done at runtime (during JIT compilation) AFAIK...
- Robert Fraser (4/5) Aug 18 2009 Does anyone sell static libraries anymore? There are too many problems
- Robert Jacques (9/14) Aug 18 2009 I don't know about selling, but (for instance) NVIDIA's CUDA shipped as ...
- HOSOKAWA Kenchi (32/42) Aug 18 2009 It seems that scopeness as-is is not a "hint to the compiler of where to...
- Jarrett Billingsley (5/6) Aug 17 2009 Just for future reference, "scope auto" is redundant. "auto" does not
- Hosokawa Kenchi (3/10) Aug 17 2009 Thanks for the nice advice.
It seems dmd 2.031 forgets scope attribute for array.ptr in some cases, so that it allows escaping a pointer to scope local array. I'm not sure this is a bug or a kind of "dangerous-but-valid". int[] a() { scope auto a = new int[1]; return a; // error; escaping reference to scope local array } int* ap() { scope auto a = new int[1]; return a.ptr; // no error; this is the problem } int* i() { int i; return &i; // error; escaping reference to local variable } int* ip() { scope int* p = new int; return p; // no error; only is "int* p" local, "new int" not scope local? }
Aug 16 2009
On Sun, 16 Aug 2009 10:13:42 -0700, HOSOKAWA Kenchi <hskwk inter7.jp> wrote:It seems dmd 2.031 forgets scope attribute for array.ptr in some cases, so that it allows escaping a pointer to scope local array. I'm not sure this is a bug or a kind of "dangerous-but-valid". int[] a() { scope auto a = new int[1]; return a; // error; escaping reference to scope local array } int* ap() { scope auto a = new int[1]; return a.ptr; // no error; this is the problem } int* i() { int i; return &i; // error; escaping reference to local variable } int* ip() { scope int* p = new int; return p; // no error; only is "int* p" local, "new int" not scope local? }I'd recommend checking to see if p* was allocated on the stack or on the heap, as the difference represents two very different bugs.
Aug 16 2009
HOSOKAWA Kenchi wrote:It seems dmd 2.031 forgets scope attribute for array.ptr in some cases, so that it allows escaping a pointer to scope local array. I'm not sure this is a bug or a kind of "dangerous-but-valid". int* ap() { scope auto a = new int[1]; return a.ptr; // no error; this is the problem }Probably should be an error, but, FWIW, scope arrays are still heap-allocated (yeah, I know it's inconsistent). So there's no chance of memory corruption, etc.; it's as if you didn't have the "scope" there.
Aug 17 2009
Robert Fraser Wrote:HOSOKAWA Kenchi wrote:I'd tried to check where 'scoped' variables are on the Jacques's advice. As a result, I reached the same conclusion to yours. They are heap-allocated, of course, would not be collected by GC (I'm not sure collect-proofness is guaranteed or not). I suppose semantics of 'scope' is still ambiguous. int[] f() { scope x = new int[6]; auto y = x[1..3]; return y; // no error, successfully escape slice-reference of 'originally' scoped array. } I'm unable to make a quick decision that it SHOULD be error or not. I wish here is a opinion which gives clear cut on this issue.It seems dmd 2.031 forgets scope attribute for array.ptr in some cases, so that it allows escaping a pointer to scope local array. I'm not sure this is a bug or a kind of "dangerous-but-valid".>int* ap() { scope auto a = new int[1]; return a.ptr; // no error; this is the problem }Probably should be an error, but, FWIW, scope arrays are still heap-allocated (yeah, I know it's inconsistent). So there's no chance of memory corruption, etc.; it's as if you didn't have the "scope" there.
Aug 17 2009
On Mon, 17 Aug 2009 16:51:20 -0400, Hosokawa Kenchi <hskwk inter7.jp> wrote:Robert Fraser Wrote:remember, scope is a storage class, not a type modifier. "scopeness" is only a hint to the compiler of where to store it originally, the hint is not passed on to other variables which point to the same data. I'm surprised it's actually an error to try and return a scope variable. One way to get scope to work as you desire is to make it a type modifier and define rules about assignment, but I'm not sure that's a good answer. Another way is to perform escape analysis, but Walter has expressed that he doesn't want to do that. It would require an intermediate interface language for imports where annotations could be added by the compiler. -SteveHOSOKAWA Kenchi wrote:I'd tried to check where 'scoped' variables are on the Jacques's advice. As a result, I reached the same conclusion to yours. They are heap-allocated, of course, would not be collected by GC (I'm not sure collect-proofness is guaranteed or not). I suppose semantics of 'scope' is still ambiguous. int[] f() { scope x = new int[6]; auto y = x[1..3]; return y; // no error, successfully escape slice-reference of 'originally' scoped array. } I'm unable to make a quick decision that it SHOULD be error or not. I wish here is a opinion which gives clear cut on this issue.It seems dmd 2.031 forgets scope attribute for array.ptr in somecases, so that it allows escaping a pointer to scope local array.I'm not sure this is a bug or a kind of "dangerous-but-valid".>int* ap() { scope auto a = new int[1]; return a.ptr; // no error; this is the problem }Probably should be an error, but, FWIW, scope arrays are still heap-allocated (yeah, I know it's inconsistent). So there's no chance of memory corruption, etc.; it's as if you didn't have the "scope" there.
Aug 18 2009
Steven Schveighoffer:Another way is to perform escape analysis, but Walter has expressed that he doesn't want to do that. It would require an intermediate interface language for imports where annotations could be added by the compiler.Why is that bad? Bye, bearophile
Aug 18 2009
On Tue, 18 Aug 2009 13:34:36 -0400, bearophile <bearophileHUGS lycos.com> wrote:Steven Schveighoffer:I don't think it's bad, but definitely a lot of work. I would be all for it. -SteveAnother way is to perform escape analysis, but Walter has expressed that he doesn't want to do that. It would require an intermediate interface language for imports where annotations could be added by the compiler.Why is that bad?
Aug 18 2009
On Tue, 18 Aug 2009 10:38:50 -0700, Steven Schveighoffer <schveiguy yahoo.com> wrote:On Tue, 18 Aug 2009 13:34:36 -0400, bearophile <bearophileHUGS lycos.com> wrote:Actually, it's really bad. Escape analysis requires whole program analysis. It would be impossible to do incremental compilation or to ship/sell D libraries in binary format. I'd recomend checking out http://en.wikipedia.org/wiki/Escape_analysis for an overview of the issues involved. You can avoid doing whole program analysis by introducing ownership types and being a bit conservative in what you allow. There's a (bit confusing) wiki page proposal an how to implement it at http://www.prowiki.org/wiki4d/wiki.cgi?OwnershipTypesInD.Steven Schveighoffer:I don't think it's bad, but definitely a lot of work. I would be all for it. -SteveAnother way is to perform escape analysis, but Walter has expressed that he doesn't want to do that. It would require an intermediate interface language for imports where annotations could be added by the compiler.Why is that bad?
Aug 18 2009
On Tue, 18 Aug 2009 13:48:23 -0400, Robert Jacques <sandford jhu.edu> wrote:On Tue, 18 Aug 2009 10:38:50 -0700, Steven Schveighoffer <schveiguy yahoo.com> wrote:Admitting I didn't read any of that, I think incremental analysis is possible as long as import files are generated by the compiler post-analysis. i.e. the compiler is able to alter the function signature indicating escape analysis information. With something like that, you could still ship in binary format, along with generated import files that describe the function signatures (if one requires building against your product). In fact, the import files could be a part of the binary files, similar to how java works. -SteveOn Tue, 18 Aug 2009 13:34:36 -0400, bearophile <bearophileHUGS lycos.com> wrote:Actually, it's really bad. Escape analysis requires whole program analysis. It would be impossible to do incremental compilation or to ship/sell D libraries in binary format. I'd recomend checking out http://en.wikipedia.org/wiki/Escape_analysis for an overview of the issues involved. You can avoid doing whole program analysis by introducing ownership types and being a bit conservative in what you allow. There's a (bit confusing) wiki page proposal an how to implement it at http://www.prowiki.org/wiki4d/wiki.cgi?OwnershipTypesInD.Steven Schveighoffer:I don't think it's bad, but definitely a lot of work. I would be all for it. -SteveAnother way is to perform escape analysis, but Walter has expressed that he doesn't want to do that. It would require an intermediate interface language for imports where annotations could be added by the compiler.Why is that bad?
Aug 18 2009
On Tue, 18 Aug 2009 10:57:50 -0700, Steven Schveighoffer <schveiguy yahoo.com> wrote:On Tue, 18 Aug 2009 13:48:23 -0400, Robert Jacques <sandford jhu.edu> wrote:*sigh* That doesn't work. From the wikipedia article:On Tue, 18 Aug 2009 10:38:50 -0700, Steven Schveighoffer <schveiguy yahoo.com> wrote:Admitting I didn't read any of that, I think incremental analysis is possible as long as import files are generated by the compiler post-analysis. i.e. the compiler is able to alter the function signature indicating escape analysis information. With something like that, you could still ship in binary format, along with generated import files that describe the function signatures (if one requires building against your product). In fact, the import files could be a part of the binary files, similar to how java works. -SteveOn Tue, 18 Aug 2009 13:34:36 -0400, bearophile <bearophileHUGS lycos.com> wrote:Actually, it's really bad. Escape analysis requires whole program analysis. It would be impossible to do incremental compilation or to ship/sell D libraries in binary format. I'd recomend checking out http://en.wikipedia.org/wiki/Escape_analysis for an overview of the issues involved. You can avoid doing whole program analysis by introducing ownership types and being a bit conservative in what you allow. There's a (bit confusing) wiki page proposal an how to implement it at http://www.prowiki.org/wiki4d/wiki.cgi?OwnershipTypesInD.Steven Schveighoffer:I don't think it's bad, but definitely a lot of work. I would be all for it. -SteveAnother way is to perform escape analysis, but Walter has expressed that he doesn't want to do that. It would require an intermediate interface language for imports where annotations could be added by the compiler.Why is that bad?In traditional static compilation, method overriding can make escape analysis impossible, as any called method might be overridden by a version that allows a pointer to escape.For example: Let's take a 3rd party pre-compiled library with class A and function f(A a), neither of which have any escapes. Now create a subclass of A, B, which does contain escapes. Does f(B) escape or not? Now, introducing some ownership types, (scope, stack, heap, shared, mobile), gives the compiler / methods the guarantees they need to get over this problem.
Aug 18 2009
On Tue, 18 Aug 2009 18:08:56 -0400, Robert Jacques <sandford jhu.edu> wrote:*sigh* That doesn't work. From the wikipedia article:It also says that Java 6, a language compiled as I proposed D could be, has escape analysis. I don't think it's easy, but it's definitely possible. Besides, this entire argument is moot if Walter doesn't want to do it... -SteveIn traditional static compilation, method overriding can make escape analysis impossible, as any called method might be overridden by a version that allows a pointer to escape.For example: Let's take a 3rd party pre-compiled library with class A and function f(A a), neither of which have any escapes. Now create a subclass of A, B, which does contain escapes. Does f(B) escape or not?
Aug 19 2009
Steven Schveighoffer wrote:It also says that Java 6, a language compiled as I proposed D could be, has escape analysis.Java's escape analysis is done at runtime (during JIT compilation) AFAIK. LDC can compile to bitcode and link-time codegen could be used to deal with escape analysis... this doesn't help (much) in generating errors, but it allows better codegen.
Aug 19 2009
Robert Jacques wrote:ship/sell D libraries in binary format.Does anyone sell static libraries anymore? There are too many problems with static linking for that to be very viable. Most libraries I've seen are sold as DLLs.
Aug 18 2009
On Tue, 18 Aug 2009 11:54:27 -0700, Robert Fraser <fraserofthenight gmail.com> wrote:Robert Jacques wrote:I don't know about selling, but (for instance) NVIDIA's CUDA shipped as a pair of static and dynamic libraries. For a long time, runtime linking wasn't possible with however they did the DLLs so you had to use the static libs. But, whole program/escape analysis also has to have access to the DLL's source code, to work, since it has to check/know what each function does with its input arguments.ship/sell D libraries in binary format.Does anyone sell static libraries anymore? There are too many problems with static linking for that to be very viable. Most libraries I've seen are sold as DLLs.
Aug 18 2009
Steven Schveighoffer Wrote:remember, scope is a storage class, not a type modifier. "scopeness" is only a hint to the compiler of where to store it originally, the hint is not passed on to other variables which point to the same data. I'm surprised it's actually an error to try and return a scope variable. One way to get scope to work as you desire is to make it a type modifier and define rules about assignment, but I'm not sure that's a good answer. Another way is to perform escape analysis, but Walter has expressed that he doesn't want to do that. It would require an intermediate interface language for imports where annotations could be added by the compiler.It seems that scopeness as-is is not a "hint to the compiler of where to store it originally" because scope reference accepts object which is not allocated at there. Here is a example: class C { int i = 1; ~this() { writeln("~C"); } void foo() {} } int* f(C arg) { scope c = arg; return &(c.i); } void main() { auto c = new C; auto p = f(c); // destructor is called after f() writeln(*p); // success, at first grance c.foo; // runtime error: access violation } The instance have been collected, hence escaped pointer probably points a garbage. The 'scope' is misleading rather than a hint in this case, since resource had been acquired far away from the 'scope'. Current behaviors of 'scope' are: 1. Prohibit to return the scope references (as you said, it is a storage class and not transitive). (compile time) 2. Calling destructor for the referenced instance when the reference goes out of scope. (runtime) I think RAII has similarity to const/immutable, which is fully cared in D2: * It is safe to reference values/objects from multi-thread, etc. if the referenced object will not change. * It is safe to destruct a object when reference goes out of scope if the object is referenced only from the scope. Consequently, the best answer is presumably make it a transitive type modifier. At least adding rules about assignment, right-hand value of 'scope' reference must be just allocated by new operator. It is too dangerous to allow referencing any object, especially parameter of the function. I agree that escape analysis is bad solution because it makes compiler implementation too hard.
Aug 18 2009
On Sun, Aug 16, 2009 at 1:13 PM, HOSOKAWA Kenchi<hskwk inter7.jp> wrote:=A0 =A0 =A0 =A0scope auto a =3D new int[1];Just for future reference, "scope auto" is redundant. "auto" does not mean "infer the type"; the absence of a type is enough to do that. "auto" is just the default storage class. "scope a =3D new int[1];" will work fine (as will "const a =3D 4;" "static a =3D 5;" etc.).
Aug 17 2009
Jarrett Billingsley Wrote:On Sun, Aug 16, 2009 at 1:13 PM, HOSOKAWA Kenchi<hskwk inter7.jp> wrote:Thanks for the nice advice. Now I become more familiar with D!Just for future reference, "scope auto" is redundant. "auto" does not mean "infer the type"; the absence of a type is enough to do that. "auto" is just the default storage class. "scope a = new int[1];" will work fine (as will "const a = 4;" "static a = 5;" etc.).
Aug 17 2009