digitalmars.D - Isolated by example
- deadalnix (70/70) May 01 2014 First the original post I made on this forum :
- Rikki Cattermole (4/76) May 02 2014 I've been thinking about something similar as being an 'obvious'
- Walter Bright (3/5) May 02 2014 It's nearly the same as http://wiki.dlang.org/DIP29 except that DIP29 tr...
- deadalnix (3/9) May 02 2014 DIP29 is not safe, does not help to construct immutables, is not
- Walter Bright (7/12) May 02 2014 immutable p = new int;
- Andrei Alexandrescu (4/15) May 02 2014 I think a more detailed comparison would do well here. My understanding
- "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> (46/75) May 02 2014 I like it a lot, as it solves several problems elegantly! Some
- deadalnix (8/36) May 02 2014 Of this.
- "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> (12/60) May 02 2014 I think the situation is different here. For nullables, you
- deadalnix (4/10) May 02 2014 No, I expect the compiler to backtrack inference when it hits an
- "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> (5/15) May 02 2014 This might work, but would require defining an order of
- deadalnix (9/25) May 02 2014 Yes, but this is unrelated to isolated. In fact this is already
- Steven Schveighoffer (40/100) May 02 2014 When you say "consumed", you mean it statically cannot be used, not that...
- Meta (7/11) May 02 2014 This will not work well with UFCS.
- Steven Schveighoffer (5/16) May 02 2014 Some interaction with pure would be in order. I don't think ints.put(3) ...
- Dylan Knutson (8/8) May 02 2014 Correct me if I'm wrong here, but this seems really similar to
- Meta (3/11) May 02 2014 I don't think bolting Rust's type system onto D is a viable
- "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> (17/25) May 02 2014 Yes, there are some parallels, although there's no "merging of
- deadalnix (13/21) May 02 2014 There is some similarity, but Rust system has a bit more
First the original post I made on this forum : http://forum.dlang.org/thread/kluaojijixhwigoujeip forum.dlang.org#post-kluaojijixhwigoujeip:40forum.dlang.org Now some sample code/explanation to get it better. D. It introduces a new qualifier. The qualifier is necessary on function signatures (inference is possible to some extent, like for pure functions), object/struct fields and globals. It is inferred on local variables (but can be explicited if one wish). An isolated is in an 'island'. The 'island' is implicit and tracked by the compiler. We have one immutable and one shared island. We have one TL island per thread. And we can have an infinity of isolated island. An isolated is consumed when: - it is returned - it is passed as argument - it is assigned to another island When an isolated goes out of scope without being consumed, the compiler can free the whole island: void foo() { A a = new A(); // a is isolated if A's ctor allows. // a is not consumed. The compiler can insert memory freeing. } As we can see, it allows the compiler to do some freeing for us to reduce GC pressure. Manu why aren't you already here cheering ? When an isolated is consumed, the island it is in is merged into the island that consumes it. All reference to the island become write only until next write. void foo() { A a = new A(); // a is isolated immutable b = a; // a's island is merged into immutable // a's island has been consumed. a is not readable at this point. // a.foo() // Error a = new A(); // OK a.foo(); // OK, we have written in a. } So far, we saw that isolated helps to construct const/immutable/shared objects and can be used by the compiler to insert free. isolated also help to bridge the RC world and the GC world. struct RC(T) is(isReferenceType!T) { private T mystuff; this(isolated T stuff) { mystuff = stuff; } // All code to do ref counting goes here... } Here, the RC struct must be constructed with an isolated. To do so, the isolated have to be passed as argument: it is consumed. As a result, the RC struct can be sure that it has the only usable reference to stuff that is around. Now some concurrency goodies: void foo() { auto tid = spawn(&spawnedFunc, thisTid); A a = new A(); send(tid, a); // OK, a is an isolated. // a can't be used here anymore as it is consumed. // You can simply pass isolated around across thread safely. } Now that is pretty sweet. First we don't have to do the crazy and unsafe dance of "cast to shared and cast back to me" and this is actually safe. Go's type system do is not safe across channel, that puts us ahead in one of the things go does best. std.parallelism can also benefit from this. There is also several benefit when the optimizer knows about isolated (different island do not alias each other). I hope the idea get across better with some sample code and will be considered. As sample code shows, isolated do not need to be specified explicitly often. User not annotating can get a lot of benefit out of the concept right away.
May 01 2014
On Friday, 2 May 2014 at 06:51:49 UTC, deadalnix wrote:First the original post I made on this forum : http://forum.dlang.org/thread/kluaojijixhwigoujeip forum.dlang.org#post-kluaojijixhwigoujeip:40forum.dlang.org Now some sample code/explanation to get it better. for D. It introduces a new qualifier. The qualifier is necessary on function signatures (inference is possible to some extent, like for pure functions), object/struct fields and globals. It is inferred on local variables (but can be explicited if one wish). An isolated is in an 'island'. The 'island' is implicit and tracked by the compiler. We have one immutable and one shared island. We have one TL island per thread. And we can have an infinity of isolated island. An isolated is consumed when: - it is returned - it is passed as argument - it is assigned to another island When an isolated goes out of scope without being consumed, the compiler can free the whole island: void foo() { A a = new A(); // a is isolated if A's ctor allows. // a is not consumed. The compiler can insert memory freeing. } As we can see, it allows the compiler to do some freeing for us to reduce GC pressure. Manu why aren't you already here cheering ? When an isolated is consumed, the island it is in is merged into the island that consumes it. All reference to the island become write only until next write. void foo() { A a = new A(); // a is isolated immutable b = a; // a's island is merged into immutable // a's island has been consumed. a is not readable at this point. // a.foo() // Error a = new A(); // OK a.foo(); // OK, we have written in a. } So far, we saw that isolated helps to construct const/immutable/shared objects and can be used by the compiler to insert free. isolated also help to bridge the RC world and the GC world. struct RC(T) is(isReferenceType!T) { private T mystuff; this(isolated T stuff) { mystuff = stuff; } // All code to do ref counting goes here... } Here, the RC struct must be constructed with an isolated. To do so, the isolated have to be passed as argument: it is consumed. As a result, the RC struct can be sure that it has the only usable reference to stuff that is around. Now some concurrency goodies: void foo() { auto tid = spawn(&spawnedFunc, thisTid); A a = new A(); send(tid, a); // OK, a is an isolated. // a can't be used here anymore as it is consumed. // You can simply pass isolated around across thread safely. } Now that is pretty sweet. First we don't have to do the crazy and unsafe dance of "cast to shared and cast back to me" and this is actually safe. Go's type system do is not safe across channel, that puts us ahead in one of the things go does best. std.parallelism can also benefit from this. There is also several benefit when the optimizer knows about isolated (different island do not alias each other). I hope the idea get across better with some sample code and will be considered. As sample code shows, isolated do not need to be specified explicitly often. User not annotating can get a lot of benefit out of the concept right away.I've been thinking about something similar as being an 'obvious' idea. It to me seems rather a good idea.
May 02 2014
On 5/1/2014 11:51 PM, deadalnix wrote:First the original post I made on this forum : http://forum.dlang.org/thread/kluaojijixhwigoujeip forum.dlang.org#post-kluaojijixhwigoujeip:40forum.dlang.orgIt's nearly the same as http://wiki.dlang.org/DIP29 except that DIP29 tries to use a library type to take the role of 'isolated' in your proposal.
May 02 2014
On Friday, 2 May 2014 at 09:05:13 UTC, Walter Bright wrote:On 5/1/2014 11:51 PM, deadalnix wrote:DIP29 is not safe, does not help to construct immutables, is not inferred, do not provide aliasing infos for the optimizer, etc...First the original post I made on this forum : http://forum.dlang.org/thread/kluaojijixhwigoujeip forum.dlang.org#post-kluaojijixhwigoujeip:40forum.dlang.orgIt's nearly the same as http://wiki.dlang.org/DIP29 except that DIP29 tries to use a library type to take the role of 'isolated' in your proposal.
May 02 2014
On 5/2/2014 2:24 AM, deadalnix wrote:DIP29 is not safe,How so?does not help to construct immutables,immutable p = new int; works.is not inferred,I'm surprised you'd say that, most of it is about inferring uniqueness.do not provide aliasing infos for the optimizer,That's right.etc...?
May 02 2014
On 5/2/14, 2:24 AM, deadalnix wrote:On Friday, 2 May 2014 at 09:05:13 UTC, Walter Bright wrote:I think a more detailed comparison would do well here. My understanding of DIP29 is quite at odds with these claims. AndreiOn 5/1/2014 11:51 PM, deadalnix wrote:DIP29 is not safe, does not help to construct immutables, is not inferred, do not provide aliasing infos for the optimizer, etc...First the original post I made on this forum : http://forum.dlang.org/thread/kluaojijixhwigoujeip forum.dlang.org#post-kluaojijixhwigoujeip:40forum.dlang.orgIt's nearly the same as http://wiki.dlang.org/DIP29 except that DIP29 tries to use a library type to take the role of 'isolated' in your proposal.
May 02 2014
I like it a lot, as it solves several problems elegantly! Some comments inline... On Friday, 2 May 2014 at 06:51:49 UTC, deadalnix wrote:An isolated is consumed when: - it is returned - it is passed as argument - it is assigned to another islandAssignment and passing to a scope variable can be exempt from this rule.When an isolated goes out of scope without being consumed, the compiler can free the whole island: void foo() { A a = new A(); // a is isolated if A's ctor allows.I guess the condition is that assignment to isolated is allowed only from a unique expression. Thanks to Walter's recent work, this is now inferred in many cases. But I guess in cases where it cannot be inferred (.di files come to mind), it needs to be annotated explicitly: class A { this() isolated; }// a is not consumed. The compiler can insert memory freeing. } As we can see, it allows the compiler to do some freeing for us to reduce GC pressure. Manu why aren't you already here cheering ?To make this more useful, turn it into a requirement. It gets us deterministic destruction for reference types. Example: ... isolated tmp = new Tempfile(); // use tmp ... // tmp is guaranteed to get cleaned up }When an isolated is consumed, the island it is in is merged into the island that consumes it. All reference to the island become write only until next write. void foo() { A a = new A(); // a is isolated immutable b = a; // a's island is merged into immutable // a's island has been consumed. a is not readable at this point. // a.foo() // Error a = new A(); // OK a.foo(); // OK, we have written in a. }This needs more elaboration. The problem is control flow: isolated a = new A(); if(...) { immutable b = a; ... } a.foo(); // <-- ??? (Similar for loops and gotos.) There are several possibilities: 1) isolateds must be consumed either in every branch or in no branch, and this is statically enforced by the compiler. 2) It's just "forbidden", but the compiler doesn't guarantee it except where it can. 3) The compiler inserts a hidden variable to track the status of the isolated, and asserts if it is used while it's in an invalid state. This can be elided if it can be proven to be unnecessary. I would prefer 3), as it is the most flexible. I also believe a similar runtime check is done in other situations (to guard against return of locals in safe code, IIRC).I hope the idea get across better with some sample code and will be considered. As sample code shows, isolated do not need to be specified explicitly often. User not annotating can get a lot of benefit out of the concept right away.That's true, but it is also a breaking change, because then suddenly some variables aren't writable anymore (or alternatively, the compiler would have to analyse all future uses of the variable first to see whether it can be inferred isolated, if that's even possible in the general case). I believe it's fine if explicit annotation is required.
May 02 2014
On Friday, 2 May 2014 at 09:41:48 UTC, Marc Schütz wrote:To make this more useful, turn it into a requirement. It gets us deterministic destruction for reference types. Example: ... isolated tmp = new Tempfile(); // use tmp ... // tmp is guaranteed to get cleaned up }No because...This needs more elaboration. The problem is control flow: isolated a = new A(); if(...) { immutable b = a; ... } a.foo(); // <-- ??? (Similar for loops and gotos.) There are several possibilities:Of this.1) isolateds must be consumed either in every branch or in no branch, and this is statically enforced by the compiler. 2) It's just "forbidden", but the compiler doesn't guarantee it except where it can. 3) The compiler inserts a hidden variable to track the status of the isolated, and asserts if it is used while it's in an invalid state. This can be elided if it can be proven to be unnecessary.These solutions are all unnecessary restrictive. If the variable may be consumed it is consumed. This is a problem solved for ages for non nullables, there is no need to brainstorm here.I would prefer 3), as it is the most flexible. I also believe a similar runtime check is done in other situations (to guard against return of locals in safe code, IIRC).3 is idiotic as the compiler can't ensure anything at compile time. Random failure at runtime for valid code is not desirable.
May 02 2014
On Friday, 2 May 2014 at 17:46:54 UTC, deadalnix wrote:On Friday, 2 May 2014 at 09:41:48 UTC, Marc Schütz wrote:I think the situation is different here. For nullables, you wouldn't gain much by more precise tracking. For isolated, as noted, we'd gain deterministic lifetimes for reference types. This is IMO important enough to accept a few minor complications (which are anyway solvable). There would be no more need for the unsafe std.typecons.scoped, which only works for classes anyway, but not slices or pointers.To make this more useful, turn it into a requirement. It gets us deterministic destruction for reference types. Example: ... isolated tmp = new Tempfile(); // use tmp ... // tmp is guaranteed to get cleaned up }No because...This needs more elaboration. The problem is control flow: isolated a = new A(); if(...) { immutable b = a; ... } a.foo(); // <-- ??? (Similar for loops and gotos.) There are several possibilities:Of this.1) isolateds must be consumed either in every branch or in no branch, and this is statically enforced by the compiler. 2) It's just "forbidden", but the compiler doesn't guarantee it except where it can. 3) The compiler inserts a hidden variable to track the status of the isolated, and asserts if it is used while it's in an invalid state. This can be elided if it can be proven to be unnecessary.These solutions are all unnecessary restrictive. If the variable may be consumed it is consumed. This is a problem solved for ages for non nullables, there is no need to brainstorm here.You can in most cases. Runtime failure is only for the cases where it's not possible.I would prefer 3), as it is the most flexible. I also believe a similar runtime check is done in other situations (to guard against return of locals in safe code, IIRC).3 is idiotic as the compiler can't ensure anything at compile time.Random failure at runtime for valid code is not desirable.It's not random, and the code was _not_ valid: it tried to use a consumed isolated.
May 02 2014
On Friday, 2 May 2014 at 09:41:48 UTC, Marc Schütz wrote:That's true, but it is also a breaking change, because then suddenly some variables aren't writable anymore (or alternatively, the compiler would have to analyse all future uses of the variable first to see whether it can be inferred isolated, if that's even possible in the general case). I believe it's fine if explicit annotation is required.No, I expect the compiler to backtrack inference when it hits an error, not to infer eagerly, because indeed, the eager inference would be a breaking change.
May 02 2014
On Friday, 2 May 2014 at 18:10:42 UTC, deadalnix wrote:On Friday, 2 May 2014 at 09:41:48 UTC, Marc Schütz wrote:s/writable/readable/ of courseThat's true, but it is also a breaking change, because then suddenly some variables aren't writable anymore (orThis might work, but would require defining an order of evaluation for static if's &co, because you could create logical cycles otherwise.alternatively, the compiler would have to analyse all future uses of the variable first to see whether it can be inferred isolated, if that's even possible in the general case). I believe it's fine if explicit annotation is required.No, I expect the compiler to backtrack inference when it hits an error, not to infer eagerly, because indeed, the eager inference would be a breaking change.
May 02 2014
On Friday, 2 May 2014 at 20:10:04 UTC, Marc Schütz wrote:On Friday, 2 May 2014 at 18:10:42 UTC, deadalnix wrote:Yes.On Friday, 2 May 2014 at 09:41:48 UTC, Marc Schütz wrote:s/writable/readable/ of courseThat's true, but it is also a breaking change, because then suddenly some variables aren't writable anymore (orYes, but this is unrelated to isolated. In fact this is already the case. static if is not deterministic. I've made a proposal to improve the situation: http://wiki.dlang.org/DIP31 But to be fair I'm not quite satisfied. This still leave some room for unspecified results, but is a great improvement over current situation.This might work, but would require defining an order of evaluation for static if's &co, because you could create logical cycles otherwise.alternatively, the compiler would have to analyse all future uses of the variable first to see whether it can be inferred isolated, if that's even possible in the general case). I believe it's fine if explicit annotation is required.No, I expect the compiler to backtrack inference when it hits an error, not to infer eagerly, because indeed, the eager inference would be a breaking change.
May 02 2014
On Fri, 02 May 2014 02:51:47 -0400, deadalnix <deadalnix gmail.com> wrote:First the original post I made on this forum : http://forum.dlang.org/thread/kluaojijixhwigoujeip forum.dlang.org#post-kluaojijixhwigoujeip:40forum.dlang.org Now some sample code/explanation to get it better. introduces a new qualifier. The qualifier is necessary on function signatures (inference is possible to some extent, like for pure functions), object/struct fields and globals. It is inferred on local variables (but can be explicited if one wish). An isolated is in an 'island'. The 'island' is implicit and tracked by the compiler. We have one immutable and one shared island. We have one TL island per thread. And we can have an infinity of isolated island. An isolated is consumed when: - it is returned - it is passed as argument - it is assigned to another island When an isolated goes out of scope without being consumed, the compiler can free the whole island: void foo() { A a = new A(); // a is isolated if A's ctor allows. // a is not consumed. The compiler can insert memory freeing. } As we can see, it allows the compiler to do some freeing for us to reduce GC pressure. Manu why aren't you already here cheering ? When an isolated is consumed, the island it is in is merged into the island that consumes it. All reference to the island become write only until next write. void foo() { A a = new A(); // a is isolated immutable b = a; // a's island is merged into immutable // a's island has been consumed. a is not readable at this point.When you say "consumed", you mean it statically cannot be used, not that it was set to null or something, right?// a.foo() // Error a = new A(); // OK a.foo(); // OK, we have written in a.OK, but what happens if I do this? auto c = a; a.foo(); At this point, a was inferred typed as isolated, but in this section, it doesn't have to be. Will the type change in different parts of the function? Or will this simply be statically disallowed?}So far, we saw that isolated helps to construct const/immutable/shared objects and can be used by the compiler to insert free. isolated also help to bridge the RC world and the GC world. struct RC(T) is(isReferenceType!T) { private T mystuff; this(isolated T stuff) { mystuff = stuff; } // All code to do ref counting goes here... } Here, the RC struct must be constructed with an isolated. To do so, the isolated have to be passed as argument: it is consumed. As a result, the RC struct can be sure that it has the only usable reference to stuff that is around.mystuff should be marked isolated too, no?Now some concurrency goodies: void foo() { auto tid = spawn(&spawnedFunc, thisTid); A a = new A(); send(tid, a); // OK, a is an isolated. // a can't be used here anymore as it is consumed. // You can simply pass isolated around across thread safely. } Now that is pretty sweet. First we don't have to do the crazy and unsafe dance of "cast to shared and cast back to me" and this is actually safe. Go's type system do is not safe across channel, that puts us ahead in one of the things go does best.Yes, I really like this idea. This is what is missing from the type system. I read in your previous post that shared can benefit from using isolated, by not having to lock all sub-objects. But I'm confused as to how that would work. If a variable is shared, by default, it can be passed around. But wouldn't the passing of the shared variable mean you have to disallow access to the isolated member? The compiler would have to be aware of the locking protection and enforce it. Essentially, you could access a locked isolated variable, but not a shared isolated variable. These kinds of requirements need to be specified somehow. Also, when you call a method, how does the island get handled? abstract class A { void foo(ref A other) { other = this;} void bar(int n); } class B { private isolated int x; void bar(int n) { x = n; } // I'm assuming this is ok, right? } auto b = new B; A a; b.foo(a); At this point, both b and a refer to the same object. How does the compiler know what to prevent here? At what point does the island become inaccessible? Another issue is with delegates. They have no reliable type for the context pointer. -Steve
May 02 2014
On Friday, 2 May 2014 at 06:51:49 UTC, deadalnix wrote:An isolated is consumed when: - it is returned - it is passed as argument - it is assigned to another islandThis will not work well with UFCS. isolated int[] ints = new int[](10); //put looks like a member function, but //this desugars to put(ints, 3), so ints //is consumed ints.put(3);
May 02 2014
On Fri, 02 May 2014 09:50:07 -0400, Meta <jared771 gmail.com> wrote:On Friday, 2 May 2014 at 06:51:49 UTC, deadalnix wrote:Some interaction with pure would be in order. I don't think ints.put(3) should consume the ints. However, you have just lost the data you put, since ints cannot have any other reference to it! -SteveAn isolated is consumed when: - it is returned - it is passed as argument - it is assigned to another islandThis will not work well with UFCS. isolated int[] ints = new int[](10); //put looks like a member function, but //this desugars to put(ints, 3), so ints //is consumed ints.put(3);
May 02 2014
Correct me if I'm wrong here, but this seems really similar to how Rust does owned pointers and move semantics. Or is there a large conceptual difference between the two that I'm missing? I believe that the issues that people are bringing up with bad interaction with UFCS, and losing isolated data after passing it to a function, are managed in Rust with the notion of a borrowed pointers. Perhaps something analogous to this could accompany the `isolated` idea?
May 02 2014
On Friday, 2 May 2014 at 18:32:13 UTC, Dylan Knutson wrote:Correct me if I'm wrong here, but this seems really similar to how Rust does owned pointers and move semantics. Or is there a large conceptual difference between the two that I'm missing? I believe that the issues that people are bringing up with bad interaction with UFCS, and losing isolated data after passing it to a function, are managed in Rust with the notion of a borrowed pointers. Perhaps something analogous to this could accompany the `isolated` idea?I don't think bolting Rust's type system onto D is a viable option at this point.
May 02 2014
On Friday, 2 May 2014 at 18:32:13 UTC, Dylan Knutson wrote:Correct me if I'm wrong here, but this seems really similar to how Rust does owned pointers and move semantics. Or is there a large conceptual difference between the two that I'm missing?Yes, there are some parallels, although there's no "merging of islands" in Rust, AFAIK.I believe that the issues that people are bringing up with bad interaction with UFCS, and losing isolated data after passing it to a function, are managed in Rust with the notion of a borrowed pointers. Perhaps something analogous to this could accompany the `isolated` idea?This will definitely need more thought. I also don't think that UFCS is special; methods should probably be just treated like free functions with a hidden parameter, just as they are for pure. Purity might be one part of the solution here: pure functions can for example take isolated arguments without consuming them, iff their parameters and the isolated variable have "incompatible" types in that it mustn't be possible to store a reference to one in the other. (The same is true for scoped variables: they cannot escape their scope, even if the pure function's params are not marked as scope.) Another option are scope/in parameters. All of this purity, isolated, scope, uniqueness business is closely intertwined... there just needs to be an elegant way to make it all fit together.
May 02 2014
On Friday, 2 May 2014 at 18:32:13 UTC, Dylan Knutson wrote:Correct me if I'm wrong here, but this seems really similar to how Rust does owned pointers and move semantics. Or is there a large conceptual difference between the two that I'm missing?There is some similarity, but Rust system has a bit more capabilities. These extra capability come at great increase in complexity, so I don't think it is worth it.I believe that the issues that people are bringing up with bad interaction with UFCS, and losing isolated data after passing it to a function, are managed in Rust with the notion of a borrowed pointers. Perhaps something analogous to this could accompany the `isolated` idea?Yes, rust handle this with burrowed pointers. You can also handle this by : - Passing data back and forth (via argument, and then return it so the callee get it back). - Using a wrapper of some kind. I don't think getting all the menagerie of Rust pointer types is a good thing. They certainly allows for a lot, but once again, come at great complexity cost. If most of it can be achieved with much lower complexity, that is a win.
May 02 2014