www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - RFC: scope and borrowing

reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
In the "Opportunities for D" thread, Walter again mentioned the 
topics ref counting, GC, uniqueness, and borrowing, from which a 
lively discussion developed [1]. I took this thread as an 
opportunity to write down some ideas about these topics. The 
result is a rather extensive proposal for the implementation of 
borrowing, and its implementations:

http://wiki.dlang.org/User:Schuetzm/scope

This is not a real DIP, but before I put more work into 
formalizing it, I'd like to hear some thoughts from the languages 
gurus here:

* Is this the general direction we want to go? Is it acceptable 
in general?
* Is the proposal internally consistent?
* How big would the effort to implement it be? (I suspect it's a 
large amount of work, but relatively straightforward.)

[1] http://forum.dlang.org/thread/lphnen$1ml7$1 digitalmars.com
Aug 24 2014
next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
Cool initiative!

It is probably a good idea to look at what other languages with 
linear type systems are doing:

http://en.m.wikipedia.org/wiki/Substructural_type_system

My gut feeling is that borrowing in D will suffer the same 
problems as  safe without a high level IR that is proven correct. 
Meaning, if the IR is correct you can add new language features 
without breaking the type system.
Aug 24 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 25 August 2014 at 01:09:59 UTC, Ola Fosheim Grøstad 
wrote:

 Cool initiative!

 It is probably a good idea to look at what other languages with 
 linear type systems are doing:

 http://en.m.wikipedia.org/wiki/Substructural_type_system


The idea is mostly inspired by Rust, but it came out very 
different. Rust's borrowing rules are way more complicated. It 
also has to deal with move semantics, for example. And it 
annotates lifetimes, not owners, and thereby gets 
non-hierarchical dependencies, as far as I understand it. I 
believe this proposal is close to the best we can achieve without 
resorting to data flow analysis.


 My gut feeling is that borrowing in D will suffer the same 
 problems as  safe without a high level IR that is proven 
 correct. Meaning, if the IR is correct you can add new language 
 features without breaking the type system.


I'm unfortunately not familiar with the theoretical foundations 
of type systems, and formal logic. So I'm not sure what exactly 
you mean here. It seems to me the problems with  safe are with 
the way it was implemented (allow everything _except_ certain 
things that are known to be bad, instead of the opposite way: 
allow nothing at first, but lift restrictions where it's safe to 
do so), they are not conceptual.

But if you see potential problems, or know a way to avoid them, 
this is exactly the kind of thing I'd like to see discussed.
Aug 25 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 25 August 2014 at 11:09:48 UTC, Marc Schütz wrote:

 I believe this proposal is close to the best we can achieve 
 without resorting to data flow analysis.


I agree with bearophile, perhaps data flow analysis would be 
desirable. So I think it would be a good idea to hold the door 
open on that.


 I'm unfortunately not familiar with the theoretical foundations 
 of type systems, and formal logic. So I'm not sure what exactly


I don't know all that much about linear type systems, but this is 
an opportunity to learn more for all of us! :-)


 you mean here. It seems to me the problems with  safe are with 
 the way it was implemented (allow everything _except_ certain 
 things that are known to be bad, instead of the opposite way: 
 allow nothing at first, but lift restrictions where it's safe 
 to do so), they are not conceptual.


Because real programming languages are hard to reason about it is 
difficult to say where things break down. So one usually will map 
the constructs of the language onto something simpler and more 
homogeneous that is easier to reason about.

When it comes to  safe I think the main problem is that D makes 
decisions on constructs and not on the "final semantics" of the 
program. If you have a dedicated high level IR you can accept any 
program segment that can be proven to be memory safe in terms of 
the IR. The point is to have an IR that is less complicated than 
the full language, but that retains needed information that is 
lost in a low level IR and which you need to prove memory safety.

memset() is not unsafe per se, it is unsafe with the wrong 
parameters. So you have to prove that the parameters are in the 
safe region. Same thing with freeing memory and borrowed pointers 
etc. You need a correctness proof even if you give up on 
generality.

You may reject many safe programs but at least verify as many 
simple safe programs as you can.

A bonus of having a high level IR is that you more easily can 
combine languages with fewer interfacing problems. That would be 
an advantage if you want a DSL to cooperate with D.


 But if you see potential problems, or know a way to avoid them, 
 this is exactly the kind of thing I'd like to see discussed.


The problem is that D is too complex to reason about with any 
reasonable level of confidence. So you need to reduce the 
language to a level where you can reason about it with confidence 
and build the other constructs on top of that.

(That way you don't have to reason about the combinatorial 
explosion of constructs, just the building blocks).

Ola.
Aug 25 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 25 August 2014 at 15:38:09 UTC, Ola Fosheim Grøstad 
wrote:

 On Monday, 25 August 2014 at 11:09:48 UTC, Marc Schütz wrote:

 I believe this proposal is close to the best we can achieve 
 without resorting to data flow analysis.


 I agree with bearophile, perhaps data flow analysis would be 
 desirable. So I think it would be a good idea to hold the door 
 open on that.


I definitely don't want to exclude anything. But we need to find 
out whether the additional complexity of full-blown DFA is really 
necessary, see my reply to bearophile.


 I'm unfortunately not familiar with the theoretical 
 foundations of type systems, and formal logic. So I'm not sure 
 what exactly


 I don't know all that much about linear type systems, but this 
 is an opportunity to learn more for all of us! :-)


 you mean here. It seems to me the problems with  safe are with 
 the way it was implemented (allow everything _except_ certain 
 things that are known to be bad, instead of the opposite way: 
 allow nothing at first, but lift restrictions where it's safe 
 to do so), they are not conceptual.


 Because real programming languages are hard to reason about it 
 is difficult to say where things break down. So one usually 
 will map the constructs of the language onto something simpler 
 and more homogeneous that is easier to reason about.

 When it comes to  safe I think the main problem is that D makes 
 decisions on constructs and not on the "final semantics" of the 
 program. If you have a dedicated high level IR you can accept 
 any program segment that can be proven to be memory safe in 
 terms of the IR. The point is to have an IR that is less 
 complicated than the full language, but that retains needed 
 information that is lost in a low level IR and which you need 
 to prove memory safety.

 memset() is not unsafe per se, it is unsafe with the wrong 
 parameters. So you have to prove that the parameters are in the 
 safe region. Same thing with freeing memory and borrowed 
 pointers etc. You need a correctness proof even if you give up 
 on generality.

 You may reject many safe programs but at least verify as many 
 simple safe programs as you can.

 A bonus of having a high level IR is that you more easily can 
 combine languages with fewer interfacing problems. That would 
 be an advantage if you want a DSL to cooperate with D.


But this would require knowledge about the inner workings of 
memset() to be part of the IR, or memset() to be implemented in 
it. The same IR (or an equivalent one) would then need to be part 
of language specification, otherwise different compilers would 
allow different operations.

IMO the general idea of the current design is not okay, because 
it's easy to implement, and easy to specify (a whitelist, or as 
currently, blacklist approach). If something has been overlooked, 
it can be added incrementally; at the same time there's always 
 trusted to let the programmer specify what the compiler can't 
prove.
Aug 25 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 25 August 2014 at 18:14:03 UTC, Marc Schütz wrote:

 But this would require knowledge about the inner workings of 
 memset() to be part of the IR, or memset() to be implemented in 
 it.


The latter. You might have "zerofill(ptr,length)" and turn it 
into "clear(data)" if it is  safe.


 The same IR (or an equivalent one) would then need to be part 
 of language specification, otherwise different compilers would 
 allow different operations.


I think you could have a low performance minimum requirement 
reference implementation for this between the front-end and the 
back-end.

As in, not written for performance, but as a baseline for testing 
the real implementation.


 currently, blacklist approach). If something has been 
 overlooked, it can be added incrementally; at the same time 
 there's always  trusted to let the programmer specify what the 
 compiler can't prove.


I think borrow,  safe and other types of correctness oriented 
analysis aspects should be seen as two aspects of the same thing. 
So it is desirable to design it as a whole IMO.

Especially if assert() is going to turn into assume(). Which I 
believe will only work reliably for preconditions on a code-scope 
up to the postconditions, but I believe you need some heavy duty 
analysis of the call-chain to get anywhere.

I presume you can do the same for allocations between 
preconditions and postconditions and that could let you establish 
 safe for a wider range of constructs this way.

I.e. the work on assert->assume for preconditions could lead to a 
wider range of  safe and also perhaps automatic memory allocation 
optimizations.

It sounds reasonable that constraining the input sometimes would 
give information that could help on establishing  safe on code 
that would otherwise have to be assume unsafe.

Ola.
Aug 25 2014
prev sibling next sibling parent "Dominikus Dittes Scherkl" writes:
On Sunday, 24 August 2014 at 13:14:45 UTC, Marc Schütz wrote:

 http://wiki.dlang.org/User:Schuetzm/scope


I like this.
It removes pretty much all of the need for manual memory 
management for me. I would love to see this implemented.
Aug 25 2014
prev sibling next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 25/08/2014 1:14 a.m., "Marc Schütz" <schuetzm gmx.net>" wrote:

 In the "Opportunities for D" thread, Walter again mentioned the topics
 ref counting, GC, uniqueness, and borrowing, from which a lively
 discussion developed [1]. I took this thread as an opportunity to write
 down some ideas about these topics. The result is a rather extensive
 proposal for the implementation of borrowing, and its implementations:

 http://wiki.dlang.org/User:Schuetzm/scope

 This is not a real DIP, but before I put more work into formalizing it,
 I'd like to hear some thoughts from the languages gurus here:

 * Is this the general direction we want to go? Is it acceptable in general?
 * Is the proposal internally consistent?
 * How big would the effort to implement it be? (I suspect it's a large
 amount of work, but relatively straightforward.)

 [1] http://forum.dlang.org/thread/lphnen$1ml7$1 digitalmars.com


Have you considered what happens when you cast away scope?
I didn't read anything about that, unless I missed something.

I do have to ask this, because what if you wanted to optionally return a 
scoped reference?
Aug 25 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 25 August 2014 at 10:06:12 UTC, Rikki Cattermole wrote:

 Have you considered what happens when you cast away scope?
 I didn't read anything about that, unless I missed something.


Right, this should be specified. But it's fairly obvious what 
would happen: you lose the safety guarantees (=>  system), and 
you're responsible to make sure you don't get dangling 
references. You'd basically get what you have with normal 
pointers. I don't think it has any deeper implications than that, 
but I'm not sure there aren't any obscure optimizations the 
compiler could make based on scope. If there are, it's no 
different from casting away const, in this respect.


 I do have to ask this, because what if you wanted to optionally 
 return a scoped reference?


Well, it's part of the return type, so it's either scoped, or it 
isn't. You could return a tuple with a scoped and a non-scoped 
pointer, though.

If it's only about returning a non-scoped pointer from a function 
with a scoped return type, that's fine, because it's adding 
scope, not removing it. (Use case: ScopeBuffer.)
Aug 25 2014
prev sibling next sibling parent reply "Kagamin" <spam here.lot> writes:
Can't scope(int*) on return type be equivalent to 
scope!(a,b)(int*)? How often this is not desired?
Aug 25 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 25 August 2014 at 10:27:53 UTC, Kagamin wrote:

 Can't scope(int*) on return type be equivalent to 
 scope!(a,b)(int*)? How often this is not desired?


You mean as a default?

It would be desired in `chooseStringAtRandom`, but not in the 
`findSubstring`, whose returned string shouldn't be limited by 
the scope of the needle. If it is made the default, there would 
need to be a way to opt out, such as removing an owner.

But note that in the `chooseStringAtRandom` example, it would be 
sufficient to declare it as:

     scope chooseStringAtRandom(scope(string) a, scope(string) b) {
         return random() % 2 == 0 ? a : b;
     }

Type deduction would then automatically add `a` and `b` as the 
owners, by the rules under "Owner tracking".
Aug 25 2014
parent "Kagamin" <spam here.lot> writes:
On Monday, 25 August 2014 at 11:48:18 UTC, Marc Schütz wrote:

 It would be desired in `chooseStringAtRandom`, but not in the 
 `findSubstring`, whose returned string shouldn't be limited by 
 the scope of the needle. If it is made the default, there would 
 need to be a way to opt out, such as removing an owner.


If `in` will be defined as scope!callee, i.e. can't be returned, 
then it can be removed from the return type.
Sep 21 2014
prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Marc Schütz:


 http://wiki.dlang.org/User:Schuetzm/scope


It looks nice. But perhaps it needs some kind of proof of 
correctness.

Have you read the old blog posts (written before the creation of 
Rust) by Bartosz Milewski regarding the borrowing in D?



Implementation of this feature is possible without doing flow 
control or interprocedural analysis.<


I remember that Walter has recently said that he's willing to add 
some kind of flow analysis to the D front-end.

Bye,
bearophile
Aug 25 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 25 August 2014 at 15:09:32 UTC, bearophile wrote:

 Marc Schütz:


 http://wiki.dlang.org/User:Schuetzm/scope


 It looks nice. But perhaps it needs some kind of proof of 
 correctness.


Hmm... First there's the assignment rules. They make sure that 
nothing with a shorter lifetime ends up in a variable with a 
longer lifetime designation. The other part is to proof that the 
type deduction and argument matching rules work.

Both parts are not difficult to reason about, but I don't know 
what a formal proof needs to look like exactly. (How formal do we 
need to go?)


 Have you read the old blog posts (written before the creation 
 of Rust) by Bartosz Milewski regarding the borrowing in D?


No, can you point me to them? I couldn't find them on his blog 
under http://bartoszmilewski.com/category/d-programming-language/
There are some posts about ownership and regions, but only in the 
context of multi-threading. I'm afraid this wouldn't easily fit 
into a hierarchical system like I have in mind.


Implementation of this feature is possible without doing flow 
control or interprocedural analysis.<


 I remember that Walter has recently said that he's willing to 
 add some kind of flow analysis to the D front-end.


Interesting. The question is: is it worth it? Maybe we can 
already cover 99% of the use cases with a simpler construct. The 
concept needs to be understandable for the users of the language, 
too. And maybe "some kind of flow analysis" just isn't enough to 
get a significant improvement, maybe it would need whole-program 
analysis...
Aug 25 2014
parent "bearophile" <bearophileHUGS lycos.com> writes:
Marc Schütz:


(How formal do we need to go?)


It's not just a matter of how much formal to go, but also a 
matter of how much D to formalize to avoid unwanted interactions 
(surprises) later.

But before going formal, you need to wait for comments from 
Walter & Andrei; and possibly also comments from a good Rust core 
developer, because they have discussed such topics in detail for 
lot of time :-)

Bye,
bearophile
Aug 25 2014
prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 24 August 2014 23:14, via Digitalmars-d <digitalmars-d puremagic.com>
wrote:


 In the "Opportunities for D" thread, Walter again mentioned the topics ref
 counting, GC, uniqueness, and borrowing, from which a lively discussion
 developed [1]. I took this thread as an opportunity to write down some
 ideas about these topics. The result is a rather extensive proposal for the
 implementation of borrowing, and its implementations:

 http://wiki.dlang.org/User:Schuetzm/scope

 This is not a real DIP, but before I put more work into formalizing it,
 I'd like to hear some thoughts from the languages gurus here:

 * Is this the general direction we want to go? Is it acceptable in general?
 * Is the proposal internally consistent?
 * How big would the effort to implement it be? (I suspect it's a large
 amount of work, but relatively straightforward.)

 [1] http://forum.dlang.org/thread/lphnen$1ml7$1 digitalmars.com


This is the initiative I've been dreaming of for years!
I'll give it a critical review when I have some time. But in the mean time
at first glance, I just wanted to say, very nice work! :)
This is largely inline with my thoughts since I first came to D, except you
flesh out a few problem areas (return scope, which we discussed at length
at dconf2013), and the ideas appear to be quite sound.

This direction really opens up the language to support manual and
user-defined memory management at a much deeper and more useful level.
In my experience, manual/custom memory management depends extensively on
templates, and also implies that practically every api in any library
anywhere must also receive template args, such that it can receive objects
templated with custom allocation strategies.
D is a language that is acutely susceptible to over-templatisation and
extreme template bloat. This work on scope is critically important to
mitigate that tendency of D in library code.
Aug 26 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 26 August 2014 at 11:47:39 UTC, Manu via 
Digitalmars-d wrote:

 This is the initiative I've been dreaming of for years!
 I'll give it a critical review when I have some time. But in 
 the mean time
 at first glance, I just wanted to say, very nice work! :)


I had a suspicion you would like it ;-)


 This is largely inline with my thoughts since I first came to 
 D, except you
 flesh out a few problem areas (return scope, which we discussed 
 at length
 at dconf2013), and the ideas appear to be quite sound.


I've heard that mentioned a few times. I guess it was an informal 
discussion only, not an official talk, right? I only know about a 
few discussions afterwards on the news group.


 This direction really opens up the language to support manual 
 and
 user-defined memory management at a much deeper and more useful 
 level.
 In my experience, manual/custom memory management depends 
 extensively on
 templates, and also implies that practically every api in any 
 library
 anywhere must also receive template args, such that it can 
 receive objects
 templated with custom allocation strategies.
 D is a language that is acutely susceptible to 
 over-templatisation and
 extreme template bloat. This work on scope is critically 
 important to
 mitigate that tendency of D in library code.


I'm looking forward to hear your thoughts about it, because 
you're one of the people who are involved in large complex 
projects where this is relevant. For me, it is until now only a 
theoretical exercise, it's important to hear from someone who can 
assess how it would work out in practice.

I'm also curious about Walter and Andrei's opinion (I believe 
they are quite busy with other things at the moment, so we'll 
have to wait), and Kenji and others who can give an estimate 
about the implementation.
Aug 26 2014
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 24 Aug 2014 13:14:43 +0000
schrieb "Marc Sch=C3=BCtz" <schuetzm gmx.net>:


 In the "Opportunities for D" thread, Walter again mentioned the=20
 topics ref counting, GC, uniqueness, and borrowing, from which a=20
 lively discussion developed [1]. I took this thread as an=20
 opportunity to write down some ideas about these topics. The=20
 result is a rather extensive proposal for the implementation of=20
 borrowing, and its implementations:
=20
 http://wiki.dlang.org/User:Schuetzm/scope


The amount of possible use-cases you listed for this extension is
staggering. It surely carries its own weight.
scope!(ident1, ident2, ...) was quite clever. inout could
borrow from this.


 This is not a real DIP, but before I put more work into=20
 formalizing it, I'd like to hear some thoughts from the languages=20
 gurus here:
=20
 * Is this the general direction we want to go? Is it acceptable=20
 in general?
 * Is the proposal internally consistent?


Can anyone tell without actually implementing it? :)
You could try to formalize some error messages and how the
compiler's reasoning would go. What happens when I pass
identifiers to scope!(=E2=80=A6) return types?
- Can the compiler look up the identifiers' types and scopes
  in all cases?
- Will the compiler deduce the return type from these
  identifiers? E.g. "scope!(someString) ref getString();" will
  work like in your example? I don't think so, because the
  "lifetime identifiers" could be structs containing the
  returned type.
- What if we used incompatible types like
  "scope!(someString, someIntPtr)" there?
- What about variables?

  static int someInt =3D 32;
  string someString;
  scope!(someString, someInt) int* x;
  x =3D &someInt;

  Is the declaration of x in error? Strings don't contain
  integers unless unsafe casts are used, so why would they
  narrow the lifetime of an integer reference?

- Is it necessary to keep around all declared lifetime
  identifiers? In the snippet above (assuming it is valid), it
  looks like the shorter lived `someString' is enough to
  establish the semantics.


 * How big would the effort to implement it be? (I suspect it's a=20
 large amount of work, but relatively straightforward.)
=20
 [1] http://forum.dlang.org/thread/lphnen$1ml7$1 digitalmars.com


--=20
Marco
Aug 26 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 26 August 2014 at 22:53:30 UTC, Marco Leise wrote:

 Can anyone tell without actually implementing it? :)


I think this is the biggest problem with all scope proposals. 
While concept is very natural figuring out all small details is 
incredibly tricky and one can never be sure it works without 
trying in practice on large-ish project. Of course implementing 
it is no small feat either which is pretty much any progress in 
this direction is so slow.
Aug 26 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Marco Leise:


 The amount of possible use-cases you listed for this extension
 is staggering.


With scope management in code like this:


import std.stdio, std.algorithm;
int foo(in int[] a) {
     return sum([0, 1] ~ a[1 .. $ - 2] ~ 0 ~
                [a[$ - 1] + 1, a[$ - 1] + 2]);
}
void main() {
     int[5] b = [10, 20, 30, 40, 50];
     b.foo.writeln;
}


The compiler in theory could lower the code to something like 
this, removing all heap allocations for the array literals and 
the concatenations, allowing 'foo' to be annotated with  nogc (I 
don't know if Rust is able to do this, I presume it can):


import std.stdio, std.algorithm, core.stdc.stdlib;
T foo(T)(in T[] a)  nogc {
     immutable size_t L = 2 + a.length - 3 + 1 + 2;
     auto ptr = alloca(T.sizeof * L);
     if (ptr == null)
         exit(1); // Or throw a stack overflow error.
     T[] b = (cast(T*)ptr)[0 .. L];
     b[0] = 0;
     b[1] = 1;
     b[2 .. $ - 3] = a[1 .. $ - 2];
     b[$ - 3] = 0;
     b[$ - 2] = a[$ - 1] + 1;
     b[$ - 1] = a[$ - 1] + 2;
     return sum(b);

}
void main() {
     int[5] c = [10, 20, 30, 40, 50];
     c.foo.writeln;
}


But in some cases you don't want those arrays to be allocated on 
the stack because you know they are very large. So the [...]s 
array literal syntax becomes useful again:

import std.stdio, std.algorithm;
int foo(const scope int[] a) {
     return sum([0, 1]s ~ a[1 .. $ - 2] ~ 0 ~
                [a[$ - 1] + 1, a[$ - 1] + 2]s);
}


But this is still not enough, because even if those parts are all 
safely stack-allocated, the result of the concatenation is still 
not stack-allocated.


This could be enough:

import std.stdio, std.algorithm;
int foo(const scope int[] a)  nogc {
     auto[$] a2 = [0, 1]s ~ a[1 .. $ - 2] ~ 0 ~
                  [a[$ - 1] + 1, a[$ - 1] + 2]s;
     return sum(a2[]);
}

Bye,
bearophile
Aug 27 2014
prev sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 26 August 2014 at 22:53:30 UTC, Marco Leise wrote:

 You could try to formalize some error messages and how the
 compiler's reasoning would go. What happens when I pass
 identifiers to scope!(…) return types?
 - Can the compiler look up the identifiers' types and scopes
   in all cases?


It has to be able to. If a non-visible identifier is specified, 
it is an error.


 - Will the compiler deduce the return type from these
   identifiers? E.g. "scope!(someString) ref getString();" will
   work like in your example? I don't think so, because the
   "lifetime identifiers" could be structs containing the
   returned type.


I see, this was a stupid mistake. Of course this function needs 
to specify the full type, as no type deduction can happen when 
the body isn't available. Fixed in on the Wiki.


 - What if we used incompatible types like
   "scope!(someString, someIntPtr)" there?
 - What about variables?


Just to be sure: You can only specify variables as owners, not 
types. I've clarified this on the Wiki.


   static int someInt = 32;
   string someString;
   scope!(someString, someInt) int* x;
   x = &someInt;

   Is the declaration of x in error? Strings don't contain
   integers unless unsafe casts are used, so why would they
   narrow the lifetime of an integer reference?


This is an interesting thought... But I would still allow this 
for a different reason. At the beginning, I only thought about 
`scope` as a tool for memory management, but I believe it can be 
applied for lifetime management of arbitrary resources. Let's 
slightly modify your example, and use different types:

     Task someProcess;
     scope!someProcess HANDLE my_handle;

`someProcess` could represent an external process that is managed 
by this program, and `my_handle` could refer to some kind of 
object in this external process. This handle is only valid as 
long as the process exists, even though it is not a memory 
reference, and `Task` might not contain any members of type 
`HANDLE`. (This is not an ideal example, of course, because the 
process could terminate for reasons outside of our control.) A 
similar example would be a connection to the X server, and a 
handle to an object allocated from it.

I already wrote this idea into the section "scope for 
non-reference types", with a simpler example. In fact, I believe 
that the entire proposal will be a bit simpler if 
references/pointers aren't treated specially.


 - Is it necessary to keep around all declared lifetime
   identifiers? In the snippet above (assuming it is valid), it
   looks like the shorter lived `someString' is enough to
   establish the semantics.


Yes, it's possible to some degree, see the section 
"Considerations for the implementation". Unfortunately, this 
doesn't work with function declarations, and is incompatible with 
the suggested `scope!(const ...)` extension.
Aug 27 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 24/08/14 15:14, "Marc Schütz" <schuetzm gmx.net>" wrote:

 In the "Opportunities for D" thread, Walter again mentioned the topics
 ref counting, GC, uniqueness, and borrowing, from which a lively
 discussion developed [1]. I took this thread as an opportunity to write
 down some ideas about these topics. The result is a rather extensive
 proposal for the implementation of borrowing, and its implementations:

 http://wiki.dlang.org/User:Schuetzm/scope


I assume with this proposal it should be safe to do more stack 
allocations and have the compiler verify references don't escape the 
scope. Would there be a good idea to and a new function, besides the 
destructor, that will be called for variables declared as "scope" when 
they go out of scope.

The problem with destructors are that they can be called both when an 
object is deleted by the GC and when an object goes of out scope.

-- 
/Jacob Carlborg
Aug 27 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 28 August 2014 at 06:52:31 UTC, Jacob Carlborg wrote:

 On 24/08/14 15:14, "Marc Schütz" <schuetzm gmx.net>" wrote:

 In the "Opportunities for D" thread, Walter again mentioned 
 the topics
 ref counting, GC, uniqueness, and borrowing, from which a 
 lively
 discussion developed [1]. I took this thread as an opportunity 
 to write
 down some ideas about these topics. The result is a rather 
 extensive
 proposal for the implementation of borrowing, and its 
 implementations:

 http://wiki.dlang.org/User:Schuetzm/scope


 I assume with this proposal it should be safe to do more stack 
 allocations and have the compiler verify references don't 
 escape the scope. Would there be a good idea to and a new 
 function, besides the destructor, that will be called for 
 variables declared as "scope" when they go out of scope.

 The problem with destructors are that they can be called both 
 when an object is deleted by the GC and when an object goes of 
 out scope.


I'd rather introduce a special method that is called only by the 
GC. Cleaning up after an object that goes out of scope has always 
been the task of the regular destructor, it's undeterministic 
destruction that needs special treatment.
Aug 28 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-08-28 11:16, "Marc Schütz" <schuetzm gmx.net>" wrote:


 I'd rather introduce a special method that is called only by the GC.
 Cleaning up after an object that goes out of scope has always been the
 task of the regular destructor, it's undeterministic destruction that
 needs special treatment.


I was think about not breaking code. But introducing a new function that 
is called by the GC which also calls the regular destructor might work.

-- 
/Jacob Carlborg
Aug 28 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 28 August 2014 at 18:53:25 UTC, Jacob Carlborg wrote:

 On 2014-08-28 11:16, "Marc Schütz" <schuetzm gmx.net>" wrote:


 I'd rather introduce a special method that is called only by 
 the GC.
 Cleaning up after an object that goes out of scope has always 
 been the
 task of the regular destructor, it's undeterministic 
 destruction that
 needs special treatment.


 I was think about not breaking code. But introducing a new 
 function that is called by the GC which also calls the regular 
 destructor might work.


The other way round would be safer: A destructor automatically 
calls as its first step a finalizer (let's use that term for a 
destructor called by the GC) if present, but a finalizer doesn't 
call the destructor. Remember that the things that are forbidden 
in a finalizer are usually fine in normal destructors. By calling 
the destructor from inside the finalizer, you bring all the 
problems back that you wanted to get rid of by introducing a 
special finalizer, right?

And this would be backwards-compatible: There is already today no 
guarantee that a destructor gets called by the GC, so never 
calling it doesn't break any valid code, strictly speaking. Then 
you could place "safe" cleanup actions (like closing a file) into 
the finalizer, and "unsafe" ones (like removing yourself from a 
linked list) into the destructor, and you don't need to duplicate 
the actions from the finalizer in the destructor. The compiler 
might then even detect unsafe operations in the finalizer and 
refuse to compile them.
Aug 28 2014
parent Jacob Carlborg <doob me.com> writes:
On 28/08/14 21:27, "Marc Schütz" <schuetzm gmx.net>" wrote:


 The other way round would be safer: A destructor automatically calls as
 its first step a finalizer (let's use that term for a destructor called
 by the GC) if present, but a finalizer doesn't call the destructor.
 Remember that the things that are forbidden in a finalizer are usually
 fine in normal destructors. By calling the destructor from inside the
 finalizer, you bring all the problems back that you wanted to get rid of
 by introducing a special finalizer, right?

 And this would be backwards-compatible: There is already today no
 guarantee that a destructor gets called by the GC, so never calling it
 doesn't break any valid code, strictly speaking. Then you could place
 "safe" cleanup actions (like closing a file) into the finalizer, and
 "unsafe" ones (like removing yourself from a linked list) into the
 destructor, and you don't need to duplicate the actions from the
 finalizer in the destructor. The compiler might then even detect unsafe
 operations in the finalizer and refuse to compile them.


Yeah, you're probably right. I got it all backwards.

-- 
/Jacob Carlborg
Aug 28 2014
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
PING

Now that there are again several GC related topics being 
discussed, I thought I'd bump this thread.

Would be nice if Walter and/or Andrei could have a look and share 
there opinions. Is this something worth pursuing further? Are 
there fundamental objections against it?

On Sunday, 24 August 2014 at 13:14:45 UTC, Marc Schütz wrote:

 In the "Opportunities for D" thread, Walter again mentioned the 
 topics ref counting, GC, uniqueness, and borrowing, from which 
 a lively discussion developed [1]. I took this thread as an 
 opportunity to write down some ideas about these topics. The 
 result is a rather extensive proposal for the implementation of 
 borrowing, and its implementations:

 http://wiki.dlang.org/User:Schuetzm/scope

 This is not a real DIP, but before I put more work into 
 formalizing it, I'd like to hear some thoughts from the 
 languages gurus here:

 * Is this the general direction we want to go? Is it acceptable 
 in general?
 * Is the proposal internally consistent?
 * How big would the effort to implement it be? (I suspect it's 
 a large amount of work, but relatively straightforward.)

 [1] http://forum.dlang.org/thread/lphnen$1ml7$1 digitalmars.com
Sep 11 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Marc Schütz:


 Now that there are again several GC related topics being 
 discussed, I thought I'd bump this thread.

 Would be nice if Walter and/or Andrei could have a look and 
 share there opinions. Is this something worth pursuing further? 
 Are there fundamental objections against it?


At the moment the focus seems to be:
1) C++ interoperability
2) GC (in theory).

Bye,
bearophile
Sep 11 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/11/14, 7:06 AM, bearophile wrote:

 Marc Schütz:


 Now that there are again several GC related topics being discussed, I
 thought I'd bump this thread.

 Would be nice if Walter and/or Andrei could have a look and share
 there opinions. Is this something worth pursuing further? Are there
 fundamental objections against it?


 At the moment the focus seems to be:
 1) C++ interoperability
 2) GC (in theory).


scope is GC-related so looking at it is appropriate. -- Andrei
Sep 11 2014
prev sibling next sibling parent reply Ivan Timokhin <timokhin.iv gmail.com> writes:
I am in no way a language guru, but here are a few things that bother me 
in your proposal. Thought I'd share.

1. AFAIK, all current D type modifiers can be safely removed from the 
topmost level (i.e. it is OK to assign immutable(int[]) to 
immutable(int)[]), because they currently apply to particular variable, 
so there's no good reason to impose same restrictions on its copy. 
Situation seems different with scope: it is absolutely not safe to cast 
away and it applies to a *value*, not a variable holding it.

This is not only inconsistent, but may also cause trouble with 
interaction with existing features. For example, what should be 
std.traits.Unqual!(scope(int*)) ?

2. Consider findSubstring from your examples. What should be 
typeof(findSubstring("", ""))? Is the following code legal?

	scope(string) a = ..., b = ...;
	...
	typeof(findSubstring("", "")) c = findSubstring(a, b);

This is a bit troublesome, because this is how things like 
std.range.ElementType work currently, so they may break. For example,
what would be ElementType!ByLineImpl (from the "scope(const...)" section)?

This troubles me the most, because currently return type of a function 
may depend only on types of its arguments, and there is a lot of 
templated code written in that assumption. With the current proposal it 
ALL could break. Maybe there's no way around it if we want a solid 
lifetime management system, but I think this is definitely a problem to 
be aware of.

3. I believe it was mentioned before, but shouldn't scope propagate 
*outwards*? This would not only make perfect sense, since the aggregate 
obviously "holds the reference" just as well as its member does, it 
would also make various range-wrappers and alike automatically 
scope-aware, in that the wrapper would automatically become scoped if 
the wrapped range is scoped.

11.09.2014 15:58, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" пишет:

 PING

 Now that there are again several GC related topics being discussed, I
 thought I'd bump this thread.

 Would be nice if Walter and/or Andrei could have a look and share there
 opinions. Is this something worth pursuing further? Are there
 fundamental objections against it?

 On Sunday, 24 August 2014 at 13:14:45 UTC, Marc Schütz wrote:

 In the "Opportunities for D" thread, Walter again mentioned the topics
 ref counting, GC, uniqueness, and borrowing, from which a lively
 discussion developed [1]. I took this thread as an opportunity to
 write down some ideas about these topics. The result is a rather
 extensive proposal for the implementation of borrowing, and its
 implementations:

 http://wiki.dlang.org/User:Schuetzm/scope

 This is not a real DIP, but before I put more work into formalizing
 it, I'd like to hear some thoughts from the languages gurus here:

 * Is this the general direction we want to go? Is it acceptable in
 general?
 * Is the proposal internally consistent?
 * How big would the effort to implement it be? (I suspect it's a large
 amount of work, but relatively straightforward.)

 [1] http://forum.dlang.org/thread/lphnen$1ml7$1 digitalmars.com
Sep 11 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 11 September 2014 at 16:32:54 UTC, Ivan Timokhin 
wrote:

 I am in no way a language guru, but here are a few things that 
 bother me in your proposal. Thought I'd share.


Neither am I :-)


 1. AFAIK, all current D type modifiers can be safely removed 
 from the topmost level (i.e. it is OK to assign 
 immutable(int[]) to immutable(int)[]), because they currently 
 apply to particular variable, so there's no good reason to 
 impose same restrictions on its copy. Situation seems different 
 with scope: it is absolutely not safe to cast away and it 
 applies to a *value*, not a variable holding it.


The types in your example are implicitly convertable, indeed no 
explicit cast is necessary. This is because when you copy a const 
value, the result doesn't need to be const. But with scope, it 
makes sense (and is of course necessary) to keep the ownership. I 
don't see that as an inconsistency, but as a consequence of the 
different things const and scope imply: mutability vs. ownership.


 This is not only inconsistent, but may also cause trouble with 
 interaction with existing features. For example, what should be 
 std.traits.Unqual!(scope(int*)) ?


Good question. I would say it needs to keep scope, as it was 
clearly designed with mutability in mind (although it also 
removes shared, which is however related to mutability in a way). 
Ownership is an orthogonal concept to mutability.


 2. Consider findSubstring from your examples. What should be 
 typeof(findSubstring("", ""))? Is the following code legal?

 	scope(string) a = ..., b = ...;
 	...
 	typeof(findSubstring("", "")) c = findSubstring(a, b);


It's not legal. String literals live forever, so `c` has an owner 
that lives longer than `a` and `b`.

An alternative interpretation would be that the literals are 
temporary expressions; then it would have a very short lifetime, 
thus the assignment would be accepted. But I guess there needs to 
be a rule that says that the specified owners must not live 
shorter than the variable itself.


 This is a bit troublesome, because this is how things like 
 std.range.ElementType work currently, so they may break. For 
 example,
 what would be ElementType!ByLineImpl (from the 
 "scope(const...)" section)?


I see... it can _not_ be:

     scope!(const ByLineImpl!(char, "\n").init)(ByLineImpl!(char, 
"\n"))

because the init value is copied and thus becomes a temporary. 
This is ugly. It would however work if ElementType would take an 
instance instead of a type.


 This troubles me the most, because currently return type of a 
 function may depend only on types of its arguments, and there 
 is a lot of templated code written in that assumption.


I'm sorry, I don't understand what you mean here. This is clearly 
not true, neither for normal functions, nor for templates.


 With the current proposal it ALL could break. Maybe there's no 
 way around it if we want a solid lifetime management system, 
 but I think this is definitely a problem to be aware of.


The answer may be that scope needs to be something independent 
from the type, indeed more like a storage class, rather than what 
I suggested a type modifier. This would also solve the problem 
about `std.traits.Unqual`, no?

I'd have to think this through, but I believe this is indeed the 
way to go. It would make several other things cleaner. On the 
other hand, it would then be impossible to have scoped member 
fields, because storage classes aren't usable there, AFAIK. This 
would need to be supported first.


 3. I believe it was mentioned before, but shouldn't scope 
 propagate *outwards*? This would not only make perfect sense, 
 since the aggregate obviously "holds the reference" just as 
 well as its member does, it would also make various 
 range-wrappers and alike automatically scope-aware, in that the 
 wrapper would automatically become scoped if the wrapped range 
 is scoped.


You mean that any aggregate that contains a member with owner X 
automatically gets X as its owner itself?

I don't think so, because assigning a struct is semantically 
equivalent to assigning its members individually one after the 
others (by default) or whatever opAssign() is implemented to do. 
This means that an assignment that violates the rules would fail 
anyway, because the ownership is codified as part of the member's 
type. Instances of wrapper types would also need to be declared 
as scope with the appropriate owner, because otherwise they could 
not contain the scoped variables.

But I'm not sure how this is supposed to work with the storage 
class version of scope.
Sep 11 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 11 September 2014 at 20:45:09 UTC, Marc Schütz wrote:

 On Thursday, 11 September 2014 at 16:32:54 UTC, Ivan Timokhin 
 wrote:

 1. AFAIK, all current D type modifiers can be safely removed 
 from the topmost level (i.e. it is OK to assign 
 immutable(int[]) to immutable(int)[]), because they currently 
 apply to particular variable, so there's no good reason to 
 impose same restrictions on its copy. Situation seems 
 different with scope: it is absolutely not safe to cast away 
 and it applies to a *value*, not a variable holding it.


 The types in your example are implicitly convertable, indeed no 
 explicit cast is necessary. This is because when you copy a 
 const value, the result doesn't need to be const. But with 
 scope, it makes sense (and is of course necessary) to keep the 
 ownership. I don't see that as an inconsistency, but as a 
 consequence of the different things const and scope imply: 
 mutability vs. ownership.


I've addressed this in the Wiki now. There were only a few 
changes to be made to move away from type modifiers. I had even 
suggested it as an implementation detail, but didn't think of 
making it part of the specification. Thank you for that insight, 
it makes the proposal more consistent and avoids the troubles 
with the types.


 This is not only inconsistent, but may also cause trouble with 
 interaction with existing features. For example, what should 
 be std.traits.Unqual!(scope(int*)) ?


 Good question. I would say it needs to keep scope, as it was 
 clearly designed with mutability in mind (although it also 
 removes shared, which is however related to mutability in a 
 way). Ownership is an orthogonal concept to mutability.


After the changes, this is now the case.


 This is a bit troublesome, because this is how things like 
 std.range.ElementType work currently, so they may break. For 
 example,
 what would be ElementType!ByLineImpl (from the 
 "scope(const...)" section)?


 I see... it can _not_ be:

     scope!(const ByLineImpl!(char, 
 "\n").init)(ByLineImpl!(char, "\n"))

 because the init value is copied and thus becomes a temporary. 
 This is ugly. It would however work if ElementType would take 
 an instance instead of a type.


Ditto, this works now. The type is now simply `char[]` (or 
whatever), the owner is tracked separately.
Sep 12 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 September 2014 18:06, via Digitalmars-d <digitalmars-d puremagic.com>
wrote:


 On Thursday, 11 September 2014 at 20:45:09 UTC, Marc Sch=C3=BCtz wrote:


 On Thursday, 11 September 2014 at 16:32:54 UTC, Ivan Timokhin wrote:


 1. AFAIK, all current D type modifiers can be safely removed from the
 topmost level (i.e. it is OK to assign immutable(int[]) to
 immutable(int)[]), because they currently apply to particular variable,=






 so

 there's no good reason to impose same restrictions on its copy. Situati=






on

 seems different with scope: it is absolutely not safe to cast away and =






it

 applies to a *value*, not a variable holding it.


 The types in your example are implicitly convertable, indeed no explicit
 cast is necessary. This is because when you copy a const value, the resu=




lt

 doesn't need to be const. But with scope, it makes sense (and is of cour=




se

 necessary) to keep the ownership. I don't see that as an inconsistency, =




but

 as a consequence of the different things const and scope imply: mutabili=




ty

 vs. ownership.


 I've addressed this in the Wiki now. There were only a few changes to be
 made to move away from type modifiers. I had even suggested it as an
 implementation detail, but didn't think of making it part of the
 specification. Thank you for that insight, it makes the proposal more
 consistent and avoids the troubles with the types.



I'm not convinced this is a good change.
It sounds like you're just trading one problem with another more sinister
problem...

What happens when a scope() thing finds it's way into generic code? If the
type doesn't carry that information, then you end up in a situation like
ref. Have you ever had to wrestle with ref in generic code?
ref is the biggest disaster zone in D, and I think all it's problems will
translate straight to scope if you do this.


This is not only inconsistent, but may also cause trouble with interaction

 with existing features. For example, what should be
 std.traits.Unqual!(scope(int*)) ?


 Good question. I would say it needs to keep scope, as it was clearly
 designed with mutability in mind (although it also removes shared, which=




 is

 however related to mutability in a way). Ownership is an orthogonal conc=




ept

 to mutability.


 After the changes, this is now the case.

  This is a bit troublesome, because this is how things like

 std.range.ElementType work currently, so they may break. For example,
 what would be ElementType!ByLineImpl (from the "scope(const...)"
 section)?


 I see... it can _not_ be:

     scope!(const ByLineImpl!(char, "\n").init)(ByLineImpl!(char, "\n"))

 because the init value is copied and thus becomes a temporary. This is
 ugly. It would however work if ElementType would take an instance instea=




d

 of a type.


 Ditto, this works now. The type is now simply `char[]` (or whatever), the
 owner is tracked separately.
Sep 12 2014
next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 13 September 2014 at 01:49:05 UTC, Manu via 
Digitalmars-d wrote:

 On 12 September 2014 18:06, via Digitalmars-d 
 <digitalmars-d puremagic.com>
 wrote:


 On Thursday, 11 September 2014 at 20:45:09 UTC, Marc Schütz 
 wrote:
 I've addressed this in the Wiki now. There were only a few 
 changes to be
 made to move away from type modifiers. I had even suggested it 
 as an
 implementation detail, but didn't think of making it part of 
 the
 specification. Thank you for that insight, it makes the 
 proposal more
 consistent and avoids the troubles with the types.



 I'm not convinced this is a good change.
 It sounds like you're just trading one problem with another 
 more sinister
 problem...

 What happens when a scope() thing finds it's way into generic 
 code? If the
 type doesn't carry that information, then you end up in a 
 situation like
 ref. Have you ever had to wrestle with ref in generic code?
 ref is the biggest disaster zone in D, and I think all it's 
 problems will
 translate straight to scope if you do this.


Could you give an example?
Sep 13 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/12/2014 6:48 PM, Manu via Digitalmars-d wrote:

 What happens when a scope() thing finds it's way into generic code? If the type
 doesn't carry that information, then you end up in a situation like ref. Have
 you ever had to wrestle with ref in generic code?
 ref is the biggest disaster zone in D, and I think all it's problems will
 translate straight to scope if you do this.


I'm unaware of this disaster zone.
Sep 20 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 21 September 2014 at 03:48:36 UTC, Walter Bright wrote:

 On 9/12/2014 6:48 PM, Manu via Digitalmars-d wrote:

 What happens when a scope() thing finds it's way into generic 
 code? If the type
 doesn't carry that information, then you end up in a situation 
 like ref. Have
 you ever had to wrestle with ref in generic code?
 ref is the biggest disaster zone in D, and I think all it's 
 problems will
 translate straight to scope if you do this.


 I'm unaware of this disaster zone.


Well it is very real. I had to duplicate bunch of code in my 
visitor generator recently because of it. Getting generic code 
ref correct is very tedious, error prone, and guarantees code 
duplication and/or various static ifs all over the place.
Sep 20 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 21 September 2014 16:02, deadalnix via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Sunday, 21 September 2014 at 03:48:36 UTC, Walter Bright wrote:


 On 9/12/2014 6:48 PM, Manu via Digitalmars-d wrote:


 What happens when a scope() thing finds it's way into generic code? If
 the type
 doesn't carry that information, then you end up in a situation like ref.
 Have
 you ever had to wrestle with ref in generic code?
 ref is the biggest disaster zone in D, and I think all it's problems will
 translate straight to scope if you do this.


 I'm unaware of this disaster zone.


 Well it is very real. I had to duplicate bunch of code in my visitor
 generator recently because of it. Getting generic code ref correct is very
 tedious, error prone, and guarantees code duplication and/or various static
 ifs all over the place.


It's also extremely hard to unittest; explodes the number of static if
paths exponentially. I'm constantly finding bugs appear a year after
writing some code because I missed some static branch paths when originally
authoring.
Sep 21 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/21/14, 4:27 AM, Manu via Digitalmars-d wrote:

 On 21 September 2014 16:02, deadalnix via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:

     On Sunday, 21 September 2014 at 03:48:36 UTC, Walter Bright wrote:

         On 9/12/2014 6:48 PM, Manu via Digitalmars-d wrote:

             What happens when a scope() thing finds it's way into
             generic code? If the type
             doesn't carry that information, then you end up in a
             situation like ref.. Have
             you ever had to wrestle with ref in generic code?
             ref is the biggest disaster zone in D, and I think all it's
             problems will
             translate straight to scope if you do this.


         I'm unaware of this disaster zone.


     Well it is very real. I had to duplicate bunch of code in my visitor
     generator recently because of it. Getting generic code ref correct
     is very tedious, error prone, and guarantees code duplication and/or
     various static ifs all over the place.


 It's also extremely hard to unittest; explodes the number of static if
 paths exponentially.. I'm constantly finding bugs appear a year after
 writing some code because I missed some static branch paths when
 originally authoring.


Is this because of problems with ref's definition, or a natural 
consequence of supporting ref parameters? -- Andrei
Sep 21 2014
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 21 September 2014 at 15:10:23 UTC, Andrei Alexandrescu 
wrote:

 Is this because of problems with ref's definition, or a natural 
 consequence of supporting ref parameters? -- Andrei


There is various reason why this is complex:
  - classes must often be handled specifically.
  - auto ref is not very controllable and tend to end up with an 
combinatorial explosion of special cases.
  - it is complex to find out if something is ref or not in 
generic code.
  - it is not possible to conditionally declare something as ref. 
It is true of other qualifiers, but ref change the semantic in a 
more deeper way than, say, dropping pure or  safe.
  - you may want to use ref for very different semantics, which 
lead to different policies about what should be ref or not. There 
is no other way to implement these policies than spaghetti static 
if with meatball code duplication.
Sep 21 2014
prev sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 22 September 2014 01:10, Andrei Alexandrescu via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On 9/21/14, 4:27 AM, Manu via Digitalmars-d wrote:


 On 21 September 2014 16:02, deadalnix via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:

     On Sunday, 21 September 2014 at 03:48:36 UTC, Walter Bright wrote:

         On 9/12/2014 6:48 PM, Manu via Digitalmars-d wrote:

             What happens when a scope() thing finds it's way into
             generic code? If the type
             doesn't carry that information, then you end up in a
             situation like ref.. Have
             you ever had to wrestle with ref in generic code?
             ref is the biggest disaster zone in D, and I think all it's
             problems will
             translate straight to scope if you do this.


         I'm unaware of this disaster zone.


     Well it is very real. I had to duplicate bunch of code in my visitor
     generator recently because of it. Getting generic code ref correct
     is very tedious, error prone, and guarantees code duplication and/or
     various static ifs all over the place.


 It's also extremely hard to unittest; explodes the number of static if
 paths exponentially.. I'm constantly finding bugs appear a year after
 writing some code because I missed some static branch paths when
 originally authoring.


 Is this because of problems with ref's definition, or a natural
 consequence of supporting ref parameters? -- Andrei


It's all because ref is not part of the type.
You can't capture ref with typeof() or templates, you can't make ref
locals, it's hard to find if something is ref or not (detection is
different than everything else), etc.
The nature of it not being a type leads to static if's in every template
that ref appears, which must detect if things are ref (which is awkward),
and produce multiple paths for a ref and not-ref version. If we're dealing
with arguments, this might lead to num-arguments^^2 paths.

The only practical conclusion I (and others too) have reached, is to
eventually give up and invent Ref!T, but then we arrive at a new world of
problems. It's surprisingly hard to write a transparent Ref template which
interacts effectively with other generic code, and no 3rd party library
will support it.
Sep 22 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/21/2014 4:27 AM, Manu via Digitalmars-d wrote:

 It's also extremely hard to unittest; explodes the number of static if paths
 exponentially. I'm constantly finding bugs appear a year after writing some
code
 because I missed some static branch paths when originally authoring.


If you throw -cov while running unittests, it'll give you a report on which
code 
was executed and which wasn't. Very simple and useful.
Sep 21 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 22 September 2014 13:19, Walter Bright via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On 9/21/2014 4:27 AM, Manu via Digitalmars-d wrote:


 It's also extremely hard to unittest; explodes the number of static if
 paths
 exponentially. I'm constantly finding bugs appear a year after writing
 some code
 because I missed some static branch paths when originally authoring.


 If you throw -cov while running unittests, it'll give you a report on
 which code was executed and which wasn't. Very simple and useful.


It is a useful tool, but you can see how going to great lengths to write
this explosion of paths is a massive pain in the first place, let alone
additional overhead to comprehensively test that it works... it should
never have been a problem to start with.

You may argue that I didn't test my code effectively. I argue that my code
should never have existed in the first place. It's wildly unsanitary, and
very difficult to maintain; I can rarely understand the complexity of ref
handling code looking back after some months.

This was my very first complaint about D, on day 1... 6 years later, I'm
still struggling with it on a daily basis.

Here's some code I've been struggling with lately, tell me you think this
is right:
https://github.com/FeedBackDevs/LuaD/blob/master/luad/conversions/functions.d#L286
<- this one only deals with the return value. is broken, need to
special-case properties that receive arguments by ref, since you can't pass
rvalue->ref
https://github.com/FeedBackDevs/LuaD/blob/master/luad/conversions/functions.d#L115
<- Ref!RT workaround, because I need a ref local
https://github.com/FeedBackDevs/LuaD/blob/master/luad/conversions/functions.d#L172
<- Ref!T again, another instance where I need a ref local
https://github.com/FeedBackDevs/LuaD/blob/master/luad/conversions/functions.d#L180
https://github.com/FeedBackDevs/LuaD/blob/master/luad/stack.d#L129 <-
special case handling for Ref!T that I could eliminate if ref was part of
the type. Note: 3rd party code never has this concession...
https://github.com/FeedBackDevs/LuaD/blob/master/luad/stack.d#L448 <- more
invasion of Ref!, because getValue may or may not return ref, which would
be lost beyond this point, and it's not practical to static-if duplicate
this entire function with another version that returns ref.
https://github.com/FeedBackDevs/LuaD/blob/master/luad/stack.d#L157 <-
special-case function, again because ref isn't part of the type
There are many more instances of these throughout this code.

This is just one example, and not even a particularly bad one (I've had
worse), because there are never multiple args involved. These sorts of
things come up on most projects I've worked on.
The trouble with these examples, is that you can't try and imagine a direct
substitution of the static branches and Ref!T with 'ref T' if ref were part
of the type. This problem has deeply interfered with the code, and API
concessions have been made throughout to handle it. If ref were part of the
type, this code would be significantly re-worked and simplified. There
would probably be one little part somewhere that did logic on T, alias with
or without 'ref' as part of the type, and the cascading noise would mostly
disappear.

Add to all of that that I still can't pass an rvalue to a ref function (5
years later!!) >_<

It's also worth noting that, with regard to 'ref', the above code only
_just_ suits my needs. It's far from bug-free; there are places in there
where I was dealing with ref, but it got so complicated, and I didn't
actually make front-end use of the case in my project, that I gave up and
ignored those cases... which is not really ideal considering this is a
fairly popular library.

I have spent days trying to get this right. If I were being paid hourly, I
think I would have burned something like $2000.
Sep 22 2014
next sibling parent reply "Kagamin" <spam here.lot> writes:
On Monday, 22 September 2014 at 11:20:57 UTC, Manu via 
Digitalmars-d wrote:

 It is a useful tool, but you can see how going to great lengths 
 to write
 this explosion of paths is a massive pain in the first place, 
 let alone
 additional overhead to comprehensively test that it works... it 
 should
 never have been a problem to start with.


Hmm... even if the code is syntactically succinct, it doesn't 
necessarily mean lower complexity or that it requires less 
testing. You provided an example yourself: you have generic code, 
which works for values, but not for references. You need a lot of 
testing not because the features have different syntax, but 
because they work differently, so code, which works for one 
thing, may not work for another.
Sep 22 2014
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 22 September 2014 23:38, Kagamin via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Monday, 22 September 2014 at 11:20:57 UTC, Manu via Digitalmars-d wrote:


 It is a useful tool, but you can see how going to great lengths to write
 this explosion of paths is a massive pain in the first place, let alone
 additional overhead to comprehensively test that it works... it should
 never have been a problem to start with.


 Hmm... even if the code is syntactically succinct, it doesn't necessarily
 mean lower complexity or that it requires less testing. You provided an
 example yourself: you have generic code, which works for values, but not
 for references. You need a lot of testing not because the features have
 different syntax, but because they work differently, so code, which works
 for one thing, may not work for another.


Eliminating static branches containing different code has a very
significant reduction in complexity. It's also DRY.
I don't think I provided that example... although it's certainly true that
there are semantic differences that may lead to distinct code paths, it is
my experience that in the majority of cases, if I just had the ref-ness as
part of the type, the rest would follow naturally. I have never encountered
a situation where I would feel hindered by ref as part of the type.
I think it's also easier to get from ref in the type to the raw type than
the reverse (which we must do now); We are perfectly happy with Unqual!T
and things like that.
Sep 22 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/22/2014 4:20 AM, Manu via Digitalmars-d wrote:

 On 22 September 2014 13:19, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:

     On 9/21/2014 4:27 AM, Manu via Digitalmars-d wrote:

         It's also extremely hard to unittest; explodes the number of static if
paths
         exponentially. I'm constantly finding bugs appear a year after writing
         some code
         because I missed some static branch paths when originally authoring.


     If you throw -cov while running unittests, it'll give you a report on which
     code was executed and which wasn't. Very simple and useful.


 It is a useful tool, but you can see how going to great lengths to write this
 explosion of paths is a massive pain in the first place, let alone additional
 overhead to comprehensively test that it works... it should never have been a
 problem to start with.


There are two separate issues here - the first is knowing whether or not the 
code is unittested. -cov solves that issue. The second is having the multiple 
code paths in the first place.

Have you tried auto ref?

I don't really know what the code you show is supposed to do from a high level. 
My first impression is you are trying to write it like you'd write C++ code. 
Perhaps there's a more D idiomatic way of doing it that doesn't lead to the 
tangle you've got.

BTW, ref (as you know) is part of the type in C++. However, I can vouch for it 
being a special case everywhere in C++, and is a horrifying quagmire of strange 
edge cases. That's why it's not part of the type in D.
Sep 23 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 23 September 2014 at 09:46:17 UTC, Walter Bright 
wrote:

 Have you tried auto ref?


For some purposes, auto ref does the wrong thing. Whether you get 
a reference depends on whether you pass an lvalue or an rvalue. 
But some templates need to take either a struct by reference, or 
a class/interface (already being a reference) by value.

This is the case deadalnix mentioned further up in this thread. I 
don't know whether it applies to Manu's code, too. AFAIUI he's 
more concerned about forwarding parameters in wrapper types.
Sep 23 2014
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 23 September 2014 20:23, via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Tuesday, 23 September 2014 at 09:46:17 UTC, Walter Bright wrote:

 Have you tried auto ref?



 For some purposes, auto ref does the wrong thing. Whether you get a
 reference depends on whether you pass an lvalue or an rvalue. But some
 templates need to take either a struct by reference, or a class/interface
 (already being a reference) by value.

 This is the case deadalnix mentioned further up in this thread. I don't know
 whether it applies to Manu's code, too. AFAIUI he's more concerned about
 forwarding parameters in wrapper types.


That's just in this case. I'm concerned with many different cases over
time. This is a half decade running agony for me ;)
Sep 23 2014
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 23 September 2014 19:45, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 9/22/2014 4:20 AM, Manu via Digitalmars-d wrote:

 On 22 September 2014 13:19, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:

     On 9/21/2014 4:27 AM, Manu via Digitalmars-d wrote:

         It's also extremely hard to unittest; explodes the number of static if
paths
         exponentially. I'm constantly finding bugs appear a year after writing
         some code
         because I missed some static branch paths when originally authoring.


     If you throw -cov while running unittests, it'll give you a report on which
     code was executed and which wasn't. Very simple and useful.


 It is a useful tool, but you can see how going to great lengths to write this
 explosion of paths is a massive pain in the first place, let alone additional
 overhead to comprehensively test that it works... it should never have been a
 problem to start with.



 There are two separate issues here - the first is knowing whether or not the
code is unittested. -cov solves that issue. The second is having the multiple
code paths in the first place.

 Have you tried auto ref?


auto ref has never been what I've wanted. Semantically, it makes ref
out of all value-type lvalues, including int/float.
I don't want the compiler attempting to presume what I want in my
generic code. If it's part of the type, then there are no surprises;
it is exactly what I give it, which I gave deliberately.



 I don't really know what the code you show is supposed to do from a high
level. My first impression is you are trying to write it like you'd write C++
code. Perhaps there's a more D idiomatic way of doing it that doesn't lead to
the tangle you've got.


The only way I can think to eliminate that is to write the lot in a
huge mixin which composes the text from bits. I avoid mixin.

D has extensive language to manipulate and compare types, it has very
little to deal with these few weird external concepts like 'storage
class'. D has the most powerful type system I'm aware of, I see no
reason to break from that.
I think 'storage class' is a source of extreme complexity in D, since
it breaks uniformity with the rest of the language.



 BTW, ref (as you know) is part of the type in C++. However, I can vouch for it
being a special case everywhere in C++, and is a horrifying quagmire of strange
edge cases. That's why it's not part of the type in D.


I've never had any problems with ref in C++. D presents the horrible
quagmire of edge cases in my experience, some of which I've presented,
but I've had many more issues in the past.
Can you give me some examples of the problems in C++ you set out to
avoid? I've been programming C++ for 20 years, and D for 5-6 years.
I've never had this problem in C++, I have it so often in D, it drives
me crazy.

I also believe that D wouldn't suffer the same problems as C++,
because we have much better type manipulation. Things like
PointerTarget!T, Unqual!T, and all the other complex type manipulation
templates are common and work well in D, but that sort of thing almost
never works well in C++. The type comparison logic in C++ is too
feeble to be comparably useful.
Sep 23 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/23/2014 4:19 AM, Manu via Digitalmars-d wrote:

 BTW, ref (as you know) is part of the type in C++. However, I can vouch for
 it being a special case everywhere in C++, and is a horrifying quagmire of
 strange edge cases. That's why it's not part of the type in D.


 I've never had any problems with ref in C++. D presents the horrible quagmire
 of edge cases in my experience, some of which I've presented, but I've had
 many more issues in the past. Can you give me some examples of the problems
 in C++ you set out to avoid?


Q: Given a T&, and type deduction, when do you get the T and when do you get
the 
T& ?

A: It's different for every situation. Nobody can remember or enumerate the 
cases. Maybe Scott Meyers.
Sep 24 2014
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 24 September 2014 17:50, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 9/23/2014 4:19 AM, Manu via Digitalmars-d wrote:

 BTW, ref (as you know) is part of the type in C++. However, I can vouch
 for
 it being a special case everywhere in C++, and is a horrifying quagmire
 of
 strange edge cases. That's why it's not part of the type in D.



 I've never had any problems with ref in C++. D presents the horrible
 quagmire
 of edge cases in my experience, some of which I've presented, but I've had
 many more issues in the past. Can you give me some examples of the
 problems
 in C++ you set out to avoid?



 Q: Given a T&, and type deduction, when do you get the T and when do you get
 the T& ?

 A: It's different for every situation. Nobody can remember or enumerate the
 cases. Maybe Scott Meyers.


Scott Myers presented on this in extraordinary detail. By my
recollection (without referring to his talk), it was different in
*one* case, not every case. It's a binary situation, so it can't
possibly be different in 'every case', there are only 2 cases, and one
of them is the common case.

I suggest if the idea were explored in D, we would see if it works
where type deduction always gives what you expect (ie, gives ref(T)
for ref(T)). Unlike C++, we have an extensive suite of tools like
PointerTarget!T, Unqual!T, isPointer!T and friends; we can use those
tools explicitly in the places where we want T from deduction yielding
ref(T). I suspect we can avoid the C++ deduction hack given the
presence of these tools.

And if it proves that the C++ edge case is necessary (I don't think it
will), then so be it. I've never had problems with it, and always
found it intuitive in 20 years, whereas D's current setup is
immeasurably more complex than C++'s edge case, and causes me the
greatest source of pain in the language.
Sep 24 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Manu, once again your posts have the message embedded twice in them, making for 
very large posts. What's happening is the posts have the text in plain text, 
then the text again in HTML.

Please configure your news editor to only produce the plain text messages. The 
HTML versions are ignored and uselessly consume bandwidth and disk space.
Sep 23 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 23 September 2014 19:48, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Manu, once again your posts have the message embedded twice in them, making
 for very large posts. What's happening is the posts have the text in plain
 text, then the text again in HTML.

 Please configure your news editor to only produce the plain text messages.
 The HTML versions are ignored and uselessly consume bandwidth and disk
 space.


I use Gmail. It was configured... but it seems that it reverts to html
mode whenever I attach an image in any email I write, and then it
remembers the setting :/
Sep 23 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/23/2014 4:21 AM, Manu via Digitalmars-d wrote:

 On 23 September 2014 19:48, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Manu, once again your posts have the message embedded twice in them, making
 for very large posts. What's happening is the posts have the text in plain
 text, then the text again in HTML.

 Please configure your news editor to only produce the plain text messages.
 The HTML versions are ignored and uselessly consume bandwidth and disk
 space.


 I use Gmail. It was configured... but it seems that it reverts to html
 mode whenever I attach an image in any email I write, and then it
 remembers the setting :/


Well, that message was right!
Sep 23 2014
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 24 September 2014 04:18, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 9/23/2014 4:21 AM, Manu via Digitalmars-d wrote:

 On 23 September 2014 19:48, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Manu, once again your posts have the message embedded twice in them,
 making
 for very large posts. What's happening is the posts have the text in
 plain
 text, then the text again in HTML.

 Please configure your news editor to only produce the plain text
 messages.
 The HTML versions are ignored and uselessly consume bandwidth and disk
 space.



 I use Gmail. It was configured... but it seems that it reverts to html
 mode whenever I attach an image in any email I write, and then it
 remembers the setting :/


 Well, that message was right!


I changed the setting back ;)
Sep 23 2014
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 21 September 2014 at 11:37:19 UTC, Manu via 
Digitalmars-d wrote:

 On 21 September 2014 16:02, deadalnix via Digitalmars-d <
 digitalmars-d puremagic.com> wrote:


 On Sunday, 21 September 2014 at 03:48:36 UTC, Walter Bright 
 wrote:


 On 9/12/2014 6:48 PM, Manu via Digitalmars-d wrote:


 What happens when a scope() thing finds it's way into 
 generic code? If
 the type
 doesn't carry that information, then you end up in a 
 situation like ref.
 Have
 you ever had to wrestle with ref in generic code?
 ref is the biggest disaster zone in D, and I think all it's 
 problems will
 translate straight to scope if you do this.


 I'm unaware of this disaster zone.


 Well it is very real. I had to duplicate bunch of code in my 
 visitor
 generator recently because of it. Getting generic code ref 
 correct is very
 tedious, error prone, and guarantees code duplication and/or 
 various static
 ifs all over the place.


 It's also extremely hard to unittest; explodes the number of 
 static if
 paths exponentially. I'm constantly finding bugs appear a year 
 after
 writing some code because I missed some static branch paths 
 when originally
 authoring.


If I understand you right, your problems come from the fact that 
sometimes in a template you want ref, and sometimes you don't.

But I think this mostly doesn't apply to scope: either you borrow 
things, or you don't. In particular, when you do borrow 
something, you're not interested in the owner your parameter has 
inside the caller, you just take it by scope (narrowing the 
lifetime). Thus there needs to be no information about it inside 
the callee, and you don't need different instantiations depending 
on it.

One special case where scope deduction might be desirable are 
template functions that apply predicates (delegates, lambdas) to 
passed-in parameters, like map and filter. For these, the 
scope-ness of the input range can depend on whether the 
predicates are able to take their parameters as scope.
Sep 22 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 22 September 2014 19:22, via Digitalmars-d <digitalmars-d puremagic.com>
wrote:


 On Sunday, 21 September 2014 at 11:37:19 UTC, Manu via Digitalmars-d wrote:


 On 21 September 2014 16:02, deadalnix via Digitalmars-d <
 digitalmars-d puremagic.com> wrote:

  On Sunday, 21 September 2014 at 03:48:36 UTC, Walter Bright wrote:

  On 9/12/2014 6:48 PM, Manu via Digitalmars-d wrote:

  What happens when a scope() thing finds it's way into generic code? If

 the type
 doesn't carry that information, then you end up in a situation like
 ref.
 Have
 you ever had to wrestle with ref in generic code?
 ref is the biggest disaster zone in D, and I think all it's problems
 will
 translate straight to scope if you do this.


 I'm unaware of this disaster zone.


 Well it is very real. I had to duplicate bunch of code in my visitor
 generator recently because of it. Getting generic code ref correct is
 very
 tedious, error prone, and guarantees code duplication and/or various
 static
 ifs all over the place.


 It's also extremely hard to unittest; explodes the number of static if
 paths exponentially. I'm constantly finding bugs appear a year after
 writing some code because I missed some static branch paths when
 originally
 authoring.


 If I understand you right, your problems come from the fact that sometimes
 in a template you want ref, and sometimes you don't.

 But I think this mostly doesn't apply to scope: either you borrow things,
 or you don't. In particular, when you do borrow something, you're not
 interested in the owner your parameter has inside the caller, you just take
 it by scope (narrowing the lifetime). Thus there needs to be no information
 about it inside the callee, and you don't need different instantiations
 depending on it.

 One special case where scope deduction might be desirable are template
 functions that apply predicates (delegates, lambdas) to passed-in
 parameters, like map and filter. For these, the scope-ness of the input
 range can depend on whether the predicates are able to take their
 parameters as scope.


Application to scope will be identical to ref. A function that returns or
receives scope that is inserted into generic code must have that property
cascaded outwards appropriately. If typeof() or alias loses 'scope', then
it will all go tits-up.
Sep 22 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 22 September 2014 at 11:45:39 UTC, Manu via 
Digitalmars-d wrote:

 Application to scope will be identical to ref. A function that 
 returns or
 receives scope that is inserted into generic code must have 
 that property
 cascaded outwards appropriately. If typeof() or alias loses 
 'scope', then
 it will all go tits-up.


For receiving it's not necessary, because whether or not the 
argument is scoped, the function can always borrow it. The 
lifetime of its parameter is narrower than what it gets passed.

For return values, the situation is a bit different: They can of 
course not be assigned to non-scoped variables. But the solution 
for this simple: the generic code needs to use scope, too. A 
function that returns scope does so for a reason after all. This 
will work even if the return value of the called function turns 
out not to be scoped for this particular instantiation. And all 
this is an implementation of the generic code, it won't bleed 
outside, unless the generic code wants to return the scoped 
value. In this case, simply apply the same technique, just one 
lever higher.

I don't see this as a problem for (new) code written with scope 
in mind. For existing code, of course some adjustments are 
necessary, but the same is true if you change existing code to be 
const correct, for example, or to be compatible with `shared`.
Sep 22 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 22 September 2014 22:14, via Digitalmars-d <digitalmars-d puremagic.com>
wrote:


 On Monday, 22 September 2014 at 11:45:39 UTC, Manu via Digitalmars-d wrote:


 Application to scope will be identical to ref. A function that returns or
 receives scope that is inserted into generic code must have that property
 cascaded outwards appropriately. If typeof() or alias loses 'scope', then
 it will all go tits-up.


 For receiving it's not necessary, because whether or not the argument is
 scoped, the function can always borrow it. The lifetime of its parameter is
 narrower than what it gets passed.


It's particularly common in D to produce templates that wrap functions.
If the wrapper doesn't propagate scope outwards, then it can no longer be
called by a caller who borrowed arguments which are to be forwarded to the
function being called. Likewise for return values.

For return values, the situation is a bit different: They can of course not

 be assigned to non-scoped variables. But the solution for this simple: the
 generic code needs to use scope, too.



This is precisely the problem with ref...
Are you saying that ALL generic code needs to be 'scope' always? That's not
semantically correct.

A function that returns scope does so for a reason after all.


And the generic code can't know what it is. That knowledge must be encoded
in the type system.

This will work even if the return value of the called function turns out

 not to be scoped for this particular instantiation. And all this is an
 implementation of the generic code, it won't bleed outside, unless the
 generic code wants to return the scoped value. In this case, simply apply
 the same technique, just one lever higher.


I can't see the solution you're trying to ilustrate, can you demonstrate?
Sep 22 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Monday, 22 September 2014 at 12:37:47 UTC, Manu via 
Digitalmars-d wrote:

 On 22 September 2014 22:14, via Digitalmars-d 
 <digitalmars-d puremagic.com>
 wrote:


 On Monday, 22 September 2014 at 11:45:39 UTC, Manu via 
 Digitalmars-d wrote:


 Application to scope will be identical to ref. A function 
 that returns or
 receives scope that is inserted into generic code must have 
 that property
 cascaded outwards appropriately. If typeof() or alias loses 
 'scope', then
 it will all go tits-up.


 For receiving it's not necessary, because whether or not the 
 argument is
 scoped, the function can always borrow it. The lifetime of its 
 parameter is
 narrower than what it gets passed.


 It's particularly common in D to produce templates that wrap 
 functions.
 If the wrapper doesn't propagate scope outwards, then it can no 
 longer be
 called by a caller who borrowed arguments which are to be 
 forwarded to the
 function being called. Likewise for return values.


You have a point there.


 For return values, the situation is a bit different: They can 
 of course not

 be assigned to non-scoped variables. But the solution for this 
 simple: the
 generic code needs to use scope, too.



 This is precisely the problem with ref...
 Are you saying that ALL generic code needs to be 'scope' 
 always? That's not
 semantically correct.


To be clear, I am referring to the implementation, the actual 
code of the generic functions, not to its signature. The 
signature of course needs to match the semantics of the generic 
function.

I also over-generalized when I said that the return value cannot 
be assigned to non-scope. It can theoretically depend on the 
input, though I'm not sure whether it's a good idea to allow this:

     scope!a T scopeFunc(scope T a, scope T b);

     T* genericFunc(T)(T* input1, T* input2) {
         ...
         // this is fine in theory: input1 points to GC or global 
data
         // (because it's not designated as scope)
         string temp = scopeFunc(input1, input2);
         ...
         return temp;
     }

Evidently, this generic function cannot accept scoped pointers, 
thus it can't take advantage of the fact that scopeFunc() does. 
It's therefore a good idea, to make any generic (and non-generic, 
too) function take its parameters by scope if at all possible:

     scope!input1 T* genericFunc(T)(scope T* input1, scope T* 
input2) {
         ...
         scope temp = scopeFunc(input1, input2);
         ...
         return temp;
     }

This second version of the function will work with scope and 
non-scope inputs alike. More importantly, it doesn't depend on 
whether it's allowed to assign a scope return value to non-scope 
if its owners aren't scoped (which I'd like to avoid).

Now, `genericFunc()` in turn returns a scoped reference, so any 
other generic code that calls it must again be treated in the 
same way. Everything else would be unsafe, after all. But note 
that this only goes as far as an actual scoped value is returned 
up the call-chain. Once you stop doing so (because you only need 
to call the scope-returning functions internally for intermediate 
results, for example), returning scope would no longer be 
necessary. It still makes sense for these higher-up functions to 
_accept_ scope, of course, if it's possible.

Of course, this is only true as long as the generic function 
knows about the semantics of `scopeFunc()`. Once you're trying to 
wrap functions (as alias predicates, opDispatch), there needs to 
be another solution. I'm not sure what this could be though. I 
see now why you mentioned ref. But the problem is not restricted 
to ref and scope, it would also apply to UDAs. Maybe, because it 
is a more general problem independent of scope, the solution 
needs to be a more general one, too.

As far as I can see, there's always a variadic template parameter 
involved (which is actually a list of aliases in most cases, 
right?). Would it work if aliases would forward their storage 
classes, too? Thinking about it, this seems natural, because 
aliases mean "pass by name".


 A function that returns scope does so for a reason after all.



 And the generic code can't know what it is. That knowledge must 
 be encoded
 in the type system.

 This will work even if the return value of the called function 
 turns out

 not to be scoped for this particular instantiation. And all 
 this is an
 implementation of the generic code, it won't bleed outside, 
 unless the
 generic code wants to return the scoped value. In this case, 
 simply apply
 the same technique, just one lever higher.


 I can't see the solution you're trying to ilustrate, can you 
 demonstrate?


I hope that the examples above illustrate what I mean. Of course, 
this doesn't solve the "perfect forwarding" problem, which should 
maybe be treated separately.

Maybe you can give counter examples too, if you think it doesn't 
work.
Sep 22 2014
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 23 September 2014 01:00, via Digitalmars-d <digitalmars-d puremagic.com>
wrote:


 On Monday, 22 September 2014 at 12:37:47 UTC, Manu via Digitalmars-d wrote:


 On 22 September 2014 22:14, via Digitalmars-d <
 digitalmars-d puremagic.com>
 wrote:

  On Monday, 22 September 2014 at 11:45:39 UTC, Manu via Digitalmars-d

 wrote:

  Application to scope will be identical to ref. A function that returns

 or
 receives scope that is inserted into generic code must have that
 property
 cascaded outwards appropriately. If typeof() or alias loses 'scope',
 then
 it will all go tits-up.


 For receiving it's not necessary, because whether or not the argument is
 scoped, the function can always borrow it. The lifetime of its parameter
 is
 narrower than what it gets passed.


 It's particularly common in D to produce templates that wrap functions.
 If the wrapper doesn't propagate scope outwards, then it can no longer be
 called by a caller who borrowed arguments which are to be forwarded to the
 function being called. Likewise for return values.


 You have a point there.


It's massive. Trust me, when you're fabricating functions from
introspecting other functions, you NEED all these details in the type. If
they're not part of the types, then you need to craft lots of junk code to
detect that information explicitly, and then you need to branch out n^^2
distinct paths (massive DRY violation) to handle all the combinations.
Imagine if 'const' was a storage class in the context of generic code...
the practicality of that would be identical to 'ref', except that const
appears everywhere, and ref appears rarely (probably because people tend to
avoid it, because it's broken).

For return values, the situation is a bit different: They can of course not

 be assigned to non-scoped variables. But the solution for this simple:
 the
 generic code needs to use scope, too.



 This is precisely the problem with ref...
 Are you saying that ALL generic code needs to be 'scope' always? That's
 not
 semantically correct.


 To be clear, I am referring to the implementation, the actual code of the
 generic functions, not to its signature. The signature of course needs to
 match the semantics of the generic function.


So...?


I also over-generalized when I said that the return value cannot be

 assigned to non-scope. It can theoretically depend on the input, though I'm
 not sure whether it's a good idea to allow this:

     scope!a T scopeFunc(scope T a, scope T b);

     T* genericFunc(T)(T* input1, T* input2) {
         ...
         // this is fine in theory: input1 points to GC or global data
         // (because it's not designated as scope)
         string temp = scopeFunc(input1, input2);
         ...
         return temp;
     }

 Evidently, this generic function cannot accept scoped pointers, thus it
 can't take advantage of the fact that scopeFunc() does. It's therefore a
 good idea, to make any generic (and non-generic, too) function take its
 parameters by scope if at all possible:

     scope!input1 T* genericFunc(T)(scope T* input1, scope T* input2) {
         ...
         scope temp = scopeFunc(input1, input2);
         ...
         return temp;
     }

 This second version of the function will work with scope and non-scope
 inputs alike. More importantly, it doesn't depend on whether it's allowed
 to assign a scope return value to non-scope if its owners aren't scoped
 (which I'd like to avoid).


We arrive at yet another case of "it should have been that way from the
start" wrt 'scope'.
The baggage of annotation, and the lack of annotation to existing code is a
pretty big pill to swallow.
If it were just part of the type, there would be no problem, T would
already be 'scope T' in the cases where you expect. I can't see any
disadvantages there.

Now, `genericFunc()` in turn returns a scoped reference, so any other

 generic code that calls it must again be treated in the same way.
 Everything else would be unsafe, after all. But note that this only goes as
 far as an actual scoped value is returned up the call-chain. Once you stop
 doing so (because you only need to call the scope-returning functions
 internally for intermediate results, for example), returning scope would no
 longer be necessary. It still makes sense for these higher-up functions to
 _accept_ scope, of course, if it's possible.

 Of course, this is only true as long as the generic function knows about
 the semantics of `scopeFunc()`. Once you're trying to wrap functions (as
 alias predicates, opDispatch), there needs to be another solution. I'm not
 sure what this could be though. I see now why you mentioned ref. But the
 problem is not restricted to ref and scope, it would also apply to UDAs.
 Maybe, because it is a more general problem independent of scope, the
 solution needs to be a more general one, too.


I think UDA's are clearly distinct from ref and scope. UDA's can attribute
types.
I strongly believe that the problem is the notion of a 'storage class',
it's a faulty concept. It has never yet proven itself be what I've ever
wanted in any case I'm aware of.
Your proposal even implies changes to the concept as it is; like being able
to create a local that is 'scope'. Is that a recognition of existing
problems? I've been asking for 'ref' local's for half a decade...


As far as I can see, there's always a variadic template parameter involved

 (which is actually a list of aliases in most cases, right?). Would it work
 if aliases would forward their storage classes, too?



I'll say, no. It's equally important that is(), typeof(), and type aliasing
all work too.
I also think this kinda undermines the notion of a storage class in
principle...




 Thinking about it, this seems natural, because aliases mean "pass by name".


Types can't be passed to alias parameters. Alias refers to a symbol, not a
type.

Even if this were to be jigged somehow, I think it really just kicks the
can forward, and ref/scope is lost somewhere else. I can't imagine ANY
situation where I would ever want that information to be lost (unless it
was deliberate, like Unqual!), so what's the end goal? We end up in a
situation where it's properly passed along everywhere that the information
is currently lost... and we have the same thing as if it were just part of
the type in the first place?


 A function that returns scope does so for a reason after all.

 And the generic code can't know what it is. That knowledge must be encoded
 in the type system.

 This will work even if the return value of the called function turns out


 not to be scoped for this particular instantiation. And all this is an
 implementation of the generic code, it won't bleed outside, unless the
 generic code wants to return the scoped value. In this case, simply apply
 the same technique, just one lever higher.


 I can't see the solution you're trying to ilustrate, can you demonstrate?


 I hope that the examples above illustrate what I mean. Of course, this
 doesn't solve the "perfect forwarding" problem, which should maybe be
 treated separately.


This would be another band-aid to a core problem. D already has plenty of
these.
I hear this "perfect forwarding" concept thrown around, and I think it's
another faulty concept. I rarely want 'perfect' forwarding... why would I
be forwarding in the first place if it's 'perfect' (there are cases, but
not so common)? I almost always want *imperfect* forwarding; that is, some
small detail(/s) about the forwarding are manipulated. I think this is the
primary case where storage class falls apart in concept.


Maybe you can give counter examples too, if you think it doesn't work.


It's complex and time consuming to do so. The situations where it all
breaks down are often fairly complex (probably why 'ref' as a storage class
seems like an okay idea at face value, although I still don't understand
the advantage conceptually), and they tend to appear when you don't expect
it. My examples with ref above are all practically applicable to scope too
though.

Let's turn this around... Why the complexity? Why would you make the change
to your proposal to make 'scope' something else outside of the type system?
What is the advantage to that complexity. D has no structured method for
dealing with that sort of meta, we only have types. Beyond that, it's just
spaghetti, as we learn from ref.



 Sep 22 2014



  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Monday, 22 September 2014 at 15:54:23 UTC, Manu via 
Digitalmars-d wrote:

 We arrive at yet another case of "it should have been that way 
 from the
 start" wrt 'scope'.
 The baggage of annotation, and the lack of annotation to 
 existing code is a
 pretty big pill to swallow.
 If it were just part of the type, there would be no problem, T 
 would
 already be 'scope T' in the cases where you expect. I can't see 
 any
 disadvantages there.


But it has very different semantics. You cannot expect the same 
code to work for scope and non-scope alike. And it is to be 
expected  that you have to adjust existing code if you want to 
take advantage of a new feature. If by "it should have been that 
way from the start" you mean that scope should be the default, 
well yes, but we're in the same situation with immutable by 
default, pure by default,  safe by default, nothrow by default, 
... That ship has sailed, unfortunately.


 I think UDA's are clearly distinct from ref and scope. UDA's 
 can attribute
 types.
 I strongly believe that the problem is the notion of a 'storage 
 class',
 it's a faulty concept. It has never yet proven itself be what 
 I've ever
 wanted in any case I'm aware of.
 Your proposal even implies changes to the concept as it is; 
 like being able
 to create a local that is 'scope'. Is that a recognition of 
 existing
 problems? I've been asking for 'ref' local's for half a 
 decade...


That you can't declare ref locals is not inherent in the concept 
of storage class. You can already today declare scope locals, it 
just doesn't have an effect. And static is a storage class, too.


 This would be another band-aid to a core problem. D already has 
 plenty of
 these.
 I hear this "perfect forwarding" concept thrown around, and I 
 think it's
 another faulty concept. I rarely want 'perfect' forwarding... 
 why would I
 be forwarding in the first place if it's 'perfect' (there are 
 cases, but
 not so common)? I almost always want *imperfect* forwarding; 
 that is, some
 small detail(/s) about the forwarding are manipulated. I think 
 this is the
 primary case where storage class falls apart in concept.


That is a straw man. Of course, perfect forwarding needs to allow 
for imperfect forwarding, too. It would indeed be not useful if 
you couldn't inspect and modify the types (and storage classes) 
of your parameters.


 Maybe you can give counter examples too, if you think it 
 doesn't work.


 It's complex and time consuming to do so. The situations where 
 it all
 breaks down are often fairly complex (probably why 'ref' as a 
 storage class
 seems like an okay idea at face value, although I still don't 
 understand
 the advantage conceptually), and they tend to appear when you 
 don't expect
 it. My examples with ref above are all practically applicable 
 to scope too
 though.


A concrete example would still be very helpful to understand your 
point of view. I.e., not only the concrete wrapper type/function, 
but also how you would want to use it, and why it wouldn't work 
without scope being part of the type.


 Let's turn this around... Why the complexity? Why would you 
 make the change
 to your proposal to make 'scope' something else outside of the 
 type system?


Well, I think Ivan gave an excellent example (ElementType) why it 
should be separate from the type. Type modifiers currently only 
deal with mutability (and the related shared). There can be some 
nasty surprises if we add another concept there.


 What is the advantage to that complexity. D has no structured 
 method for
 dealing with that sort of meta, we only have types. Beyond 
 that, it's just
 spaghetti, as we learn from ref.


Then it's better to introduce such a method, IMO.



 Sep 23 2014



  parent  Manu via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On 23 September 2014 21:02, via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 22 September 2014 at 15:54:23 UTC, Manu via Digitalmars-d wrote:

 We arrive at yet another case of "it should have been that way from the
 start" wrt 'scope'.
 The baggage of annotation, and the lack of annotation to existing code is
 a
 pretty big pill to swallow.
 If it were just part of the type, there would be no problem, T would
 already be 'scope T' in the cases where you expect. I can't see any
 disadvantages there.



 But it has very different semantics. You cannot expect the same code to work
 for scope and non-scope alike. And it is to be expected  that you have to
 adjust existing code if you want to take advantage of a new feature. If by
 "it should have been that way from the start" you mean that scope should be
 the default, well yes, but we're in the same situation with immutable by
 default, pure by default,  safe by default, nothrow by default, ... That
 ship has sailed, unfortunately.


You can't expect the same code to work for int and immutable(int)
alike either... or int and int[], they all have to be handled somewhat
explicitly.
Generic code shouldn't try to work with is(T == scope(U), U) if the
code doesn't support scope. If the generic code is to support scope,
then the generic code either specifies it's argument to be scope(T),
which works the same as const(T) wrt mutability, or it detects and
handles is(T == scope(U), U) internally.

And yes, that's what I meant by 'should have been that way from the start' :)



 This would be another band-aid to a core problem. D already has plenty of
 these.
 I hear this "perfect forwarding" concept thrown around, and I think it's
 another faulty concept. I rarely want 'perfect' forwarding... why would I
 be forwarding in the first place if it's 'perfect' (there are cases, but
 not so common)? I almost always want *imperfect* forwarding; that is, some
 small detail(/s) about the forwarding are manipulated. I think this is the
 primary case where storage class falls apart in concept.



 That is a straw man. Of course, perfect forwarding needs to allow for
 imperfect forwarding, too. It would indeed be not useful if you couldn't
 inspect and modify the types (and storage classes) of your parameters.


Well you suggested like perfect forwarding was a discrete problem to solve.
Imperfect forwarding implies composing a signature from a complex set
of typeof(), alias, template arguments, traits, etc. We can't be
losing ref-ness in any of these cases, or the forwarding is broken.
scope likewise.
(im)perfect forwarding would work just fine right now if there wasn't
random bits of information that were separated from the type system.



 Maybe you can give counter examples too, if you think it doesn't work.


 It's complex and time consuming to do so. The situations where it all
 breaks down are often fairly complex (probably why 'ref' as a storage
 class
 seems like an okay idea at face value, although I still don't understand
 the advantage conceptually), and they tend to appear when you don't expect
 it. My examples with ref above are all practically applicable to scope too
 though.



 A concrete example would still be very helpful to understand your point of
 view. I.e., not only the concrete wrapper type/function, but also how you
 would want to use it, and why it wouldn't work without scope being part of
 the type.


Generating function signatures is certainly the simplest and also the
most common example of these problems.
The problem is simple; if it's not part of the type, then the only
solution is a static if with explicit detection for the thing, and
code duplication with/without the non-type attribute.
Templates are rarely flat either, they are usually a composition of
other templates. Everything in the chain needs to have special case
handling for ref (and scope) that lives outside the type system.


 Let's turn this around... Why the complexity? Why would you make the
 change
 to your proposal to make 'scope' something else outside of the type
 system?



 Well, I think Ivan gave an excellent example (ElementType) why it should be
 separate from the type. Type modifiers currently only deal with mutability
 (and the related shared). There can be some nasty surprises if we add
 another concept there.


I don't think shared is really related, also what about *, [], [n],
etc. these are all type modifiers that cause the same sort of
'surprises'.
We have tools for dealing with all of these: Unqual!T,
PointerTarget!T, isPointer!T, isArray!T, etc... they all work well, I
haven't had any complaints about them personally. scope detection
would have no significant impact on the mix.

We also have tools like is(T : U), which should be extended to work as
expected with scope.
In the same way as "is(X : const(T)) == true" works for T == mutable,
immutable, or const, we would have the same is(X : scope(T)) to detect
if the lifetimes are compatible.
We need scope in the type system so we can make use of all the
powerful type manipulation features that D has. As far as I'm
concerned, D's type system *IS* D. (this is also true for ref)



 What is the advantage to that complexity. D has no structured method for
 dealing with that sort of meta, we only have types. Beyond that, it's just
 spaghetti, as we learn from ref.



 Then it's better to introduce such a method, IMO.


Please no. D does NOT need a parallel suite of syntax to deal with
'storage class' meta.



 Sep 23 2014



prev sibling  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Saturday, 13 September 2014 at 01:49:05 UTC, Manu via 
Digitalmars-d wrote:

 I'm not convinced this is a good change.
 It sounds like you're just trading one problem with another 
 more sinister
 problem...


Ok, I thought about it some more. I'm still not convinced 
completely, but I'm warming up to the idea of making scope a type 
modifier. However, Ivan's objections need to be addressed.

Maybe let's start with a list of problems of the type modifier 
way. So far:

* What is ElementType!(ByLineImpl!(char, "\n")) in the example 
from the wiki page [1]?
* Should Unqual!T strip `scope`?

Anything else? How can we solve these problems?

Another argument against storage class is a syntactical ambiguity 
in conncection with methods:

     struct S {
         scope!this int* foo() scope;
     }

It's ambiguous whether any given scope keyword applies to the 
return value or `this`. One could argue though that this is a 
consequence of the general function attribute problem and should 
preferably be fixed there. (`ref` doesn't have this problem 
because it cannot apply to `this`.)

[1] 
http://wiki.dlang.org/User:Schuetzm/scope#scope.21.28const_....29



 Sep 29 2014



  parent reply Ivan Timokhin <timokhin.iv gmail.com>  writes:


29.09.2014 18:17, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" пишет:

 * What is ElementType!(ByLineImpl!(char, "\n")) in the example from the
 wiki page [1]?


OK, I think I have an idea, but it's not overly elegant.

First of all, I would like to note that the same issue exists with, for 
example, findSubstring function (from "scope with owners" section), 
which does not have a definite return type (so it's unclear what 
ReturnType!findSubstring should be).

Now, an idea that I have is that bare scope should be just a syntactic 
sugar for self-owned scope; i.e., `scope(int*) x;` would be just a short 
way of writing `scope!x(int*) x;`. This would imply that every scoped 
variable has its own unique type, which is probably not that terrible, 
considering that they can't be assigned freely to each other either way. 
This is also somewhat more natural, because it means that there is no 
such thing as just "scoped" variable, it is always connected to a 
particular lifetime.

Then, the following:
---
	string findSubstring(scope(string) haystack,
		             scope(string) needle)
---
would be equivalent to
---
	string findSubstring(scope!haystack(string) haystack,
		             scope!needle(string) needle)
---
so each argument is self-owned and all ownership information is lost 
(but function is still callable with scoped arguments, because scope 
narrowing is allowed).

To preserve ownership information, which is encoded in an arguments' 
types, template is needed:
---
	scope!haystackOwner(string)
	findSubstring(alias haystackOwner)
		            (scope!haystackOwner(string) haystack,
		             scope(string) needle)
---
So, findSubstring *still* doesn't have a definite return type, but now 
it's because it is a function template, not a function.

As for passing unscoped strings to findSubstring, I see two alternatives:
1) Declare that all unscoped references are implicitly convertible to 
scope!GC or something like that (and this conversion is used in such 
cases). This one is probably better.
2) Require a separate overload for an unscoped string.

Now, to address ElementType problem, I would like to reconsider `this` 
lifetime. Since
---
	struct S {
		void f() {}
	}
---
is more or less equivalent to
---
	// not a valid D code
	struct S {}
	void f(ref S this) {}
---
(not strictly equivalent, of course, but the general idea is like that),
`this` inside a method body would behave like a ref parameter and have 
approximately the same lifetime as normal parameters.

Then this code:
---
	 property scope!(const this)(Char[]) front() const
---
would become illegal, since the return value is declared as having 
function-local scope. However, this code:
---
	 property scope!(const this)(Char[])
		front(alias thisOwner)() const scope!thisOwner
---
would work just fine, because it explicitly propagates current object's 
scope (the return type seems the same, but `this` itself now has a 
different type). The downside is that `front` is now callable only for 
scoped variables (it seems inappropriate to convert structs to scope!GC).

Now, ByLineImpl!(char, "\n") wouldn't be an input range, because front 
would not be defined in an unscoped case, and `scope ByLineImpl!(char, 
"\n")` is not a type (because bare scope would be purely a syntactic 
sugar in declarations), so not a range.

However, with this declaration:
---
	scope(ByLineImpl!(char, "\n")) x = ...;
---
typeof(x) would be scope!x(ByLineImpl!(char, "\n")), and 
ElementType!(typeof(x)) would be scope!(const x)(char[]).

As I have said in the beginning, not too elegant, but I think it may 
work. As a bonus, this is a little bit more straightforward and requires 
less special-casing from the compiler side: has only one scope type 
instead of two and no magic tricks with scoped return values. The only 
problems that I can see right away are that the code now is a little 
verbose, and that 'front' is restricted to scoped variables in a new 
version.

Any thoughts?



 Oct 03 2014



  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Friday, 3 October 2014 at 19:08:10 UTC, Ivan Timokhin wrote:

 29.09.2014 18:17, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" пишет:

 * What is ElementType!(ByLineImpl!(char, "\n")) in the example 
 from the
 wiki page [1]?


 OK, I think I have an idea, but it's not overly elegant.

 First of all, I would like to note that the same issue exists 
 with, for example, findSubstring function (from "scope with 
 owners" section), which does not have a definite return type 
 (so it's unclear what ReturnType!findSubstring should be).


Indeed. It's everywhere that an owner depends on another function 
argument (including `this`), but AFAICS nowhere else.


 Now, an idea that I have is that bare scope should be just a 
 syntactic sugar for self-owned scope; i.e., `scope(int*) x;` 
 would be just a short way of writing `scope!x(int*) x;`. This 
 would imply that every scoped variable has its own unique type, 
 which is probably not that terrible, considering that they 
 can't be assigned freely to each other either way. This is also 
 somewhat more natural, because it means that there is no such 
 thing as just "scoped" variable, it is always connected to a 
 particular lifetime.


I've already suggested this as an implementation detail.


 Then, the following:
 ---
 	string findSubstring(scope(string) haystack,
 		             scope(string) needle)
 ---
 would be equivalent to
 ---
 	string findSubstring(scope!haystack(string) haystack,
 		             scope!needle(string) needle)
 ---
 so each argument is self-owned and all ownership information is 
 lost (but function is still callable with scoped arguments, 
 because scope narrowing is allowed).

 To preserve ownership information, which is encoded in an 
 arguments' types, template is needed:
 ---
 	scope!haystackOwner(string)
 	findSubstring(alias haystackOwner)
 		            (scope!haystackOwner(string) haystack,
 		             scope(string) needle)
 ---
 So, findSubstring *still* doesn't have a definite return type, 
 but now it's because it is a function template, not a function.


This of course has the unfortunate side effect of incredible 
template bloat: For any distinct passed argument (which in 
practice means for almost every call), we'd get a new template 
instance. IMO this is not acceptable.


 As for passing unscoped strings to findSubstring, I see two 
 alternatives:
 1) Declare that all unscoped references are implicitly 
 convertible to scope!GC or something like that (and this 
 conversion is used in such cases). This one is probably better.


OTOH it would preclude automatic demoting of GC allocations to 
stack allocations. And it would marry us to the GC is "default" 
allocation strategy, which we might want to move away from 
(probably in favor of generic allocators changeable at any point 
in time).

We also need to allow borrowing of temporaries if we want to have 
rvalue references. And this runs into problems if we are dealing 
with scoped non-references, for which I'm seeing more use cases 
now than just file descriptors (in particular reference 
counting). In short, I'd like to avoid it.


 2) Require a separate overload for an unscoped string.


Ugly, of course, and I believe it's not necessary.


 Now, to address ElementType problem, I would like to reconsider 
 `this` lifetime. Since
 ---
 	struct S {
 		void f() {}
 	}
 ---
 is more or less equivalent to
 ---
 	// not a valid D code
 	struct S {}
 	void f(ref S this) {}
 ---
 (not strictly equivalent, of course, but the general idea is 
 like that),
 `this` inside a method body would behave like a ref parameter 
 and have approximately the same lifetime as normal parameters.

 Then this code:
 ---
 	 property scope!(const this)(Char[]) front() const
 ---
 would become illegal, since the return value is declared as 
 having function-local scope. However, this code:
 ---
 	 property scope!(const this)(Char[])
 		front(alias thisOwner)() const scope!thisOwner
 ---
 would work just fine, because it explicitly propagates current 
 object's scope (the return type seems the same, but `this` 
 itself now has a different type). The downside is that `front` 
 is now callable only for scoped variables (it seems 
 inappropriate to convert structs to scope!GC).


And of course, again, the template bloat :-( This time, a new 
instance of `front` for every instance of the range.


 Now, ByLineImpl!(char, "\n") wouldn't be an input range, 
 because front would not be defined in an unscoped case, and 
 `scope ByLineImpl!(char, "\n")` is not a type (because bare 
 scope would be purely a syntactic sugar in declarations), so 
 not a range.

 However, with this declaration:
 ---
 	scope(ByLineImpl!(char, "\n")) x = ...;
 ---
 typeof(x) would be scope!x(ByLineImpl!(char, "\n")), and 
 ElementType!(typeof(x)) would be scope!(const x)(char[]).

 As I have said in the beginning, not too elegant, but I think 
 it may work. As a bonus, this is a little bit more 
 straightforward and requires less special-casing from the 
 compiler side: has only one scope type instead of two and no 
 magic tricks with scoped return values. The only problems that 
 I can see right away are that the code now is a little verbose, 
 and that 'front' is restricted to scoped variables in a new 
 version.

 Any thoughts?


I think the key is in separating the scope attribute and the 
owner. The former needs to be part of the type, the latter 
doesn't. In this vein, it's probably a good idea to restrict the 
`scope!owner` syntax to function signatures, where it may only 
refer to other parameters and `this`. The use cases for it 
elsewhere are very marginal (if they exist at all).

This naturally makes the return type of `findSubstring()` just 
`scope(string)`; declaring a variable of it simply works:

     typeof(findSubstring("", "")) s = findSubstring("Hello, 
world", "world");

is equivalent to:

     scope(string) s = findSubstring("Hello, world", "world");

This is a valid assignment, and owner propagation would even take 
care of preserving the owners (though only on declaration, but 
that's natural because the owners are only known at the call 
site):

     string haystack, needle;
     scope(string) s = findSubstring(haystack, needle);
     // type of `s` is scope(string), owner is `haystack`

In other words, the type part of `scope!a(T)` is just `scope(T)`, 
the owner is not part of the type and tracked separately.

Let's look at isInputRange:

     template isInputRange(R)
     {
         enum bool isInputRange = is(typeof(
             (inout int = 0)
             {
                 R r = R.init;     // can define a range object
                 if (r.empty) {}   // can test for empty
                 r.popFront();     // can invoke popFront()
                 auto h = r.front; // can get the front of the 
range
             }));
     }

     auto byline = stdin.byLine();

`isInputRange!(typeof(byline))` expands to:

     scope(ByLineImpl...) r = scope(ByLineImpl...).init;
     if (r.empty) {}
     r.popFront();
     auto h = r.front;

The first line is valid, the last line too, because `scope` is 
now part of the type and will of course be deduced by `auto`.

So this solves the template bloat problem, along with making 
ElementType usable. That still leaves us with the problem of 
forwarding and wrappers.

     auto trace(alias func, Args...)(Args args) {
         writeln("Calling " ~ func.stringof ~ "(" ~ args.stringof 
~ ")");
         return func(args);
     }

     auto s = trace!findSubstring(haystack, needle);

Here, `Args` becomes `(scope(string), scope(string))`. This 
cannot work, of course, because it drops the owners. How can we 
a) preserve the owners on the parameters, and b) propagate the 
result's owner back outward?

 Manu: Under the condition that we'd find a solution for "perfect 
forwarding", or imperfect if you like ;-), preferrably one also 
applicable to `ref`, would you be okay with the above?



 Oct 04 2014



  parent reply Ivan Timokhin <timokhin.iv gmail.com>  writes:


04.10.2014 17:38, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" пишет:

 On Friday, 3 October 2014 at 19:08:10 UTC, Ivan Timokhin wrote:

 29.09.2014 18:17, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>"
 пишет:
 ...
 Now, an idea that I have is that bare scope should be just a syntactic
 sugar for self-owned scope; i.e., `scope(int*) x;` would be just a
 short way of writing `scope!x(int*) x;`. This would imply that every
 scoped variable has its own unique type, which is probably not that
 terrible, considering that they can't be assigned freely to each other
 either way. This is also somewhat more natural, because it means that
 there is no such thing as just "scoped" variable, it is always
 connected to a particular lifetime.


 I've already suggested this as an implementation detail.


Sorry then, must have missed it.



 ...


 This of course has the unfortunate side effect of incredible template
 bloat: For any distinct passed argument (which in practice means for
 almost every call), we'd get a new template instance. IMO this is not
 acceptable.


They could be merged in an executable, but a symbol table would be 
cluttered. That does sound like a major issue.


 As for passing unscoped strings to findSubstring, I see two alternatives:
 1) Declare that all unscoped references are implicitly convertible to
 scope!GC or something like that (and this conversion is used in such
 cases). This one is probably better.


 OTOH it would preclude automatic demoting of GC allocations to stack
 allocations. And it would marry us to the GC is "default" allocation
 strategy, which we might want to move away from (probably in favor of
 generic allocators changeable at any point in time).


It doesn't necessarily have to be GC, just any object that we can safely 
treat as an owner.


 ...


 I think the key is in separating the scope attribute and the owner. The
 former needs to be part of the type, the latter doesn't. In this vein,
 it's probably a good idea to restrict the `scope!owner` syntax to
 function signatures, where it may only refer to other parameters and
 `this`. The use cases for it elsewhere are very marginal (if they exist
 at all).

 This naturally makes the return type of `findSubstring()` just
 `scope(string)`; declaring a variable of it simply works:

      typeof(findSubstring("", "")) s = findSubstring("Hello, world",
 "world");

 is equivalent to:

      scope(string) s = findSubstring("Hello, world", "world");

 This is a valid assignment, and owner propagation would even take care
 of preserving the owners (though only on declaration, but that's natural
 because the owners are only known at the call site):

      string haystack, needle;
      scope(string) s = findSubstring(haystack, needle);
      // type of `s` is scope(string), owner is `haystack`

 In other words, the type part of `scope!a(T)` is just `scope(T)`, the
 owner is not part of the type and tracked separately.


That's ok with me (in fact, it looks very nice), but I think Manu's 
point was that dealing with anything that isn't part of the type is 
troublesome.


 ...





 Oct 04 2014



  parent reply Ivan Timokhin <timokhin.iv gmail.com>  writes:


04.10.2014 21:01, Ivan Timokhin пишет:

 04.10.2014 17:38, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>"
пишет:

 I think the key is in separating the scope attribute and the owner. The
 former needs to be part of the type, the latter doesn't. In this vein,
 it's probably a good idea to restrict the `scope!owner` syntax to
 function signatures, where it may only refer to other parameters and
 `this`. The use cases for it elsewhere are very marginal (if they exist
 at all).

 This naturally makes the return type of `findSubstring()` just
 `scope(string)`; declaring a variable of it simply works:

      typeof(findSubstring("", "")) s = findSubstring("Hello, world",
 "world");

 is equivalent to:

      scope(string) s = findSubstring("Hello, world", "world");

 This is a valid assignment, and owner propagation would even take care
 of preserving the owners (though only on declaration, but that's natural
 because the owners are only known at the call site):

      string haystack, needle;
      scope(string) s = findSubstring(haystack, needle);
      // type of `s` is scope(string), owner is `haystack`

 In other words, the type part of `scope!a(T)` is just `scope(T)`, the
 owner is not part of the type and tracked separately.


 That's ok with me (in fact, it looks very nice), but I think Manu's
 point was that dealing with anything that isn't part of the type is
 troublesome.


On the second thought, doesn't syntax look a bit awkward now? I mean, 
`scope!a(T)` certainly looks like `scope!a` as a whole is a type 
modifier. Since this syntax is pretty much limited to function return 
types with this proposal, maybe a better solution would be to have an 
owner as a separate annotation on a function?

Also, would it really make much sense to track the owner further than 
the assignment of a function's return value? That seems to complicate 
things a lot by adding a hidden attribute to a variable that is not only 
invisible at the declaration (even though the full type is spelled out), 
but, in fact, cannot be specified explicitly (because there's no syntax 
for that, now that scope with owners is limited to function signatures).

How about this:
---
     scope(string) haystack, needle;
     // next assignment is okay, because `s` is guaranteed not to outlive
     // `haystack`.
     scope(string) s = findSubstring(haystack, needle);
     // type of `s` is now scope(string), no additional information
     // attached

     // so the next assignment is disallowed:
     //needle = s; // error!
---

This could be unnecessarily limiting, but would it really cause much 
trouble?



 Oct 04 2014



  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Saturday, 4 October 2014 at 18:13:03 UTC, Ivan Timokhin wrote:

 On the second thought, doesn't syntax look a bit awkward now? I 
 mean, `scope!a(T)` certainly looks like `scope!a` as a whole is 
 a type modifier. Since this syntax is pretty much limited to 
 function return types with this proposal, maybe a better 
 solution would be to have an owner as a separate annotation on 
 a function?


I agree, but it should still stay closely associated to the 
return type.


 Also, would it really make much sense to track the owner 
 further than the assignment of a function's return value? That 
 seems to complicate things a lot by adding a hidden attribute 
 to a variable that is not only invisible at the declaration 
 (even though the full type is spelled out), but, in fact, 
 cannot be specified explicitly (because there's no syntax for 
 that, now that scope with owners is limited to function 
 signatures).

 How about this:
 ---
     scope(string) haystack, needle;
     // next assignment is okay, because `s` is guaranteed not 
 to outlive
     // `haystack`.
     scope(string) s = findSubstring(haystack, needle);
     // type of `s` is now scope(string), no additional 
 information
     // attached

     // so the next assignment is disallowed:
     //needle = s; // error!
 ---

 This could be unnecessarily limiting, but would it really cause 
 much trouble?


I think you're right, I thought about this after I replied to 
you. It would be the logical next step. On the other hand, I 
wouldn't want to lose const borrowing, because it turned out to 
be a requirement for safe moving. But I think it can still be 
tracked internally (owner tracking is necessary for 
implementation anyway).



 Oct 04 2014



  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Saturday, 4 October 2014 at 19:22:45 UTC, Marc Schütz wrote:

 On Saturday, 4 October 2014 at 18:13:03 UTC, Ivan Timokhin 
 wrote:

 Also, would it really make much sense to track the owner 
 further than the assignment of a function's return value? That 
 seems to complicate things a lot by adding a hidden attribute 
 to a variable that is not only invisible at the declaration 
 (even though the full type is spelled out), but, in fact, 
 cannot be specified explicitly (because there's no syntax for 
 that, now that scope with owners is limited to function 
 signatures).

 How about this:
 ---
    scope(string) haystack, needle;
    // next assignment is okay, because `s` is guaranteed not 
 to outlive
    // `haystack`.
    scope(string) s = findSubstring(haystack, needle);
    // type of `s` is now scope(string), no additional 
 information
    // attached

    // so the next assignment is disallowed:
    //needle = s; // error!
 ---

 This could be unnecessarily limiting, but would it really 
 cause much trouble?


 I think you're right, I thought about this after I replied to 
 you. It would be the logical next step. On the other hand, I 
 wouldn't want to lose const borrowing, because it turned out to 
 be a requirement for safe moving. But I think it can still be 
 tracked internally (owner tracking is necessary for 
 implementation anyway).


Owner tracking is then completely limited to one expression. I 
think this will simplify the implementation a lot. Besides, it 
has precedences: uniqueness is also only tracked inside an 
expression, AFAIK.



 Oct 04 2014



  parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Saturday, 4 October 2014 at 19:26:01 UTC, Marc Schütz wrote:

 On Saturday, 4 October 2014 at 19:22:45 UTC, Marc Schütz wrote:

 On Saturday, 4 October 2014 at 18:13:03 UTC, Ivan Timokhin 
 wrote:

 Also, would it really make much sense to track the owner 
 further than the assignment of a function's return value? 
 That seems to complicate things a lot by adding a hidden 
 attribute to a variable that is not only invisible at the 
 declaration (even though the full type is spelled out), but, 
 in fact, cannot be specified explicitly (because there's no 
 syntax for that, now that scope with owners is limited to 
 function signatures).

 How about this:
 ---
   scope(string) haystack, needle;
   // next assignment is okay, because `s` is guaranteed not 
 to outlive
   // `haystack`.
   scope(string) s = findSubstring(haystack, needle);
   // type of `s` is now scope(string), no additional 
 information
   // attached

   // so the next assignment is disallowed:
   //needle = s; // error!
 ---

 This could be unnecessarily limiting, but would it really 
 cause much trouble?


 I think you're right, I thought about this after I replied to 
 you. It would be the logical next step. On the other hand, I 
 wouldn't want to lose const borrowing, because it turned out 
 to be a requirement for safe moving. But I think it can still 
 be tracked internally (owner tracking is necessary for 
 implementation anyway).


 Owner tracking is then completely limited to one expression. I 
 think this will simplify the implementation a lot. Besides, it 
 has precedences: uniqueness is also only tracked inside an 
 expression, AFAIK.


... and value range propagation.

I've worked in the changes talked about so far, and I think I've 
found a practicable solution for the forwarding problem. The idea 
is to make the owner constraints (i.e. the things we declare in 
function signatures) part of the type, but make types that only 
differ in their constraints equivalent. This avoids template 
bloat, but allows them to be forwarded where necessary. This is 
at the cost of needing to spread this information outwards when a 
template is going to be instantiated, which is however doable and 
inexpensive, as the compiler needs to do a full analysis of the 
template body anyway:

     scope!haystack(string) findSubstring(
         scope(string) haystack,
         scope(string) needle
     );

     auto trace(alias func, Args...)(Args args) {
         import std.conv : to;
         writeln("Calling " ~ func.stringof ~
                 "(" ~ args.to!string ~ ")");
         return func(args);
     }

     auto s = trace!findSubstring(haystack, needle);

     // expands to:
     scope!haystack(string) trace_findSubstring(
         scope(string) haystack,
         scope(string) needle
     ) {
         import std.conv : to;
         writeln("Calling findSubstring(" ~
                 args.to!string ~ ")");
         return findSubstring(args);
     }

Full proposal is here:
http://wiki.dlang.org/User:Schuetzm/scope



 Oct 23 2014



prev sibling  parent  Ivan Timokhin <timokhin.iv gmail.com>  writes:


11.09.2014 22:45, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" пишет:

...

 This troubles me the most, because currently return type of a function
 may depend only on types of its arguments, and there is a lot of
 templated code written in that assumption.


 I'm sorry, I don't understand what you mean here. This is clearly not
 true, neither for normal functions, nor for templates.


Well, me and my bad English. I was trying to say that currently, AFAIK,
for any symbols f, a, b, if f(a) is valid and the types of a and b 
match, then f(b) is also valid and types of f(a) and f(b) match. OTOH, 
if the types of a and b are different, types of f(a) and f(b) may also 
be different (because of overloading or templates like T f(T)(T x)).

This would change if scope became a type qualifier and f was something 
like scope!x(T) f(T)(scope(T) x).


...





 Sep 12 2014



prev sibling next sibling parent  Marco Leise <Marco.Leise gmx.de>  writes:


Am Thu, 11 Sep 2014 13:58:38 +0000
schrieb "Marc Sch=C3=BCtz" <schuetzm gmx.net>:


 PING
=20
 Now that there are again several GC related topics being=20
 discussed, I thought I'd bump this thread.
=20
 Would be nice if Walter and/or Andrei could have a look and share=20
 there opinions. Is this something worth pursuing further? Are=20
 there fundamental objections against it?


I just needed this again for a stack based allocator. It would
make such idioms safer where you return a pointer into an RAII
struct and need to make sure it doesn't outlive the struct.
It got me a nasty overwritten stack. I cannot comment on the
implementation, just that I have long felt it is missing.

--=20
Marco



 Sep 11 2014



prev sibling  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


PING again

On Thursday, 11 September 2014 at 13:58:40 UTC, Marc Schütz wrote:

 PING

 Now that there are again several GC related topics being 
 discussed, I thought I'd bump this thread.

 Would be nice if Walter and/or Andrei could have a look and 
 share there opinions. Is this something worth pursuing further? 
 Are there fundamental objections against it?

 On Sunday, 24 August 2014 at 13:14:45 UTC, Marc Schütz wrote:

 In the "Opportunities for D" thread, Walter again mentioned 
 the topics ref counting, GC, uniqueness, and borrowing, from 
 which a lively discussion developed [1]. I took this thread as 
 an opportunity to write down some ideas about these topics. 
 The result is a rather extensive proposal for the 
 implementation of borrowing, and its implementations:

 http://wiki.dlang.org/User:Schuetzm/scope

 This is not a real DIP, but before I put more work into 
 formalizing it, I'd like to hear some thoughts from the 
 languages gurus here:

 * Is this the general direction we want to go? Is it 
 acceptable in general?
 * Is the proposal internally consistent?
 * How big would the effort to implement it be? (I suspect it's 
 a large amount of work, but relatively straightforward.)

 [1] http://forum.dlang.org/thread/lphnen$1ml7$1 digitalmars.com







 Sep 19 2014



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 9/19/14, 4:34 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 PING again


Thanks for your work. I've put it on my todo list. -- Andrei



 Sep 19 2014



prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


Previous discussions:

http://www.digitalmars.com/d/archives/digitalmars/D/borrowed_pointers_vs_ref_232090.html

http://www.digitalmars.com/d/archives/digitalmars/D/RFC_scope_and_borrowing_240834.html

http://www.digitalmars.com/d/archives/digitalmars/D/scope_escaping_222858.html

http://www.digitalmars.com/d/archives/digitalmars/D/ref_is_unsafe_184935.html



 Sep 20 2014



  parent  "Vladimir Panteleev" <vladimir thecybershadow.net>  writes:


On Sunday, 21 September 2014 at 01:28:19 UTC, Walter Bright wrote:

 Previous discussions:


Forum links:

http://forum.dlang.org/?group=digitalmars.D&artnum=232090
http://forum.dlang.org/?group=digitalmars.D&artnum=240834
http://forum.dlang.org/?group=digitalmars.D&artnum=222858	
http://forum.dlang.org/?group=digitalmars.D&artnum=184935



 Sep 20 2014



prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


I think it's a well thought out proposal. Thanks for doing this!

A couple thoughts:

1. const can be both a storage class and a type constructor. Scope is only a 
storage class. The scope(int) syntax implies scope is a type constructor, too.

     const int* a;  // const used as storage class
     const(int*) b; // const used as type constructor

The type constructor syntax should be disallowed for const.


2. I think there is quite a bit of overlap between scope and ref. Essentially, 
ref does everything scope does, except deal with classes. I'm not terribly 
comfortable with such a large overlap, it implies something is wrong. I don't 
have an answer at the moment.



 Sep 20 2014



 next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2014-09-21 05:38, Walter Bright wrote:


 2. I think there is quite a bit of overlap between scope and ref.
 Essentially, ref does everything scope does, except deal with classes.
 I'm not terribly comfortable with such a large overlap, it implies
 something is wrong. I don't have an answer at the moment.


Am I missing something but isn't "ref" for passing something by 
reference instead of by value. "scope", in this proposal, is for dealing 
with lifetime? Or do you have any other proposal for what "ref" might 
become?

-- 
/Jacob Carlborg



 Sep 21 2014



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 9/21/2014 1:25 AM, Jacob Carlborg wrote:

 Am I missing something but isn't "ref" for passing something by reference
 instead of by value. "scope", in this proposal, is for dealing with lifetime?
Or
 do you have any other proposal for what "ref" might become?


See this discussion:

http://www.digitalmars.com/d/archives/digitalmars/D/borrowed_pointers_vs_ref_232090.html



 Sep 21 2014



prev sibling  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Sunday, 21 September 2014 at 03:39:24 UTC, Walter Bright wrote:

 I think it's a well thought out proposal. Thanks for doing this!

 A couple thoughts:

 1. const can be both a storage class and a type constructor. 
 Scope is only a storage class. The scope(int) syntax implies 
 scope is a type constructor, too.

     const int* a;  // const used as storage class
     const(int*) b; // const used as type constructor

 The type constructor syntax should be disallowed for const.


(... disallowed for _scope_, I assume)

I originally intended it to be part of the type. Ivan Timokhin 
pointed out severe problems with that [1], so I removed it from 
the proposal. The syntax is a remainder of that.

But before I remove it too, I have a question: Will it still be 
possible to use the storage class syntax for members of 
aggregates?

     struct S {
         scope!myAllocator int* p;
     }


 2. I think there is quite a bit of overlap between scope and 
 ref. Essentially, ref does everything scope does, except deal 
 with classes. I'm not terribly comfortable with such a large 
 overlap, it implies something is wrong. I don't have an answer 
 at the moment.



I also see the overlap, but there are also large differences to 
the point that I cannot see how the two concepts could be 
unified. For one, `ref` with classes (and reference types in 
general) is troublesome, and would introduce a double indirection 
for borrowing, when a simple copy of the reference would be 
sufficient. Then, `scope` could also for non-reference types. 
File handles come to mind, which are often integers, and just 
need to be copied.

[1] 
http://forum.dlang.org/thread/etjuormplgfbomwdrurp forum.dlang.org?page=3#post-lusirm:2421d9:241:40digitalmars.com



 Sep 21 2014



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 9/21/2014 2:11 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 On Sunday, 21 September 2014 at 03:39:24 UTC, Walter Bright wrote:

 I think it's a well thought out proposal. Thanks for doing this!

 A couple thoughts:

 1. const can be both a storage class and a type constructor. Scope is only a
 storage class. The scope(int) syntax implies scope is a type constructor, too.

     const int* a;  // const used as storage class
     const(int*) b; // const used as type constructor

 The type constructor syntax should be disallowed for const.


 (... disallowed for _scope_, I assume)


Yes, my mistake.



 I originally intended it to be part of the type. Ivan Timokhin pointed out
 severe problems with that [1], so I removed it from the proposal. The syntax is
 a remainder of that.

 But before I remove it too, I have a question: Will it still be possible to use
 the storage class syntax for members of aggregates?

      struct S {
          scope!myAllocator int* p;
      }


Possible, but exactly how that would work remains to be seen.



 Sep 21 2014



prev sibling  parent reply "bearophile" <bearophileHUGS lycos.com>  writes:


Marc Schütz:


 http://wiki.dlang.org/User:Schuetzm/scope


If a mutable argument of a function is tagged as unique, the type 
system guarantees that there are no other references to it. So 
can a function 'foo' like this be "strongly pure"?


int[] foo(unique int[] a) pure {
     a[0]++;
     return a;
}

Bye,
bearophile



 Sep 23 2014



 next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Tuesday, 23 September 2014 at 09:17:48 UTC, bearophile wrote:

 Marc Schütz:


 http://wiki.dlang.org/User:Schuetzm/scope


 If a mutable argument of a function is tagged as unique, the 
 type system guarantees that there are no other references to 
 it. So can a function 'foo' like this be "strongly pure"?


 int[] foo(unique int[] a) pure {
     a[0]++;
     return a;
 }


I think so. But note that `unique` is not part of my proposal, I 
merely used it in the example. I think it could be implemented 
relatively easily, because DMD internally already has a concept 
of uniqueness that is used for converting things to immutable 
implicitly. But this would be a different proposal.



 Sep 23 2014



  parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Tuesday, 23 September 2014 at 10:16:26 UTC, Marc Schütz wrote:

 On Tuesday, 23 September 2014 at 09:17:48 UTC, bearophile wrote:

 Marc Schütz:


 http://wiki.dlang.org/User:Schuetzm/scope


 If a mutable argument of a function is tagged as unique, the 
 type system guarantees that there are no other references to 
 it. So can a function 'foo' like this be "strongly pure"?


 int[] foo(unique int[] a) pure {
    a[0]++;
    return a;
 }


 I think so.


Ok, I take it back ;-) Steven is right. It is however the case 
that this function's return value would still be unique.



 Sep 23 2014



  parent reply Steven Schveighoffer <schveiguy yahoo.com>  writes:


On 9/23/14 6:26 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:

 Ok, I take it back ;-) Steven is right. It is however the case that this
 function's return value would still be unique.


Yes, it could be unique. I haven't read this thread really, so I don't 
know what has been proposed, but looking at the snippet, wouldn't you 
have to tag the return value? You tagged the parameter with unique.

-Steve



 Sep 23 2014



 next sibling parent  "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Tuesday, 23 September 2014 at 10:29:25 UTC, Steven 
Schveighoffer wrote:

 On 9/23/14 6:26 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 Ok, I take it back ;-) Steven is right. It is however the case 
 that this
 function's return value would still be unique.


 Yes, it could be unique. I haven't read this thread really, so 
 I don't know what has been proposed, but looking at the 
 snippet, wouldn't you have to tag the return value? You tagged 
 the parameter with unique.


Bearophile did, not me. But yes, you would have to, absent an 
extension to return type inference. As I already replied to him, 
uniqueness was really just used in an example because it made it 
cleaner; it's mostly unrelated to my proposal.



 Sep 23 2014



prev sibling  parent  "deadalnix" <deadalnix gmail.com>  writes:


On Tuesday, 23 September 2014 at 10:29:25 UTC, Steven
Schveighoffer wrote:

 On 9/23/14 6:26 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:

 Ok, I take it back ;-) Steven is right. It is however the case 
 that this
 function's return value would still be unique.


 Yes, it could be unique. I haven't read this thread really, so 
 I don't know what has been proposed, but looking at the 
 snippet, wouldn't you have to tag the return value? You tagged 
 the parameter with unique.

 -Steve


Unique is a bad name. You want to have various reference locally,
and the whole discussion is about scope.

Which bring us to the main point here, before discussing
borrowing, we'd better define ownership.



 Sep 23 2014



prev sibling  parent reply Steven Schveighoffer <schveiguy yahoo.com>  writes:


On 9/23/14 5:17 AM, bearophile wrote:

 Marc Schütz:


 http://wiki.dlang.org/User:Schuetzm/scope


 If a mutable argument of a function is tagged as unique, the type system
 guarantees that there are no other references to it. So can a function
 'foo' like this be "strongly pure"?


 int[] foo(unique int[] a) pure {
      a[0]++;
      return a;
 }


I don't think so. Strong pure function optimizations would not work for 
something like:

auto x = foo(a) ~ foo(a);

Which for a strong pure function could be optimized to:

auto r = foo(a);
auto x = r ~ r;

-Steve



 Sep 23 2014



  parent reply "bearophile" <bearophileHUGS lycos.com>  writes:


Steven Schveighoffer:


 int[] foo(unique int[] a) pure {


 ...
 I don't think so. Strong pure function optimizations would not 
 work for something like:

 auto x = foo(a) ~ foo(a);


This is similar to:

unique x1 = foo(a);
unique x2 = foo(a);
unique x = x1 ~ x2;

When the call to the first foo ends you have a x1 reference to 
the array data. Such reference x1 is unique, so now 'a' is not 
usable any more, you can't pass 'a' to foo once more.
I need to learn more about such stuff of linear typing.

Bye,
bearophile



 Sep 23 2014



  parent reply Steven Schveighoffer <schveiguy yahoo.com>  writes:


On 9/23/14 7:11 AM, bearophile wrote:

 Steven Schveighoffer:


 int[] foo(unique int[] a) pure {


 ...
 I don't think so. Strong pure function optimizations would not work
 for something like:

 auto x = foo(a) ~ foo(a);


 This is similar to:

 unique x1 = foo(a);
 unique x2 = foo(a);
 unique x = x1 ~ x2;

 When the call to the first foo ends you have a x1 reference to the array
 data. Such reference x1 is unique, so now 'a' is not usable any more,
 you can't pass 'a' to foo once more.
 I need to learn more about such stuff of linear typing.


This begs the question, what is the point of having "strong purity" if 
you can't optimize based on it?

-Steve



 Sep 23 2014



  parent  "bearophile" <bearophileHUGS lycos.com>  writes:


Steven Schveighoffer:


 This begs the question, what is the point of having "strong 
 purity" if you can't optimize based on it?


Linear typing gives some guarantees that help the GC a lot, and 
avoid some coding mistakes. And some people could answer you that 
having (strongly) pure functions is a quality regardless of 
optimizations, because it makes both unit testing and code 
understanding simpler :-) I am not very expert on this stuff, I 
need to learn more.

Bye,
bearophile



 Sep 23 2014