www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Deprecating this(this)

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
We need to have a simple recipe on how to define a canonical object. 
That would include requirements such as:

* should work with mutable, const, immutable, and shared
* the right way to define constructor(s)
* the right way to define copying
* the right way to define destructor(s)

I've asked my student Razvan to document the behavior of this(this). His 
work:

https://github.com/dlang/dmd/pull/8055
https://github.com/dlang/dlang.org/pull/2281
https://github.com/dlang/dlang.org/pull/2299

... reveals a puzzling array of behaviors. Sometimes the typechecking is 
wrong, too.

I think it's very important for us to have a simple, correct, and 
canonical way of defining structs in the D language that work with the 
language features: qualifiers, pure, safe, and nogc.

Once we have that, we can encapsulate desirable abstractions (such as 
 nogc safe collections that work in pure code), regardless of how 
difficult their implementations might be. It seems that currently 
this(this) does not allow us to do that.

Eduard, another student I work with, has made steps toward a collections 
library that rant into difficulties:

*  safe is achievable with relative ease

* immutable and const are very difficult, but we have an attack 
(assuming copy construction gets taken care of)

*  nogc is doable with a couple of conventions for allocators

* pure is difficult

* making them work together is very difficult

We need to offer features, tools, and guidance toward creating simple 
encapsulated types that work well with D's own abstractions. Once we 
have that we can build libraries to work satisfactorily for any domain.

I think the way to move forward is to deprecate this(this) entirely and 
create a DIP that allows people to define truly encapsulated structs. 
This is important, urgent, and of huge impact.

I am looking for folks to assist me in creating a DIP for that. There 
will be a _lot_ of work involved, so don't take it lightly.


Thanks,

Andrei
Mar 31 2018
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, March 31, 2018 19:38:06 Andrei Alexandrescu via Digitalmars-d 
wrote:

 We need to have a simple recipe on how to define a canonical object.
 That would include requirements such as:

 * should work with mutable, const, immutable, and shared
 * the right way to define constructor(s)
 * the right way to define copying
 * the right way to define destructor(s)

 I've asked my student Razvan to document the behavior of this(this). His
 work:

 https://github.com/dlang/dmd/pull/8055
 https://github.com/dlang/dlang.org/pull/2281
 https://github.com/dlang/dlang.org/pull/2299

 ... reveals a puzzling array of behaviors. Sometimes the typechecking is
 wrong, too.

 I think it's very important for us to have a simple, correct, and
 canonical way of defining structs in the D language that work with the
 language features: qualifiers, pure, safe, and nogc.

 Once we have that, we can encapsulate desirable abstractions (such as
  nogc safe collections that work in pure code), regardless of how
 difficult their implementations might be. It seems that currently
 this(this) does not allow us to do that.

 Eduard, another student I work with, has made steps toward a collections
 library that rant into difficulties:

 *  safe is achievable with relative ease

 * immutable and const are very difficult, but we have an attack
 (assuming copy construction gets taken care of)

 *  nogc is doable with a couple of conventions for allocators

 * pure is difficult

 * making them work together is very difficult

 We need to offer features, tools, and guidance toward creating simple
 encapsulated types that work well with D's own abstractions. Once we
 have that we can build libraries to work satisfactorily for any domain.

 I think the way to move forward is to deprecate this(this) entirely and
 create a DIP that allows people to define truly encapsulated structs.
 This is important, urgent, and of huge impact.

 I am looking for folks to assist me in creating a DIP for that. There
 will be a _lot_ of work involved, so don't take it lightly.


So, is the idea then that we'd switch to copy constructors?

- Jonathan M Davis
Mar 31 2018
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/18 8:25 PM, Jonathan M Davis wrote:

 So, is the idea then that we'd switch to copy constructors?


Something like that. We'll need to define them carefully to obey safety 
and purity.
Mar 31 2018
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Mar 31, 2018 at 07:38:06PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]

 Once we have that, we can encapsulate desirable abstractions (such as
  nogc safe collections that work in pure code), regardless of how
 difficult their implementations might be. It seems that currently
 this(this) does not allow us to do that.


What exactly is it about this(this) that blocks us from doing that?

Removing this(this) is going to be a huge breaking change far bigger
than, say, removing autodecoding ever will be.



 Eduard, another student I work with, has made steps toward a
 collections library that rant into difficulties:
 
 *  safe is achievable with relative ease
 
 * immutable and const are very difficult, but we have an attack
 (assuming copy construction gets taken care of)


A lot of us here have essentially given up on const except for a few
very narrow cases.  The transitive nature of const makes it extremely
difficult to work with in the general case, even though the simplest use
cases are workable.

One of the biggest stumbling blocks is that whenever ranges are
involved, const is practically out of the question, because even though
it can be made to work for most cases, there will almost always be that
one pathological case where it's impossible / too hard to work around,
and that ruins it for everything else, so that it's much simpler to just
avoid it altogether.

Also, as far as containers are concerned, the lack of a standard way to
construct head-mutable / tail-const types that works analogously with
built-in arrays makes it very difficult to write generic containers that
work well with const/immutable.  It's not too hard to make it work for
specific types, but very difficult to write a truly *generic* container
that can be deployed in all situations where const/immutable are
involved.


[...]

 * pure is difficult


[...]

The one nagging question I've been having about pure is: how much are we
actually taking advantage of the guarantees provided by pure?  We have
developed very clever ways of extending the traditional definition of
pure and invented creative ways of making more things pure, which is all
great.  But AFAIK the only place where it's actually taken advantage of
is to elide some redundant function calls inside a single expression.
And perhaps infer uniqueness in some cases for implicit casting to
immutable.

While these are indisputably useful, they seem so far to be only
relatively minor benefits.  If pure is indeed so difficult to support
generically, it begs the question, is it worth the effort just to gain
these niggling benefits?  Whatever happened to larger-scale benefits
conferred by purity?


T

-- 
Computers aren't intelligent; they only think they are.
Mar 31 2018
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/18 8:32 PM, H. S. Teoh wrote:

 On Sat, Mar 31, 2018 at 07:38:06PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]

 Once we have that, we can encapsulate desirable abstractions (such as
  nogc safe collections that work in pure code), regardless of how
 difficult their implementations might be. It seems that currently
 this(this) does not allow us to do that.


 
 What exactly is it about this(this) that blocks us from doing that?


See the updated docs. Too many bugs in design and implementation.


 Removing this(this) is going to be a huge breaking change far bigger
 than, say, removing autodecoding ever will be.


We're not removing it as much as evolving it: we define an alternate 
copying mechanism, and once that is in tip-top shape, we deprecate 
this(this).


 A lot of us here have essentially given up on const except for a few
 very narrow cases.  The transitive nature of const makes it extremely
 difficult to work with in the general case, even though the simplest use
 cases are workable.


Immutable is where it's at, in terms of usefulness. Const is a mere 
servant of it (and of mutable).


 One of the biggest stumbling blocks is that whenever ranges are
 involved, const is practically out of the question, because even though
 it can be made to work for most cases, there will almost always be that
 one pathological case where it's impossible / too hard to work around,
 and that ruins it for everything else, so that it's much simpler to just
 avoid it altogether.


Yah, the DIP might address that, too. Consider:

void fun(R)(R arr)
{
     pragma(msg, typeof(arr));
}
void main()
{
     immutable(int[]) arr = [ 1, 2, 3 ];
     pragma(msg, typeof(arr));
     fun(arr);
}

The program prints during compilation:

immutable(int[])
immutable(int)[]

Interesting! So the type of the array changes during template matching, 
which is an exception to the rule that templates always glom to the 
exact type passed.

This is a hack introduced in the compiler in response to the issues you 
mention. But we don't need hacks and special casing - we need a means 
for types to say "here's what needs to happen when a template parameter 
is matched against this type".

So the DIP would address manipulating qualified ranges as a perk.


 The one nagging question I've been having about pure is: how much are we
 actually taking advantage of the guarantees provided by pure?


Very little, but that doesn't matter. The problem is it's 
underspecified. So now it's like a vague threat - whenever we mess with 
fear somebody comes asking, but what about an aggressive compiler doing 
some unexpected optimizations based on such and such interpretation?

We need to lock pure down.


Andrei
Mar 31 2018
parent reply ag0aep6g <anonymous example.com> writes:
On 04/01/2018 03:08 AM, Andrei Alexandrescu wrote:

 On 3/31/18 8:32 PM, H. S. Teoh wrote:


[...]

 What exactly is it about this(this) that blocks us from doing that?


 
 See the updated docs. Too many bugs in design and implementation.
 

 Removing this(this) is going to be a huge breaking change far bigger
 than, say, removing autodecoding ever will be.


 
 We're not removing it as much as evolving it: we define an alternate 
 copying mechanism, and once that is in tip-top shape, we deprecate 
 this(this).


Is there a fundamental flaw in the postblit idea, or are you just going 
to give postblit a new syntax, and try to avoid all the issues that 
`this(this)` currently has?

If there's a fundamental flaw, I'd be interested in what it is. I can't 
make it out in your additions to the spec, if it's in there. I can see 
that `this(this)` is  a mess, but it also looks like a lot could be 
fixed. For example, how it interacts with const/immutable is ridiculous, 
but that could probably be fixed.

If you're just going for a clean slate, I can see the appeal. You avoid 
dealing with the hard breakage that fixing `this(this)` would most 
probably bring.
Apr 01 2018
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, April 01, 2018 14:55:07 ag0aep6g via Digitalmars-d wrote:

 On 04/01/2018 03:08 AM, Andrei Alexandrescu wrote:

 On 3/31/18 8:32 PM, H. S. Teoh wrote:


 [...]


 What exactly is it about this(this) that blocks us from doing that?


 See the updated docs. Too many bugs in design and implementation.


 Removing this(this) is going to be a huge breaking change far bigger
 than, say, removing autodecoding ever will be.


 We're not removing it as much as evolving it: we define an alternate
 copying mechanism, and once that is in tip-top shape, we deprecate
 this(this).


 Is there a fundamental flaw in the postblit idea, or are you just going
 to give postblit a new syntax, and try to avoid all the issues that
 `this(this)` currently has?

 If there's a fundamental flaw, I'd be interested in what it is. I can't
 make it out in your additions to the spec, if it's in there. I can see
 that `this(this)` is  a mess, but it also looks like a lot could be
 fixed. For example, how it interacts with const/immutable is ridiculous,
 but that could probably be fixed.


One issue is that postblit constructors fundamentally don't work with const.
The problem is that a postblit constructor works by copying the object and
_then_ mutating it, and you can't mutate a const object. To cleanly deal
with const, you need something more like a copy constructor where you
initialize it with the adjusted values directly rather than mutating the
copy.


 If you're just going for a clean slate, I can see the appeal. You avoid
 dealing with the hard breakage that fixing `this(this)` would most
 probably bring.


Avoiding any breakage would be ideal, but it's unlikely that that can be
done if we want to make copying const objects work properly - not unless we
did something like use postblit constructors for mutable objects and copy
constructors for const objects, and that would just cause other problems
(including having to duplicate the code that deals with copying an object if
it's going to work with const). On the bright side, if we're replacing
postblit constructors with some other type of constructor for copying, it
should be a pretty straightforward process. But there isn't much point in
worrying about how much breakage there's going to be before we really know
where we want to go with this. At this point, it's just clear that as things
stand, postblit constructors have some definite problems (some which are
implementation issues and some which are language design issues), and based
on past discussions on this and previous attempts to fix some of the
problems with postblit constructors, it seems pretty unlikely that we can
fully fix postblit constructors. I'm sure that we could fix some of the
issues, but others (most notably, the issues with const) seem pretty
intractable.

- Jonathan M Davis
Apr 01 2018
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 1 April 2018 at 13:37:43 UTC, Jonathan M Davis wrote:

 One issue is that postblit constructors fundamentally don't 
 work with const. The problem is that a postblit constructor 
 works by copying the object and _then_ mutating it, and you 
 can't mutate a const object. To cleanly deal with const, you 
 need something more like a copy constructor where you 
 initialize it with the adjusted values directly rather than 
 mutating the copy.


I've always wondered about that, is the difference between that 
anything more than philosophical? Put another way if a this(this) 
is weakly pure, is there any safety issues with the compiler 
permitting the mutation on a (non-shared? not sure if this would 
be a requirement) const object? I'm not sure what the spec says, 
but if you take the view that the const object is no fully 
initialised until the postblit is done, then I don't see the 
problem.
Apr 01 2018
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Apr 01, 2018 at 02:31:06PM +0000, Nicholas Wilson via Digitalmars-d
wrote:

 On Sunday, 1 April 2018 at 13:37:43 UTC, Jonathan M Davis wrote:

 One issue is that postblit constructors fundamentally don't work
 with const. The problem is that a postblit constructor works by
 copying the object and _then_ mutating it, and you can't mutate a
 const object. To cleanly deal with const, you need something more
 like a copy constructor where you initialize it with the adjusted
 values directly rather than mutating the copy.


 
 I've always wondered about that, is the difference between that
 anything more than philosophical? Put another way if a this(this) is
 weakly pure, is there any safety issues with the compiler permitting
 the mutation on a (non-shared? not sure if this would be a
 requirement) const object? I'm not sure what the spec says, but if you
 take the view that the const object is no fully initialised until the
 postblit is done, then I don't see the problem.


Yeah I've been wondering about this too.  I mean, currently, ctors are
allowed to assign to immutable fields, because, well, it's
initialization, not after-the-fact mutation.  Why can't we extend this
to postblits?  Now, there's certainly the issue of ctors "leaking"
mutable references to immutable fields if not implemented properly, but
since postblits are run after a built-into-the-language copying of the
original, supposedly an opaque process, it seems reasonable enough to
allow the postblit to be regarded as initialization and able to assign
to const/immutable fields once.

At the very least, allow rebinding of const/immutable references in the
postblit. (Allowing straight-out reassignment may have adverse effects
if the copy shares a reference to an immutable object.)


T

-- 
What is Matter, what is Mind? Never Mind, it doesn't Matter.
Apr 02 2018
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, April 02, 2018 08:56:41 H. S. Teoh via Digitalmars-d wrote:

 On Sun, Apr 01, 2018 at 02:31:06PM +0000, Nicholas Wilson via Digitalmars-


d wrote:

 On Sunday, 1 April 2018 at 13:37:43 UTC, Jonathan M Davis wrote:

 One issue is that postblit constructors fundamentally don't work
 with const. The problem is that a postblit constructor works by
 copying the object and _then_ mutating it, and you can't mutate a
 const object. To cleanly deal with const, you need something more
 like a copy constructor where you initialize it with the adjusted
 values directly rather than mutating the copy.


 I've always wondered about that, is the difference between that
 anything more than philosophical? Put another way if a this(this) is
 weakly pure, is there any safety issues with the compiler permitting
 the mutation on a (non-shared? not sure if this would be a
 requirement) const object? I'm not sure what the spec says, but if you
 take the view that the const object is no fully initialised until the
 postblit is done, then I don't see the problem.


 Yeah I've been wondering about this too.  I mean, currently, ctors are
 allowed to assign to immutable fields, because, well, it's
 initialization, not after-the-fact mutation.  Why can't we extend this
 to postblits?  Now, there's certainly the issue of ctors "leaking"
 mutable references to immutable fields if not implemented properly, but
 since postblits are run after a built-into-the-language copying of the
 original, supposedly an opaque process, it seems reasonable enough to
 allow the postblit to be regarded as initialization and able to assign
 to const/immutable fields once.

 At the very least, allow rebinding of const/immutable references in the
 postblit. (Allowing straight-out reassignment may have adverse effects
 if the copy shares a reference to an immutable object.)


The core problem is that in a postblit, you're reading a member variable and
then assigning to it, which violates const. For some simple cases, we could
essentially make the member variables tail-const within the postblit,
because we could safely say that the member variable was a distinct copy and
that overwriting it wouldn't really cause problems - e.g. assigning to an
int wouldn't really be a problem, and assigning to const(T)* wouldn't really
be a problem. For classes, it's a bit hinky, because there's no such thing
as tail-const for classes, but they're conceptually the same as const(T)*,
so we could cheat in a postblit and still let the reference be assigned
once. However, the real problem is once you start dealing with structs. Not
only is there no real concept of tail-const with structs, and they can
contain any combination of value types, reference types, pseudo-reference
types, etc. but they can overload opAssign and postblit. And of course, the
postblit constructor for member variables is run before the postblit
constructor for the object containing them, and there's no guarantee that
the postblit actually did _anything_ involving making a deep copy (e.g. it
could simply have been printing out that the object was copied).

So, figuring out how independent a copy the struct is from the original
isn't necessarily very straightforward, and even if it were, how would we do
the equivalent of const(T)* to allow the struct to be reassigned without
mucking with any data that's not independent? It might contain members that
have their data directly embedded in the struct and members which are
reference types, but it's still one unit, and stuff like opAssign is not
designed with the idea that you just overwrite some of the struct. And of
course, once opAssign is overloaded, who knows what the semantics of
assigning to the struct are. As such, how do you reason about the safety of
relaxing the type system to allow assignment to a struct that has overloaded
opAssign?

At one point, Kenji was working on a solution to the problem (I _think_ that
he had a DIP on it, but it's been a while, so I don't remember), and as I
recall, Walter and Andrei vetoed it, because it was too complicated. And I
don't even know if he actually, fully solved the problem.

Ultimately, this is all much, much cleaner if copying an object involves
initializing it directly rather than doing a shallow copy and then doing a
deep copy and reassigning pieces of the object. That sucks for the simple
cases, because if you don't care about stuff like const, and your object has
a bunch of members in it, with a postblit constructor, you might only have
to manually do something with a few of them (whereas in C++, you'd have to
list them all individually), but as far as I can tell, it's pretty much
required for the complex cases.

Off the top of my head, what I would probably do if I were redesigning this
would be to go with copy constructors but improve the syntax so that you
don't have to list any of the member variables unless you're actually going
to be giving them different values. If that could be done cleanly, then it
might end up being a bit more verbose depending on what we had to do, or it
might look pretty much the same as now, except that you'd then be assigning
from a parameter (be it implicit or explicit) representing the object being
copied rather than the object being constructed. Either way, I wouldn't go
with an actual postblit constructor. It's a great idea if const isn't a
thing, but since const is a thing, it's not so great an idea. It also isn't
possible with postblit constructors to look at the original object to get
stuff like its address, which could matter in the uses cases that the DIP
for opMove is trying to solve. So, I'm inclined to think that copy
constructors would ultimately be a better solution - especially if we can
keep the syntax such that you don't have to explicitly initialize any
members that you just want copied. At that point, syntactically, they'd be
pretty much the same as postblit constructors.

- Jonathan M Davis
Apr 02 2018
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/1/18 9:37 AM, Jonathan M Davis wrote:

 One issue is that postblit constructors fundamentally don't work with const.


Actually they do...
Apr 01 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, April 01, 2018 10:31:46 Andrei Alexandrescu via Digitalmars-d 
wrote:

 On 4/1/18 9:37 AM, Jonathan M Davis wrote:

 One issue is that postblit constructors fundamentally don't work with
 const.


 Actually they do...


How so? In the postblit, you're dealing with a copy of an object where
everything is already initialized. Mutating the object would violate const.
It could be made to work for primitive types that the compiler understands
and knows that the member variable is truly independent - e.g. it could be
allowed to mutate an int, or it could be allowed to mutate a pointer while
treating what it points to as const, but as soon as you're dealing with
user-defined types, you're screwed - especially if you're dealing with
something like a struct with a user-defined opAssign. You're reading an
existing value and then mutating it, and it has to be at least tail-const,
because the original was const - and tail-const is pretty meaningless for
structs and can't really be represented in the type system for classes. So,
I don't see how postblit could be made to work with a const object of any
real complexity. It can be made to work in some corner cases but not in
general.

Kenji worked on a solution to the problem with const and postblit several
years ago (and I'm not sure how close he got to really solving it), but as I
recall, you and Walter shot it down because it was overly complicated. How
are you proposing that const work with postblit?

- Jonathan M Davis
Apr 01 2018
prev sibling parent reply ag0aep6g <anonymous example.com> writes:
On Sunday, 1 April 2018 at 13:37:43 UTC, Jonathan M Davis wrote:

 One issue is that postblit constructors fundamentally don't 
 work with const. The problem is that a postblit constructor 
 works by copying the object and _then_ mutating it, and you 
 can't mutate a const object.


I'm not so sure if that's fundamental. Can't we just say that the 
copy is head-mutable at the time when the postblit function is 
called, and it only becomes fully const after that?

The destination can't be const/immutable already, or you wouldn't 
be able to write there anyway.
Apr 01 2018
next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 1 April 2018 at 14:34:01 UTC, ag0aep6g wrote:

 On Sunday, 1 April 2018 at 13:37:43 UTC, Jonathan M Davis wrote:

 One issue is that postblit constructors fundamentally don't 
 work with const. The problem is that a postblit constructor 
 works by copying the object and _then_ mutating it, and you 
 can't mutate a const object.


 I'm not so sure if that's fundamental. Can't we just say that 
 the copy is head-mutable at the time when the postblit function 
 is called, and it only becomes fully const after that?

 The destination can't be const/immutable already, or you 
 wouldn't be able to write there anyway.


Ah, you said it much better than I did.
Apr 01 2018
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/1/18 10:34 AM, ag0aep6g wrote:

 On Sunday, 1 April 2018 at 13:37:43 UTC, Jonathan M Davis wrote:

 One issue is that postblit constructors fundamentally don't work with 
 const. The problem is that a postblit constructor works by copying the 
 object and _then_ mutating it, and you can't mutate a const object.


 
 I'm not so sure if that's fundamental. Can't we just say that the copy 
 is head-mutable at the time when the postblit function is called, and it 
 only becomes fully const after that?
 
 The destination can't be const/immutable already, or you wouldn't be 
 able to write there anyway.


Yes, precisely what I had been arguing here: 
https://issues.dlang.org/show_bug.cgi?id=18417#c5, however 
const/immutable postblit was recently deprecated by: 
https://github.com/dlang/dmd/pull/8032

So I don't think D is going to allow const postblit any more. Maybe not 
even postblit any more.

-Steve
Apr 02 2018
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 2 April 2018 at 14:07:21 UTC, Steven Schveighoffer 
wrote:

 On 4/1/18 10:34 AM, ag0aep6g wrote:

 On Sunday, 1 April 2018 at 13:37:43 UTC, Jonathan M Davis 
 wrote:

 One issue is that postblit constructors fundamentally don't 
 work with const. The problem is that a postblit constructor 
 works by copying the object and _then_ mutating it, and you 
 can't mutate a const object.


 
 I'm not so sure if that's fundamental. Can't we just say that 
 the copy is head-mutable at the time when the postblit 
 function is called, and it only becomes fully const after that?
 
 The destination can't be const/immutable already, or you 
 wouldn't be able to write there anyway.


 Yes, precisely what I had been arguing here: 
 https://issues.dlang.org/show_bug.cgi?id=18417#c5, however 
 const/immutable postblit was recently deprecated by: 
 https://github.com/dlang/dmd/pull/8032

 So I don't think D is going to allow const postblit any more. 
 Maybe not even postblit any more.

 -Steve


Andrei did post an example where treating the designation as 
mutable, stuffing references to mutable data into is and then 
treating it as immutable leaves you with an immutable reference 
to immutable data.

However i think that loophole is fixed if you only allow 
assignment to const/immutable from a pure postblit.
Apr 02 2018
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/2/18 10:24 AM, Nicholas Wilson wrote:

 On Monday, 2 April 2018 at 14:07:21 UTC, Steven Schveighoffer wrote:

 On 4/1/18 10:34 AM, ag0aep6g wrote:

 On Sunday, 1 April 2018 at 13:37:43 UTC, Jonathan M Davis wrote:

 One issue is that postblit constructors fundamentally don't work 
 with const. The problem is that a postblit constructor works by 
 copying the object and _then_ mutating it, and you can't mutate a 
 const object.


 I'm not so sure if that's fundamental. Can't we just say that the 
 copy is head-mutable at the time when the postblit function is 
 called, and it only becomes fully const after that?

 The destination can't be const/immutable already, or you wouldn't be 
 able to write there anyway.


 Yes, precisely what I had been arguing here: 
 https://issues.dlang.org/show_bug.cgi?id=18417#c5, however 
 const/immutable postblit was recently deprecated by: 
 https://github.com/dlang/dmd/pull/8032

 So I don't think D is going to allow const postblit any more. Maybe 
 not even postblit any more.


 
 Andrei did post an example where treating the designation as mutable, 
 stuffing references to mutable data into is and then treating it as 
 immutable leaves you with an immutable reference to immutable data.


I think you meant immutable reference to mutable data.


 
 However i think that loophole is fixed if you only allow assignment to 
 const/immutable from a pure postblit.
 


It's pretty straightforward I would think: If a postblit is being called 
on an immutable, it *must* be new data, because if it was data that just 
got overwritten, that data couldn't possibly be immutable. Therefore, 
there's leeway to mutate the head at that point, because nothing else 
can see it yet. Doesn't need to be pure.

Same leeway as an immutable constructor I think.

But all this discussion is moot if the postblit is going away.

-Steve
Apr 02 2018
prev sibling parent reply ag0aep6g <anonymous example.com> writes:
On Monday, 2 April 2018 at 14:24:20 UTC, Nicholas Wilson wrote:

 On Monday, 2 April 2018 at 14:07:21 UTC, Steven Schveighoffer 
 wrote:

 On 4/1/18 10:34 AM, ag0aep6g wrote:




[...]

 I'm not so sure if that's fundamental. Can't we just say that 
 the copy is head-mutable at the time when the postblit 
 function is called, and it only becomes fully const after 
 that?






[...]

 Yes, precisely what I had been arguing here: 
 https://issues.dlang.org/show_bug.cgi?id=18417#c5




[...]

 Andrei did post an example where treating the designation as 
 mutable, stuffing references to mutable data into is and then 
 treating it as immutable leaves you with an immutable reference 
 to immutable data.


This one?

----
int[] sneaky;
struct A
{
     private int[] innocent;
     this(this)
     {
         sneaky = innocent;
     }
}
void main()
{
     immutable a = A([1, 2, 3]);
     auto b = a;
     sneaky[1] = 42; // oops
     import std.stdio;
     writeln(a.innocent); // ooooops
}
----

That wouldn't be possible if `innocent` were only head-mutable in 
the postblit function, instead of fully mutable as it is 
currently (bug).

`innocent` would be typed as `immutable(int)[]`, and you could 
not assign it to the fully mutable `sneaky`.


 However i think that loophole is fixed if you only allow 
 assignment to const/immutable from a pure postblit.


Still must be head-mutable only. With fully mutable, you can 
mutate data that is seen as immutable elsewehre, even if the 
postblit function is pure:

----
struct S
{
     int* x;
     this(this) pure { *x = 13; }
}
void main()
{
     auto s = immutable S(new int(42));
     auto s2 = s;
     assert(*s.x == 42); /* fails; immutability has been broken */
}
----

Issue 18357 covers both Andrei's example, and this one.
https://issues.dlang.org/show_bug.cgi?id=18357
Apr 02 2018
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/02/2018 10:59 AM, ag0aep6g wrote:

 
 That wouldn't be possible if `innocent` were only head-mutable in the 
 postblit function, instead of fully mutable as it is currently (bug).
 
 `innocent` would be typed as `immutable(int)[]`, and you could not 
 assign it to the fully mutable `sneaky`.


Problem is we don't have head-mutable in the language. Yes, for built-in 
slices the mechanism is simple - just change qualifier(T[]) to 
qualifier(T)[]. For a struct S, there is no way to convert from 
qualifier(S) to tailqualifier(S).

I plan to attack this directly in the DIP - provide a way for structs to 
express "here's what implicit conversion should be applied when doing 
template matching".

Andrei
Apr 02 2018
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/2/18 11:57 AM, Andrei Alexandrescu wrote:

 On 04/02/2018 10:59 AM, ag0aep6g wrote:

 That wouldn't be possible if `innocent` were only head-mutable in the 
 postblit function, instead of fully mutable as it is currently (bug).

 `innocent` would be typed as `immutable(int)[]`, and you could not 
 assign it to the fully mutable `sneaky`.


 
 Problem is we don't have head-mutable in the language.


This is the wrong way to look at it. Head mutable is a specialized 
concept already implemented in constructors:

struct S
{
    int x;
    int *ptr;
    int *ptr2;
    this(int xval, immutable int *ptrval, int *cantuse) immutable
    {
       x = xval; // ok, head mutable
       //x = x + 1; // error, immutable field initialized multiple times
       ptr = ptrval; // ok
       // ptr2 = cantuse; // error, can't assign mutable to immutable
    }
}

We could have the same mechanism for postblit. Essentially, you should 
be able to assign immutable fields once, but they shouldn't lose their 
const protections (note the cantuse example).

As was mentioned, because postblit on an immutable (or const) is ONLY 
allowed for new data, there shouldn't be an issue.

-Steve
Apr 02 2018
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/02/2018 12:53 PM, Steven Schveighoffer wrote:

 On 4/2/18 11:57 AM, Andrei Alexandrescu wrote:

 On 04/02/2018 10:59 AM, ag0aep6g wrote:

 That wouldn't be possible if `innocent` were only head-mutable in the 
 postblit function, instead of fully mutable as it is currently (bug).

 `innocent` would be typed as `immutable(int)[]`, and you could not 
 assign it to the fully mutable `sneaky`.


 Problem is we don't have head-mutable in the language.


 
 This is the wrong way to look at it. Head mutable is a specialized 
 concept already implemented in constructors:
 
 struct S
 {
     int x;
     int *ptr;
     int *ptr2;
     this(int xval, immutable int *ptrval, int *cantuse) immutable
     {
        x = xval; // ok, head mutable
        //x = x + 1; // error, immutable field initialized multiple times
        ptr = ptrval; // ok
        // ptr2 = cantuse; // error, can't assign mutable to immutable
     }
 }


I've asked Razvan to document how immutable constructors are exactly 
typechecked. My understanding is that they don't rely on some form of 
internal "head const" but instead on simple data flow - the first 
assignment counts as a constructor call. Consider the following example, 
which contains opaque methods instead of built-ins:

struct S
{
     int[] payload;
     S* another;
     this(int) immutable;
}

struct HasS
{
     S member;
     this(int  x) immutable
     {
         member = immutable S(1);
     }
}

This fails to link; the assignment is just a call to the (declared but 
undefined) constructor.

Such behavior is nice and easy to generalize to copy construction.


 We could have the same mechanism for postblit. Essentially, you should 
 be able to assign immutable fields once, but they shouldn't lose their 
 const protections (note the cantuse example).


Yes, with the distinction that is not "assign" but really "construction 
with assignment syntax".


 As was mentioned, because postblit on an immutable (or const) is ONLY 
 allowed for new data, there shouldn't be an issue.


The problem with postblit is there's "double construction", one done by 
the compiler, after which the user may want to assign something else. 
That's more difficult to typecheck than direct initialization.


Andrei
Apr 02 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/2/18 1:08 PM, Andrei Alexandrescu wrote:

 On 04/02/2018 12:53 PM, Steven Schveighoffer wrote:

 As was mentioned, because postblit on an immutable (or const) is ONLY 
 allowed for new data, there shouldn't be an issue.


 
 The problem with postblit is there's "double construction", one done by 
 the compiler, after which the user may want to assign something else. 
 That's more difficult to typecheck than direct initialization.


I was going to argue that it's not full construction -- it's just a copy 
of bits. But I am incorrect.

My understanding was that postblit is called after the bits are copied, 
but that isn't the case. Currently, the postblit of members is called 
BEFORE the postblit of the container (or maybe as part of the postblit 
of the container, but always at the start). This means that a member is 
not a moved copy at that point, but a fully postblitted item. This also 
means that it's not safe to assume head-const-ness.

This makes some sense -- you don't want to have to deal with member 
postblits if you don't have to. But it also makes it impossible to 
intercept copying to do something different (as you would need to do for 
this case).

Indeed, something other than the current postblit mechanism looks more 
attractive and powerful, even if it isn't as straightforward.

-Steve
Apr 02 2018
prev sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 2 Apr 2018 11:57:55 -0400
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 Problem is we don't have head-mutable in the language. Yes, for built-in 
 slices the mechanism is simple - just change qualifier(T[]) to 
 qualifier(T)[]. For a struct S, there is no way to convert from 
 qualifier(S) to tailqualifier(S).
 
 I plan to attack this directly in the DIP - provide a way for structs to 
 express "here's what implicit conversion should be applied when doing 
 template matching".
 
 Andrei


You are hitting a prominent type system flaw here. What may
look like a hurdle on the path to fix this(this) is also at
the core of getting "shared" into a good shape and probably
affects how we will discuss "immutable destructors" and their
kin in the future. The question is "How transitive is a
qualifier when we strip it top-level on an aggregate?"

In https://issues.dlang.org/show_bug.cgi?id=8295 I've been
arguing for removing all qualifiers on shallow copies and the
case you mentioned where top level qualifiers are stripped for
template matching reconfirms me that there is generally some
merit to that semantic, that should be explored.

Shared structs need elaborate code to be copied, that's for
sure. There may be a mutex to be used or values may be copied
using atomic loads. The result would be what you dubbed
"tailqualifier(S)". I.e. in case of shared, a thread-local
copy of the fields that make up the struct.

But then it starts to become messy:
* Are there cases where we want references contained in the
  struct to become unshared, too?
* If yes, what if these references were marked shared
  themselves in the struct's definition?
* If all fields become unshared, shouldn't the now superfluous
  mutex be removed from the struct? If so, what started out as
  a bit blit, now produces a different type entirely.

I'm interested to hear more on your thoughs on
"tailqualifier(S)".

-- 
Marco
Apr 02 2018
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, April 02, 2018 20:47:31 Marco Leise via Digitalmars-d wrote:

 Am Mon, 2 Apr 2018 11:57:55 -0400

 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 Problem is we don't have head-mutable in the language. Yes, for built-in
 slices the mechanism is simple - just change qualifier(T[]) to
 qualifier(T)[]. For a struct S, there is no way to convert from
 qualifier(S) to tailqualifier(S).

 I plan to attack this directly in the DIP - provide a way for structs to
 express "here's what implicit conversion should be applied when doing
 template matching".

 Andrei


 You are hitting a prominent type system flaw here. What may
 look like a hurdle on the path to fix this(this) is also at
 the core of getting "shared" into a good shape and probably
 affects how we will discuss "immutable destructors" and their
 kin in the future. The question is "How transitive is a
 qualifier when we strip it top-level on an aggregate?"

 In https://issues.dlang.org/show_bug.cgi?id=8295 I've been
 arguing for removing all qualifiers on shallow copies and the
 case you mentioned where top level qualifiers are stripped for
 template matching reconfirms me that there is generally some
 merit to that semantic, that should be explored.

 Shared structs need elaborate code to be copied, that's for
 sure. There may be a mutex to be used or values may be copied
 using atomic loads. The result would be what you dubbed
 "tailqualifier(S)". I.e. in case of shared, a thread-local
 copy of the fields that make up the struct.

 But then it starts to become messy:
 * Are there cases where we want references contained in the
   struct to become unshared, too?
 * If yes, what if these references were marked shared
   themselves in the struct's definition?
 * If all fields become unshared, shouldn't the now superfluous
   mutex be removed from the struct? If so, what started out as
   a bit blit, now produces a different type entirely.

 I'm interested to hear more on your thoughs on
 "tailqualifier(S)".


Copying shared structs is a bit sketchy in general. In theory, we've been
trying to make it so non-atomic operations on shared objects aren't legal,
because they're not safe unless the object is protected by a mutex. So, the
normal way to deal with shared if you want to do anything with it is to
protect a section of code with a mutex and temporarily cast away shared
within that section of code (if we had synchronized classes, then in some
cases, that cast would be automatic, but as it stands, it's always manual).
And if doing a non-atomic operation on a shared object isn't legal, then
copying a shared object doesn't really make sense. In that case, I would
expect that copying a shared object wouldn't be legal (and as such the way
to copy a shared object would be to protect that with a mutex and cast away
shared when it's copied, and then cast the new object to shared if
appropriate). On the other hand, that could get _really_ annoying, and if we
were dealing with a copy constructor rather than a postblit constructor,
then it would probably be possible to lock a mutex and inside the copy
constructor so that the copy and whatever casting was necessary could be
done safely inside the struct rather than having to externally protect it
with a mutex and cast away shared there.

So, I don't know what the answer is, but if we're trying to make operations
which aren't guaranteed to be thread-safe illegal on shared objects, then we
have a bit of a problem with what to do with copying shared objects (or
assigning shared objects). But I suppose that part of the problem is that
while the _idea_ of shared is very well defined, the actual details aren't
(e.g. a number of non-atomic operations aren't legal on shared objects, but
some are). And copying is definitely one area where that has not been
properly sorted out yet.

- Jonathan M Davis
Apr 02 2018
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/02/2018 02:47 PM, Marco Leise wrote:

 Am Mon, 2 Apr 2018 11:57:55 -0400
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:
 

 Problem is we don't have head-mutable in the language. Yes, for built-in
 slices the mechanism is simple - just change qualifier(T[]) to
 qualifier(T)[]. For a struct S, there is no way to convert from
 qualifier(S) to tailqualifier(S).

 I plan to attack this directly in the DIP - provide a way for structs to
 express "here's what implicit conversion should be applied when doing
 template matching".

 Andrei


 
 You are hitting a prominent type system flaw here. What may
 look like a hurdle on the path to fix this(this) is also at
 the core of getting "shared" into a good shape and probably
 affects how we will discuss "immutable destructors" and their
 kin in the future. The question is "How transitive is a
 qualifier when we strip it top-level on an aggregate?"


Roger. My hope is to solve that for primitive types, then use that to 
typecheck constructors and destructors, then use the signatures of 
(typechecked) constructors and destructors to address composition. 
Ideally we'd get away without defining another kind of qualifier - 
 tail(const) or whatever. That would complicate the language a great deal.


Andrei
Apr 03 2018
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/1/18 8:55 AM, ag0aep6g wrote:

 On 04/01/2018 03:08 AM, Andrei Alexandrescu wrote:

 On 3/31/18 8:32 PM, H. S. Teoh wrote:


 [...]

 What exactly is it about this(this) that blocks us from doing that?


 See the updated docs. Too many bugs in design and implementation.


 Removing this(this) is going to be a huge breaking change far bigger
 than, say, removing autodecoding ever will be.


 We're not removing it as much as evolving it: we define an alternate 
 copying mechanism, and once that is in tip-top shape, we deprecate 
 this(this).


 
 Is there a fundamental flaw in the postblit idea, or are you just going 
 to give postblit a new syntax, and try to avoid all the issues that 
 `this(this)` currently has?
 
 If there's a fundamental flaw, I'd be interested in what it is. I can't 
 make it out in your additions to the spec, if it's in there. I can see 
 that `this(this)` is  a mess, but it also looks like a lot could be 
 fixed. For example, how it interacts with const/immutable is ridiculous, 
 but that could probably be fixed.
 
 If you're just going for a clean slate, I can see the appeal. You avoid 
 dealing with the hard breakage that fixing `this(this)` would most 
 probably bring.


There's a mix of fundamental flaws and bugs. I'll get to the flaws in a 
second. About the bugs: people have altered their code in various ways 
to work with the bizarre semantics of this(this). Now, if we fix various 
bugs in this(this) by virtually redefining it, then we'll break a lot of 
code in a lot of ways. To wit, we fixed a small issue and it already 
created problems: https://github.com/dlang/dmd/pull/8032. That didn't 
contribute to the decision but is quite illustrative.

I found two fundamental flaws with this(this):

1. For immutable objects, typechecking in the presence of successive 
modifications of data (first assignment by the compiler, then 
modification by the user) is very difficult if not impossible. I don't 
know how to do it. The single initialization model (raw/cooked) used 
currently in regular immutable constructors works reasonably well and is 
robust.

2. For shared objects, the part done by the compiler and the part done 
by this(this) should be synchronized together. This makes it impossible 
for the user to e.g. define a struct that gets copied atomically.

There'd be an additional issue - this(this) is non-templated, which 
requires combinatorial additions when qualifiers are present on the 
source or destination side.

Please note that fixing one or two of these issues doesn't make 
this(this) viable - I'm mentioning various issues, each of which is a 
showstopper. Nevertheless knowing them is necessary so we don't make the 
same mistake again!


Andrei
Apr 01 2018
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 1 April 2018 at 14:31:24 UTC, Andrei Alexandrescu 
wrote:

 There's a mix of fundamental flaws and bugs. I'll get to the 
 flaws in a second. About the bugs: people have altered their 
 code in various ways to work with the bizarre semantics of 
 this(this). Now, if we fix various bugs in this(this) by 
 virtually redefining it, then we'll break a lot of code in a 
 lot of ways. To wit, we fixed a small issue and it already 
 created problems: https://github.com/dlang/dmd/pull/8032. That 
 didn't contribute to the decision but is quite illustrative.

 I found two fundamental flaws with this(this):

 1. For immutable objects, typechecking in the presence of 
 successive modifications of data (first assignment by the 
 compiler, then modification by the user) is very difficult if 
 not impossible. I don't know how to do it. The single 
 initialization model (raw/cooked) used currently in regular 
 immutable constructors works reasonably well and is robust.

 2. For shared objects, the part done by the compiler and the 
 part done by this(this) should be synchronized together. This 
 makes it impossible for the user to e.g. define a struct that 
 gets copied atomically.


See my other reply: but why is it necessary to consider the blit 
logically distinct from the postblit w.r.t to program flow 
observability?

for 1. consider
immutable foo = ...;
immutable bar = foo;
to be
immutable foo = ...;
immutable bar = () {mutable _ = bitcopy(foo); _.__postblit(); 
return _;}();

for 2. you would have to synchronize anyway for shared, it makes 
no difference.


 There'd be an additional issue - this(this) is non-templated, 
 which requires combinatorial additions when qualifiers are 
 present on the source or destination side.


(Perhaps this is what you're referring to, but all you have said 
so far is "this doesn't work and we need to fix it") the post 
blit is surely like a destructor: there's only one way to do it, 
irrespective of the attributes, especially of the intermediate is 
considered mutable until the end of post blit, like static module 
constructors initialising global immutables.


 Please note that fixing one or two of these issues doesn't make 
 this(this) viable - I'm mentioning various issues, each of 
 which is a showstopper. Nevertheless knowing them is necessary 
 so we don't make the same mistake again!


 Andrei


I agree that we should fix any type checking bugs that may be 
present, and that we should strive to not make the same mistake 
twice, however I wouldn't call either of the above cases a 
showstopper. You will need to show more of what is broken and why 
it is broken given the expected breakage.
Apr 01 2018
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/1/18 10:59 AM, Nicholas Wilson wrote:

 On Sunday, 1 April 2018 at 14:31:24 UTC, Andrei Alexandrescu wrote:

 There's a mix of fundamental flaws and bugs. I'll get to the flaws in 
 a second. About the bugs: people have altered their code in various 
 ways to work with the bizarre semantics of this(this). Now, if we fix 
 various bugs in this(this) by virtually redefining it, then we'll 
 break a lot of code in a lot of ways. To wit, we fixed a small issue 
 and it already created problems: 
 https://github.com/dlang/dmd/pull/8032. That didn't contribute to the 
 decision but is quite illustrative.

 I found two fundamental flaws with this(this):

 1. For immutable objects, typechecking in the presence of successive 
 modifications of data (first assignment by the compiler, then 
 modification by the user) is very difficult if not impossible. I don't 
 know how to do it. The single initialization model (raw/cooked) used 
 currently in regular immutable constructors works reasonably well and 
 is robust.

 2. For shared objects, the part done by the compiler and the part done 
 by this(this) should be synchronized together. This makes it 
 impossible for the user to e.g. define a struct that gets copied 
 atomically.


 
 See my other reply: but why is it necessary to consider the blit 
 logically distinct from the postblit w.r.t to program flow observability?
 
 for 1. consider
 immutable foo = ...;
 immutable bar = foo;
 to be
 immutable foo = ...;
 immutable bar = () {mutable _ = bitcopy(foo); _.__postblit(); return _;}();


Negative. The problem is typechecking postblit itself, not its invocation.


 for 2. you would have to synchronize anyway for shared, it makes no 
 difference.


Negative. Consider:

shared struct Point
{
     private long x, y, z;
     private Mutex mutex;
     ...
}

Task: define the copy primitive of Point so atomically copy x, y, and z 
using mutex. The problem is, the compiler will memcpy the three longs 
non-atomically before the user even gets a crack at intercepting the 
operation.


 There'd be an additional issue - this(this) is non-templated, which 
 requires combinatorial additions when qualifiers are present on the 
 source or destination side.


 
 (Perhaps this is what you're referring to, but all you have said so far 
 is "this doesn't work and we need to fix it") the post blit is surely 
 like a destructor: there's only one way to do it, irrespective of the 
 attributes, especially of the intermediate is considered mutable until 
 the end of post blit, like static module constructors initialising 
 global immutables.


Negative. Ignoring qualifiers during copying opens holes in the type 
system the size of China. Or at least Australia as it were :o). Consider:

int[] sneaky;
struct A
{
     private int[] innocent;
     this(this)
     {
         sneaky = innocent;
     }
}
void main()
{
     immutable a = A([1, 2, 3]);
     auto b = a;
     sneaky[1] = 42; // oops
     import std.stdio;
     writeln(a.innocent); // ooooops
}

Sadly this (and many similar ones) compiles and runs warning-free on 
today's compiler. We really need to close this loop, like, five years ago.


 I agree that we should fix any type checking bugs that may be present, 
 and that we should strive to not make the same mistake twice, however I 
 wouldn't call either of the above cases a showstopper. You will need to 
 show more of what is broken and why it is broken given the expected 
 breakage.


Such discussions will be indeed present in the DIP.


Andrei
Apr 01 2018
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 1 April 2018 at 17:08:37 UTC, Andrei Alexandrescu 
wrote:

 On 4/1/18 10:59 AM, Nicholas Wilson wrote:

 for 1. consider
 immutable foo = ...;
 immutable bar = foo;
 to be
 immutable foo = ...;
 immutable bar = () {mutable _ = bitcopy(foo); _.__postblit(); 
 return _;}();


 Negative. The problem is typechecking postblit itself, not its 
 invocation.


Nit sure that I follow but, see comment below on your immutable 
example.


 for 2. you would have to synchronize anyway for shared, it 
 makes no difference.


 Negative. Consider:

 shared struct Point
 {
     private long x, y, z;
     private Mutex mutex;
     ...
 }

 Task: define the copy primitive of Point so atomically copy x, 
 y, and z using mutex. The problem is, the compiler will memcpy 
 the three longs non-atomically before the user even gets a 
 crack at intercepting the operation.


What I meant was:
Point p = ...;
auto pp = p; // bit copy + postblit = WRONG
Point pp;
synchronized (p.mutex) { pp = p; } // synchronised: bitcopy is 
inseparable from the postblit

Perhaps there is a way to make that automatic / nicer to use?


 [...]
 int[] sneaky;
 struct A
 {
     private int[] innocent;
     this(this)
     {
         sneaky = innocent;
     }
 }
 void main()
 {
     immutable a = A([1, 2, 3]);
     auto b = a;
     sneaky[1] = 42; // oops
     import std.stdio;
     writeln(a.innocent); // ooooops
 }

 Sadly this (and many similar ones) compiles and runs 
 warning-free on today's compiler. We really need to close this 
 loop, like, five years ago.


How much of this class of bug would be eliminated by requiring 
that `this(this)` be pure for assignment to const and immutable 
objects? Arguably this(this) should always be pure in any sane 
program. The only reason I can think of is if you're trying to 
perf the number of copies you're making, but there is compiler 
help for that.
Apr 01 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, April 02, 2018 00:25:52 Nicholas Wilson via Digitalmars-d wrote:

 On Sunday, 1 April 2018 at 17:08:37 UTC, Andrei Alexandrescu
 wrote:

 On 4/1/18 10:59 AM, Nicholas Wilson wrote:
 [...]
 int[] sneaky;
 struct A
 {

     private int[] innocent;
     this(this)
     {

         sneaky = innocent;

     }

 }
 void main()
 {

     immutable a = A([1, 2, 3]);
     auto b = a;
     sneaky[1] = 42; // oops
     import std.stdio;
     writeln(a.innocent); // ooooops

 }

 Sadly this (and many similar ones) compiles and runs
 warning-free on today's compiler. We really need to close this
 loop, like, five years ago.


 How much of this class of bug would be eliminated by requiring
 that `this(this)` be pure for assignment to const and immutable
 objects? Arguably this(this) should always be pure in any sane
 program. The only reason I can think of is if you're trying to
 perf the number of copies you're making, but there is compiler
 help for that.


All kinds of things could be done with a postlbit costructor that don't
actually involve copying. These include logging or printing something, and
they could include stuff like reference counting, which may or may not need
access to something external. debug statements would solve some uses cases
but not all. Requiring that this(this) be pure has some of the same issues
that requiring opEquals to be pure or const or whatever has. It makes sense
in _most_ cases, but occasionally, there are good reasons for it not to be -
especially in a systems language.

We have to be _very_ careful about requiring any particular attribute much
of anywhere. Doing so typically causes problems - e.g. those we have with
Object's opEquals, opCmp, toHash, and toString. That decision really needs
to be left up to a particular code base. It's also a big part of why
templates infer attributes. Someone can write their code in such a way that
their code base requires a particular attribute, but in general, language
features shouldn't be requiring any specific attributes, or we'll just be
backing ourselves into another corner.

- Jonathan M Davis
Apr 01 2018
parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 2 April 2018 at 00:44:14 UTC, Jonathan M Davis wrote:

 On Monday, April 02, 2018 00:25:52 Nicholas Wilson via 
 Digitalmars-d wrote:

 On Sunday, 1 April 2018 at 17:08:37 UTC, Andrei Alexandrescu 
 wrote:

 On 4/1/18 10:59 AM, Nicholas Wilson wrote:
 [...]
 int[] sneaky;
 struct A
 {

     private int[] innocent;
     this(this)
     {

         sneaky = innocent;

     }

 }
 void main()
 {

     immutable a = A([1, 2, 3]);
     auto b = a;
     sneaky[1] = 42; // oops
     import std.stdio;
     writeln(a.innocent); // ooooops

 }

 Sadly this (and many similar ones) compiles and runs 
 warning-free on today's compiler. We really need to close 
 this loop, like, five years ago.


 How much of this class of bug would be eliminated by requiring 
 that `this(this)` be pure for assignment to const and 
 immutable objects? Arguably this(this) should always be pure 
 in any sane program. The only reason I can think of is if 
 you're trying to perf the number of copies you're making, but 
 there is compiler help for that.


 All kinds of things could be done with a postlbit costructor 
 that don't actually involve copying. These include logging or 
 printing something, and they could include stuff like reference 
 counting, which may or may not need access to something 
 external. debug statements would solve some uses cases but not 
 all. Requiring that this(this) be pure has some of the same 
 issues that requiring opEquals to be pure or const or whatever 
 has. It makes sense in _most_ cases, but occasionally, there 
 are good reasons for it not to be - especially in a systems 
 language.


I wasn't suggesting this a global requirement of all this(this)s,
only to the _postblit assignment to const and immutable objects_ 
( where
being pure would to disallow the above bug). Note that this should
still be able to be worked around by cast()s (which are un safe) 
and therefore require
 trusted to work in  safe code, in which case the programmer has 
presumably
thought about the situation and knows what he's doing.
Apr 02 2018
prev sibling next sibling parent ag0aep6g <anonymous example.com> writes:
On Sunday, 1 April 2018 at 14:31:24 UTC, Andrei Alexandrescu 
wrote:

 Now, if we fix various bugs in this(this) by virtually 
 redefining it, then we'll break a lot of code in a lot of ways. 
 To wit, we fixed a small issue and it already created problems: 
 https://github.com/dlang/dmd/pull/8032. That didn't contribute 
 to the decision but is quite illustrative.


Yeah, I absolutely see value in starting fresh with different 
syntax, even if you were just implementing the same postblit idea 
again.


 I found two fundamental flaws with this(this):

 1. For immutable objects, typechecking in the presence of 
 successive modifications of data (first assignment by the 
 compiler, then modification by the user) is very difficult if 
 not impossible. I don't know how to do it. The single 
 initialization model (raw/cooked) used currently in regular 
 immutable constructors works reasonably well and is robust.


I'd think that just letting the const/immutable postblit function 
see head-mutable fields would work.

But maybe that's way harder to implement than writing it down the 
forum. If that's so, then fair enough. I know that I won't be 
able to implement it.


 2. For shared objects, the part done by the compiler and the 
 part done by this(this) should be synchronized together. This 
 makes it impossible for the user to e.g. define a struct that 
 gets copied atomically.


Interesting. I've got no armchair expertise on this one.


 There'd be an additional issue - this(this) is non-templated, 
 which requires combinatorial additions when qualifiers are 
 present on the source or destination side.


I think I don't understand this one. Could you give an example in 
code?

Are you saying that we'd need to define all these:

     this(this)
     this(this) const
     this(this) immutable

even if they do the same thing? Wouldn't `this(this) inout` take 
care of this?
Apr 01 2018
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
On Sunday, 1 April 2018 at 14:31:24 UTC, Andrei Alexandrescu 
wrote:

 1. For immutable objects, typechecking in the presence of 
 successive modifications of data (first assignment by the 
 compiler, then modification by the user) is very difficult if 
 not impossible. I don't know how to do it. The single 
 initialization model (raw/cooked) used currently in regular 
 immutable constructors works reasonably well and is robust.


Do the same as in const constructor.


 2. For shared objects, the part done by the compiler and the 
 part done by this(this) should be synchronized together. This 
 makes it impossible for the user to e.g. define a struct that 
 gets copied atomically.


If you mean reference counting, the easy solution with copy 
constructor is bad performance wise. Good solution can be done 
architecturally now and doesn't rely on copy constructor.
Apr 02 2018
parent reply ag0aep6g <anonymous example.com> writes:
On Monday, 2 April 2018 at 14:01:22 UTC, Kagamin wrote:

 On Sunday, 1 April 2018 at 14:31:24 UTC, Andrei Alexandrescu 
 wrote:

 1. For immutable objects, typechecking in the presence of 
 successive modifications of data (first assignment by the 
 compiler, then modification by the user) is very difficult if 
 not impossible. I don't know how to do it. The single 
 initialization model (raw/cooked) used currently in regular 
 immutable constructors works reasonably well and is robust.


 Do the same as in const constructor.


The way it works in a const constructor is that `this.foo = bar;` 
is considered initialization, not assignment.

In a postblit function, we can't say it's initialization, because 
the field already has a value that can't be ignored.
Apr 02 2018
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/02/2018 10:42 AM, ag0aep6g wrote:

 On Monday, 2 April 2018 at 14:01:22 UTC, Kagamin wrote:

 On Sunday, 1 April 2018 at 14:31:24 UTC, Andrei Alexandrescu wrote:

 1. For immutable objects, typechecking in the presence of successive 
 modifications of data (first assignment by the compiler, then 
 modification by the user) is very difficult if not impossible. I 
 don't know how to do it. The single initialization model (raw/cooked) 
 used currently in regular immutable constructors works reasonably 
 well and is robust.


 Do the same as in const constructor.


 
 The way it works in a const constructor is that `this.foo = bar;` is 
 considered initialization, not assignment.


Affirmative. First assignment is a call to the member's constructor. We 
typecheck that reasonably well already in qualified constructors, and 
it's the most promising approach for the DIP.


 In a postblit function, we can't say it's initialization, because the 
 field already has a value that can't be ignored.


Affirmative. That's what makes it so darn difficult to typecheck. If we 
don't let the compiler do the initial blitting (and instead start with 
T.init), the copy ctor is typechecked exactly like the regular ctor.


Andrei
Apr 02 2018
prev sibling parent reply Kagamin <spam here.lot> writes:
On Monday, 2 April 2018 at 14:42:17 UTC, ag0aep6g wrote:

 The way it works in a const constructor is that `this.foo = 
 bar;` is considered initialization, not assignment.


Do you mean the spec? Andrei complained about implementation. 
Const constructors are already implemented as needed for postblit.


 In a postblit function, we can't say it's initialization, 
 because the field already has a value that can't be ignored.


Fields are initialized before const constructor too.
Apr 03 2018
parent reply ag0aep6g <anonymous example.com> writes:
On 04/03/2018 09:39 AM, Kagamin wrote:

 On Monday, 2 April 2018 at 14:42:17 UTC, ag0aep6g wrote:

 The way it works in a const constructor is that `this.foo = bar;` is 
 considered initialization, not assignment.


 
 Do you mean the spec? Andrei complained about implementation.


Andrei complained about both, no? His words: "Too many bugs in design 
and implementation." Design = spec.

I'm also talking about the implementation. `this.foo = bar;` in a 
constructor is at least not normal assignment. If foo has opAssign, that 
won't be called.


 Const constructors are already implemented as needed for postblit.


Maybe. But we can't explain the special assignment semantics with it 
being initialization.

Or can we?

For constructors, we say that the first assignment is actually 
initialization. The compiler might or might not put the .init value down 
before calling the constructor. Doesn't matter, because the constructor 
will overwrite it anyway, and nothing of value is lost.

We can do the same with the postblit function: First assignment is 
actually initialization. When the compiler sees that the postblit 
function initializes a field, it can skip that field when blitting. But 
it can also just blit the whole struct, because it doesn't matter if the 
value just gets overwritten.

In other words, a postblit function can either:

1) use the blitted value as a starting point, like a constructor can use 
the .init value, or it can
2) initialize the field itself.

Would make perfect sense to me.
Apr 03 2018
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/03/2018 07:36 AM, ag0aep6g wrote:

 For constructors, we say that the first assignment is actually 
 initialization. The compiler might or might not put the .init value down 
 before calling the constructor. Doesn't matter, because the constructor 
 will overwrite it anyway, and nothing of value is lost.


What happens in fact is you are guaranteed the .init value is there. 
Much later, well after semantic checking, the backend optimizer removes 
dead assignments on primitive data.


 We can do the same with the postblit function: First assignment is 
 actually initialization. When the compiler sees that the postblit 
 function initializes a field, it can skip that field when blitting.


What if the user code reads the value?

* Often people use this(this) to bump a reference count a la "if (pcnt) 
++*pcnt;"

* People may pass the field by reference to an opaque function. What 
type does the field have?


 But 
 it can also just blit the whole struct, because it doesn't matter if the 
 value just gets overwritten.
 
 In other words, a postblit function can either:
 
 1) use the blitted value as a starting point, like a constructor can use 
 the .init value, or it can
 2) initialize the field itself.
 
 Would make perfect sense to me.


In case (1) things can get quite confusing. Inside a postblit,

field = TypeOfField(100);

is a call to the constructor, whereas

field = TypeOfField(field.x + 100);

is a call to the assignment operator.


Andrei
Apr 03 2018
parent reply ag0aep6g <anonymous example.com> writes:
On Tuesday, 3 April 2018 at 12:52:00 UTC, Andrei Alexandrescu 
wrote:

 On 04/03/2018 07:36 AM, ag0aep6g wrote:

 For constructors, we say that the first assignment is actually 
 initialization. The compiler might or might not put the .init 
 value down before calling the constructor. Doesn't matter, 
 because the constructor will overwrite it anyway, and nothing 
 of value is lost.


 What happens in fact is you are guaranteed the .init value is 
 there. Much later, well after semantic checking, the backend 
 optimizer removes dead assignments on primitive data.


So constructors, including const/immutable ones, basically work 
the same as postblit already? You get an object pre-filled with 
some values, and then you can "initialize" the fields some more 
if you want.

[...]

 What if the user code reads the value?

 * Often people use this(this) to bump a reference count a la 
 "if (pcnt) ++*pcnt;"


Because of the indirection, that can only be done in a mutable 
`this(this)`. Otherwise you violate the const/immutable guarantee 
of the original object.


 * People may pass the field by reference to an opaque function. 
 What type does the field have?


Fully const. Same as in a constructor.

[...]

 In case (1) things can get quite confusing. Inside a postblit,

 field = TypeOfField(100);

 is a call to the constructor, whereas

 field = TypeOfField(field.x + 100);

 is a call to the assignment operator.


A const constructor currently accepts both of those. So the 
second one can apparently be considered "initialization" as well? 
Or should a const constructor not be allowed to do that?
Apr 03 2018
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/3/18 10:21 AM, ag0aep6g wrote:

 On Tuesday, 3 April 2018 at 12:52:00 UTC, Andrei Alexandrescu wrote:

 On 04/03/2018 07:36 AM, ag0aep6g wrote:

 For constructors, we say that the first assignment is actually 
 initialization. The compiler might or might not put the .init value 
 down before calling the constructor. Doesn't matter, because the 
 constructor will overwrite it anyway, and nothing of value is lost.


 What happens in fact is you are guaranteed the .init value is there. 
 Much later, well after semantic checking, the backend optimizer 
 removes dead assignments on primitive data.


 
 So constructors, including const/immutable ones, basically work the same 
 as postblit already? You get an object pre-filled with some values, and 
 then you can "initialize" the fields some more if you want.


Unfortunately, I found out that it's not just "pre-filled with some 
values". Member postblits are run before the containing postblit.

https://run.dlang.io/is/mt6eGa

So this means, the data that is available to the postblit has already 
been processed.

It would only make sense to allow const postblits to have the same 
constructor mechanism if the members all had no postblits.

-Steve
Apr 03 2018
parent reply ag0aep6g <anonymous example.com> writes:
On 04/03/2018 05:13 PM, Steven Schveighoffer wrote:

 Unfortunately, I found out that it's not just "pre-filled with some 
 values". Member postblits are run before the containing postblit.
 
 https://run.dlang.io/is/mt6eGa
 
 So this means, the data that is available to the postblit has already 
 been processed.


There's a similar situation with constructors: A constructor can call 
another constructor, which can lead to double initialization of fields.

Example:

----
class C
{
     int x;
     this() immutable
     {
         this(42); /* Initializes x. */
         x = 13; /* Breaking immutable, or ok? */
     }
     this(int x) immutable
     {
         this.x = x;
     }
}
----

If there's a problem with running two postblits on the same field, then 
I think constructors probably have similar issue. I'm having a hard time 
finding a good example, though. One where we could break immutable in an 
obvious way or some such.


 It would only make sense to allow const postblits to have the same 
 constructor mechanism if the members all had no postblits.


Less drastically, we could also disallow writing to those fields that 
have their own postblit.

The analog with constructors would be disallowing writing to fields that 
have already been initialized by an implicit `super` call. Actually, 
that seems to be how it works, kinda-sorta:

----
class C
{
     int x;
}

class D : C
{
     int y;
     this() immutable
     {
         y = 1; /* accepted */
         x = 2; /* Error: cannot modify this.x in immutable function */
     }
}
----

Obviously, DMD doesn't check what C's constructor actually does (because 
it generally can't check that). It just considers initializing x to be 
C's responsibility.
Apr 03 2018
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/3/18 4:26 PM, ag0aep6g wrote:

 On 04/03/2018 05:13 PM, Steven Schveighoffer wrote:

 Unfortunately, I found out that it's not just "pre-filled with some 
 values". Member postblits are run before the containing postblit.

 https://run.dlang.io/is/mt6eGa

 So this means, the data that is available to the postblit has already 
 been processed.


 
 There's a similar situation with constructors: A constructor can call 
 another constructor, which can lead to double initialization of fields.
 
 Example:
 
 ----
 class C
 {
      int x;
      this() immutable
      {
          this(42); /* Initializes x. */
          x = 13; /* Breaking immutable, or ok? */


IMO, breaking immutable.


      }
      this(int x) immutable
      {
          this.x = x;
      }
 }
 ----
 
 If there's a problem with running two postblits on the same field, then 
 I think constructors probably have similar issue. I'm having a hard time 
 finding a good example, though. One where we could break immutable in an 
 obvious way or some such.


You may NOT want to run a postblit on the member. If all you are going 
to do, for example, is reassign a variable, then running the postblit, 
and then the destructor, just so you can overwrite it is pointless.

But more importantly, if the postblit of the member does something crazy 
like stores a reference to itself as an immutable elsewhere, and then 
the compiler allows overwriting, we now have problems.

I think the better mechanism for immutable copying would be to have a 
copy constructor that starts off with T.init, and is passed the object 
to copy from. That seems to be a direction Andrei is considering.

-Steve
Apr 03 2018
parent reply ag0aep6g <anonymous example.com> writes:
On 04/03/2018 10:51 PM, Steven Schveighoffer wrote:

 On 4/3/18 4:26 PM, ag0aep6g wrote:


[...]

 If there's a problem with running two postblits on the same field, 
 then I think constructors probably have similar issue. I'm having a 
 hard time finding a good example, though. One where we could break 
 immutable in an obvious way or some such.


 
 You may NOT want to run a postblit on the member. If all you are going 
 to do, for example, is reassign a variable, then running the postblit, 
 and then the destructor, just so you can overwrite it is pointless.


Same with class constructors: You may not want to run `super` when 
you're just going to overwrite what it did. But the language doesn't 
give you a choice. It'll be called one way or another.

I'm not saying that imitating how constructors work will make the best 
possible copying mechanism. Something else might be superior in every 
way. It's just that so far the arguments against a constructor-like 
postblit also seem to apply to constructors as they are implemented.

So I'm thinking that a postblit modeled after constructors could work as 
well as they do. But maybe the real takeaway is that constructors don't 
work very well, and shouldn't be imitated.


 But more importantly, if the postblit of the member does something crazy 
 like stores a reference to itself as an immutable elsewhere, and then 
 the compiler allows overwriting, we now have problems.


I'd love to see an example of this in code. The best I can come up with 
would be something like this (doesn't compile):

----
import std.stdio;

immutable(int)* p;

struct S
{
     int x;
     this(this) immutable
     {
         x = 42; /* First write. */
         .p = &this.x;
         writeln(p, " ", *p); /* Prints some address and 42. */
     }
}

struct T
{
     S s;
     this(this) immutable
     {
         s = S(13); /* Second write. Breaking immutable? */
         writeln(p, " ", *p); /* Same address, but 13. */
     }
}

void main()
{
     immutable T foo;
     immutable bar = foo;
}
----

But that's essentially the same as the class example I posted. `*p` 
would only change values during the postblit run. Just like a 
constructor chain can write to the same field multiple times.

That's kinda iffy, but I can't find a way to demonstrate some real, 
obvious damage.


 I think the better mechanism for immutable copying would be to have a 
 copy constructor that starts off with T.init, and is passed the object 
 to copy from. That seems to be a direction Andrei is considering.


No objection from me. If Andrei et al. can find a better solution, great.
Apr 03 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/3/18 5:44 PM, ag0aep6g wrote:

 On 04/03/2018 10:51 PM, Steven Schveighoffer wrote:

 On 4/3/18 4:26 PM, ag0aep6g wrote:


 [...]

 If there's a problem with running two postblits on the same field, 
 then I think constructors probably have similar issue. I'm having a 
 hard time finding a good example, though. One where we could break 
 immutable in an obvious way or some such.


 You may NOT want to run a postblit on the member. If all you are going 
 to do, for example, is reassign a variable, then running the postblit, 
 and then the destructor, just so you can overwrite it is pointless.


 
 Same with class constructors: You may not want to run `super` when 
 you're just going to overwrite what it did. But the language doesn't 
 give you a choice. It'll be called one way or another.


At least you can invoke the one you want, with postblit there is only 
one "choice".

But this is a red herring -- we already have struct constructors, and 
that requirement of invoking constructors on members is not present.

With structs, we have the possibility of initialization via different 
mechanisms: constructor, postblit, .init. All of these are supported by 
the struct member, but currently you can only invoke postblit if you are 
in a postblit. And only at the beginning. I would like to see more 
flexibility for copying.


 I'm not saying that imitating how constructors work will make the best 
 possible copying mechanism. Something else might be superior in every 
 way. It's just that so far the arguments against a constructor-like 
 postblit also seem to apply to constructors as they are implemented.


For structs, using .init is a valid initialization, so it's completely 
different from classes, where a constructor MUST be invoked. Indeed, 
there is no mechanism to require calling struct member constructors in 
the owner's ctor.

Stop thinking class constructors, and think struct constructors instead.


 But more importantly, if the postblit of the member does something 
 crazy like stores a reference to itself as an immutable elsewhere, and 
 then the compiler allows overwriting, we now have problems.


 
 I'd love to see an example of this in code. The best I can come up with 
 would be something like this (doesn't compile):
 
 ----
 import std.stdio;
 
 immutable(int)* p;
 
 struct S
 {
      int x;
      this(this) immutable
      {
          x = 42; /* First write. */
          .p = &this.x;
          writeln(p, " ", *p); /* Prints some address and 42. */
      }
 }
 
 struct T
 {
      S s;
      this(this) immutable
      {
          s = S(13); /* Second write. Breaking immutable? */


Of course this doesn't compile, because s is considered immutable by 
now. What I was saying is that we can't allow postblit to modify data 
that has already been postblitted, because of the reason this example is 
showing.


          writeln(p, " ", *p); /* Same address, but 13. */
      }
 }
 
 void main()
 {
      immutable T foo;
      immutable bar = foo;
 }
 ----
 
 But that's essentially the same as the class example I posted. `*p` 
 would only change values during the postblit run. Just like a 
 constructor chain can write to the same field multiple times.


I don't think you should be able to write to the same field multiple 
times in an immutable/const constructor. If so, that's a bug.


 That's kinda iffy, but I can't find a way to demonstrate some real, 
 obvious damage.


Any place where an immutable can be observed to change between two reads 
is breaking immutable.

-Steve
Apr 04 2018
next sibling parent ag0aep6g <anonymous example.com> writes:
On 04/04/2018 12:37 PM, Steven Schveighoffer wrote:

 With structs, we have the possibility of initialization via different 
 mechanisms: constructor, postblit, .init. All of these are supported by 
 the struct member, but currently you can only invoke postblit if you are 
 in a postblit. And only at the beginning. I would like to see more 
 flexibility for copying.


[...]

 For structs, using .init is a valid initialization, so it's completely
 different from classes, where a constructor MUST be invoked. Indeed,
 there is no mechanism to require calling struct member constructors in 
 the owner's ctor.


[...]

To paraphrase your point: We should be able to initialize a member with 
.init or by calling its constructor (over .init). We should not be 
forced to initialize it with its postblit.

Makes sense. But I see one tricky case: Can I also choose to use the 
blitted value of the member without calling its postblit? I'd say that 
can't be allowed.

So the compiler would have to identify that case and reject it. And now 
we're in uncharted territory. As far as I see, this is not something 
that constructors do (struct or class). They happily accept any .init 
value. But postblits can't accept any blitted value. So we can't say: 
"Just do what constructors do." We have to come up with new rules.

But we don't want to come up with new rules, we want to say: "Just do 
what constructors do." So we require the member postblit, and we say 
that the outer postblit operates on the resulting value like a 
constructor operates on .init.

And then we see that it breaks the type system, and that any analog 
behavior of constructors just means that constructors/.init are broken, too.

[...]

 struct S
 {
      int x;
      this(this) immutable
      {
          x = 42; /* First write. */




[...]

      }
 }

 struct T
 {
      S s;
      this(this) immutable
      {
          s = S(13); /* Second write. Breaking immutable? */


 
 Of course this doesn't compile, because s is considered immutable by 
 now.


It doesn't compile, because this(this) is a broken mess currently.

The first write doesn't compile either. Obviously, with a properly 
implemented immutable this(this), you would at least be allowed to write 
once.

With mutable(!) `this(this)`s, both writes are accepted.


 What I was saying is that we can't allow postblit to modify data 
 that has already been postblitted, because of the reason this example is 
 showing.


Still, I'd like to have a more explosive example. The breakage here is 
very localized, and could possibly be defined away somehow. Like: 
"Unique immutable data is considered 'raw' in constructors and postblit 
functions. That means, the data can be mutated. Only when the 
constructor/postblit returns does it become 'cooked' and truly immutable."

[...]

 ----




[...]

 I don't think you should be able to write to the same field multiple 
 times in an immutable/const constructor. If so, that's a bug.


Are you counting .init as a write, or is a constructor allowed to build 
on that?

----
import std.stdio;
struct S
{
     int x = 1;
     int y = 3;
     this(int dummy) immutable
     {
         writeln(x); /* 1 */
         ++x; /* Breaking immutable? */
         writeln(x); /* 2 */

         ++y; /* Breaking immutable even though y hasn't been printed 
yet? */
         writeln(y); /* 4 */
     }
}
void main()
{
     auto s = immutable S(0);
}
----

I think those increments could be considered breaking immutable. Maybe 
they must be.

If this were to be outlawed, I'd agree a constructors and postblits are 
fundamentally different.
Apr 04 2018
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Wednesday, 4 April 2018 at 10:37:57 UTC, Steven Schveighoffer 
wrote:

 I would like to see more flexibility for copying.


It's a tradeoff between control and ergonomics.


 import std.stdio;
 
 immutable(int)* p;
 
 struct S
 {
      int x;
      this(this) immutable
      {
          x = 42; /* First write. */
          .p = &this.x;
          writeln(p, " ", *p); /* Prints some address and 42. */
      }
 }
 
 struct T
 {
      S s;
      this(this) immutable
      {
          s = S(13); /* Second write. Breaking immutable? */


 Of course this doesn't compile, because s is considered 
 immutable by now. What I was saying is that we can't allow 
 postblit to modify data that has already been postblitted, 
 because of the reason this example is showing.


Assignment ends lifetime of the previous value, it will be just a 
dangling pointer.

AFAIK this is how reassignment works wrt destructors:

{
   immutable tmp=S(13);
   s.__dtor(); //if any
   s=tmp; //move into immutable storage
}


 I don't think you should be able to write to the same field 
 multiple times in an immutable/const constructor. If so, that's 
 a bug.


Stores to const fields are special, so it should be possible to 
track them in postblits too.
Apr 12 2018
prev sibling parent Kagamin <spam here.lot> writes:
On Wednesday, 4 April 2018 at 10:37:57 UTC, Steven Schveighoffer 
wrote:

 Any place where an immutable can be observed to change between 
 two reads is breaking immutable.


That happens in constructor too.
Apr 12 2018
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/03/2018 04:26 PM, ag0aep6g wrote:

 On 04/03/2018 05:13 PM, Steven Schveighoffer wrote:

 Unfortunately, I found out that it's not just "pre-filled with some 
 values". Member postblits are run before the containing postblit.

 https://run.dlang.io/is/mt6eGa

 So this means, the data that is available to the postblit has already 
 been processed.


 
 There's a similar situation with constructors: A constructor can call 
 another constructor, which can lead to double initialization of fields.
 
 Example:
 
 ----
 class C
 {
      int x;
      this() immutable
      {
          this(42); /* Initializes x. */
          x = 13; /* Breaking immutable, or ok? */
      }
      this(int x) immutable
      {
          this.x = x;
      }
 }
 ----


Let's replace "int" with an UDT:

struct S
{
     int x = -1;
     this(int y) immutable { x = y; }
     void opAssign(int) immutable;
}

class C
{
     S x;
     this() immutable
     {
         this(42); /* Initializes x. */
         x = 13; /* Breaking immutable, or ok? */
     }
     this(int x) immutable
     {
         this.x = x;
     }
}

This code compiles, and calls the constructor twice for the same object. 
Clearly that shouldn't be allowed to pass. I've submitted 
https://issues.dlang.org/show_bug.cgi?id=18719 - thanks! (The problem 
seems to occur even without immutable, it's endemic to forwarding 
constructors.)

Andre
Apr 03 2018
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/03/2018 10:21 AM, ag0aep6g wrote:

 On Tuesday, 3 April 2018 at 12:52:00 UTC, Andrei Alexandrescu wrote:

 On 04/03/2018 07:36 AM, ag0aep6g wrote:

 For constructors, we say that the first assignment is actually 
 initialization. The compiler might or might not put the .init value 
 down before calling the constructor. Doesn't matter, because the 
 constructor will overwrite it anyway, and nothing of value is lost.


 What happens in fact is you are guaranteed the .init value is there. 
 Much later, well after semantic checking, the backend optimizer 
 removes dead assignments on primitive data.


 
 So constructors, including const/immutable ones, basically work the same 
 as postblit already? You get an object pre-filled with some values, and 
 then you can "initialize" the fields some more if you want.


Well... not really. This is because .init is really an inert state - 
null indirections, no state allocated etc. Makes typechecking easy, and 
calling constructor on top of .init is what happens already. In 
contrast, the postblit situash is very different - the fields already 
contain "interesting" data, allocated resources etc. Calling a 
constructor on top of that is not defined.
Apr 03 2018
parent reply ag0aep6g <anonymous example.com> writes:
On 04/03/2018 08:57 PM, Andrei Alexandrescu wrote:

 Well... not really. This is because .init is really an inert state - 
 null indirections, no state allocated etc.


.init can have non-null indirections:

     struct S { int[] a = [1, 2, 3]; }
     static assert(S.init.a.ptr !is null); /* passes */

But maybe it shouldn't. It leads to problems with immutable, because the 
same .init with the same `a` is used for both mutable and immutable 
objects. https://issues.dlang.org/show_bug.cgi?id=10376

As far as I see, a const/immutable postblit working like a constructor 
wouldn't have that particular problem, because the copy is always made 
from another const/immutable object, so there wouldn't be aliasing with 
mutable data.

But there might of course be other problems with indirections, which I 
don't see right now. I'd love to see any examples, though. They probably 
reveal weaknesses in .init/constructors, too.


 Makes typechecking easy, and 
 calling constructor on top of .init is what happens already. In 
 contrast, the postblit situash is very different - the fields already 
 contain "interesting" data, allocated resources etc. Calling a 
 constructor on top of that is not defined.


Well, the idea would be to define it. But if postblit goes away for 
other reasons anyway (like the atomic copy thing, or another mechanism 
being simply superior), then there's no point in pursuing this, of course.
Apr 03 2018
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/03/2018 04:29 PM, ag0aep6g wrote:

 But if postblit goes away for other reasons anyway (like the atomic copy 
 thing, or another mechanism being simply superior), then there's no 
 point in pursuing this, of course.


The DIP will definitely need to make a solid case supporting whatever it 
proposes.
Apr 03 2018
prev sibling parent Kagamin <spam here.lot> writes:
On Tuesday, 3 April 2018 at 11:36:27 UTC, ag0aep6g wrote:

 Maybe. But we can't explain the special assignment semantics 
 with it being initialization.

 Or can we?


Postblit is a part of initialization of a copy. Until postblit 
completes the object is in invalid state, same as in constructor.
Apr 05 2018
prev sibling parent Kagamin <spam here.lot> writes:
On Sunday, 1 April 2018 at 14:31:24 UTC, Andrei Alexandrescu 
wrote:

 There's a mix of fundamental flaws and bugs. I'll get to the 
 flaws in a second. About the bugs: people have altered their 
 code in various ways to work with the bizarre semantics of 
 this(this). Now, if we fix various bugs in this(this) by 
 virtually redefining it, then we'll break a lot of code in a 
 lot of ways. To wit, we fixed a small issue and it already 
 created problems: https://github.com/dlang/dmd/pull/8032. That 
 didn't contribute to the decision but is quite illustrative.


Since poorly thought decisions keep being made for the language 
it's doubtful that "no breakage" policy has real backing.
Apr 02 2018
prev sibling parent reply Shachar Shemesh <shachar weka.io> writes:
On 01/04/18 03:32, H. S. Teoh wrote:

 The one nagging question I've been having about pure is: how much are we
 actually taking advantage of the guarantees provided by pure?


My problem is that pure do not provide many guarantees.


  We have
 developed very clever ways of extending the traditional definition of
 pure and invented creative ways of making more things pure, which is all
 great.


Can anyone explain to me what good are the guarantees provided by a 
function that is pure but not strictly pure? I couldn't find them.


  But AFAIK the only place where it's actually taken advantage of
 is to elide some redundant function calls inside a single expression.


You cannot even do that unless the function is strictly pure. For all 
D's extension of the pure concept, it weakened, rather than enhanced, 
what it means.


 And perhaps infer uniqueness in some cases for implicit casting to
 immutable.


Can you expand on that one?

Shachar
Apr 01 2018
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, April 02, 2018 09:56:19 Shachar Shemesh via Digitalmars-d wrote:

 On 01/04/18 03:32, H. S. Teoh wrote:

 The one nagging question I've been having about pure is: how much are we
 actually taking advantage of the guarantees provided by pure?


 My problem is that pure do not provide many guarantees.


  We have

 developed very clever ways of extending the traditional definition of
 pure and invented creative ways of making more things pure, which is all
 great.


 Can anyone explain to me what good are the guarantees provided by a
 function that is pure but not strictly pure? I couldn't find them.


Honestly, I think at this point pure is easier to understand if you think of
it as  noglobal and don't think about functional purity at all. What pure
does is make it so that the function cannot access global, mutable state
except through its arguments. So, all that it's working with (except for
constants such as enums or immutable, module-level variables) is what it's
given via its arguments. As such, the primary benefit of pure is that you
know that no global state is being mucked with unless it was passed to the
function as an argument. _That_ is the guarantee that pure provides.
Everything else about pure is just built on top of that guarantee and what
the compiler can infer from it.

If the function is "strongly" pure (i.e. its parameters are immutable or
implicitly convertible to immutable) then the compiler knows that if the
function is called multiple times with the exact same arguments, then it
knows that the result will be the same each time (though it does have to
take into account the fact that each call could return a newly allocated
object - they'd just be equivalent objects every time). So, when you're
dealing with a strongly pure function, you're then dealing with actual,
functional purity, and the compiler can choose to optimize calls such as
foo(5) * foo(5) so that foo(5) is only called once.

Weakly pure functions (i.e. pure functions that aren't strongly pure) don't
have those same optimization benefits, because calling them could result in
the arguments being mutated, but because weakly pure functions don't access
global, mutable state, the compiler can safely call them from a strongly
pure function without violating the guarantee that multiple calls with the
same arguments to a strongly pure function are supposed to give the same
result.

Most pure functions are weakly pure (e.g. unless a pure member function is
immutable, then it's weakly pure), so the optimization benefits are pretty
minimal in most cases. And even if a function is strongly pure, the
optimization is only done within an expression (or maybe statement - I can
never remember which), because going beyond that would require data flow
analysis, which the compiler rarely does. As such, pretty much the only time
you end up with function calls being elided thanks to pure is when you have
a strongly pure function where you do something like foo(5) * foo(5). So,
the optimization benefits of pure are pretty minimal. It's that guarantee
about not touching global, mutable state which is the main benefit. As such,
if we were adding pure now, I would strongly argue for calling it something
like  noglobal. I think that it would reduce the confusion considerably. As
it stands, while it helps make functional purity possible in some cases, it
ultimately doesn't have much to do with functional purity in spite of its
name.


 And perhaps infer uniqueness in some cases for implicit casting to
 immutable.


 Can you expand on that one?


int[] foo()
{
    return [1, 2, 3, 4, 5];
}

void main()
{
    immutable arr = foo();
}

does not compile, because the return value is mutable, and you can't
implicitly convert int[] to immutable int[]. However,

int[] foo() pure
{
    return [1, 2, 3, 4, 5];
}

void main()
{
    immutable arr = foo();
}

compiles just fine. Because foo is pure, and the compiler knows that the
return value could not possibly have come from the function's arguments, it
knows that the return value is unique and that it won't violate the type
system to implicitly cast it to immutable. As such, you can write a function
as complicated as you want to create the return value, and so long as the
function is pure, and the compiler can determine that the return value did
not come via an argument, you can implicitly convert the return value to
immutable rather than having to use something like
std.exception.assumeUnique or an explicit cast, which relies on the
programmer verifying that the object being cast is indeed unique rather than
having the compiler guarantee it.

Whether the implicit cast is allowed ultimately depends on the types of the
pure function's parameters and the actual arguments, so it's not always
obvious whether it will work or not, but in general, it works very well for
initializing complex, immutable objects without having to rely on getting
casts right. e.g. this still compiles

int[] foo(int[] a) pure
{
    return [1, 2, 3, 4, 5];
}

void main()
{
    immutable arr = foo([1, 2]);
}

because the compiler can see that the argument being passed is unique and
that therefore the return value is unique whether it came from the argument
or not, whereas this doesn't compile

int[] foo(int[] a) pure
{
    return [1, 2, 3, 4, 5];
}

void main()
{
    int[] a = [1, 2];
    immutable arr = foo(a);
}

because the compiler can't guarantee that the return value isn't a slice of
the argument (at least not without looking at the implementation, but all
the compiler looks at is the signature).

- Jonathan M Davis
Apr 02 2018
parent reply Shachar Shemesh <shachar weka.io> writes:
On 02/04/18 10:45, Jonathan M Davis wrote:

 Honestly, I think at this point pure is easier to understand if you think of
 it as  noglobal and don't think about functional purity at all.


That's fine. My point was that the only optimizations possible are 
possible on strictly pure functions (the immutable cast one included). 
Weakly pure functions add nothing.

But merely having them around means that when I annotate a function with 
"pure", I do not promise any guarantees that the compiler can actually 
use to perform optimizations.

Shachar
Apr 02 2018
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, April 02, 2018 11:04:08 Shachar Shemesh via Digitalmars-d wrote:

 On 02/04/18 10:45, Jonathan M Davis wrote:

 Honestly, I think at this point pure is easier to understand if you
 think of it as  noglobal and don't think about functional purity at
 all.


 That's fine. My point was that the only optimizations possible are
 possible on strictly pure functions (the immutable cast one included).
 Weakly pure functions add nothing.

 But merely having them around means that when I annotate a function with
 "pure", I do not promise any guarantees that the compiler can actually
 use to perform optimizations.


It means that a strongly pure function could call the function, which is the
entire reason that the definition of pure was widened to simply mean that it
guarantees that the function doesn't access global, mutable state instead of
also including the requirements about parameters which are placed on
strongly pure functions. So, a weakly pure function _can_ help with
optimizations in that it helps to implement strongly pure functions, and
without weakly pure functions, what you can do with strongly pure functions
can be very limited, making weakly pure functions very important even if all
you care about is optimizations, but no, a call to a weakly pure function
cannot be elided based on the fact that it's weakly pure. Now, pure combined
with const could provide some optimizations in rare cases, since the
compiler can guarantee that a pure function doesn't mutate a const argument
via another reference if no such mutable reference could be accessed through
one of the arguments, but I doubt that such optimizations are done at this
point, and it wouldn't involve eliding function calls. But in theory, the
fact that the compiler knows that a function can't access anything except
through its arguments could allow the compiler to optimize some code, even
if it doesn't involve eliding function calls.

Ultimately, I think that it's a mistake to think about pure having much to
do with optimizations. Much as such optimizations do exist, they're just too
limited. The primary advantage is that when you see that a function is pure,
you know that the function is just using what it's given via its arguments,
just like when you see a variable is immutable, you know that it can't be
mutated. Any optimizations that can be gotten via pure are therefore mostly
just gravy.

So, if your only motivation in dealing with pure is optimizations, then
there's a good chance that it really isn't worth your time, but personally,
I think that it's quite valuably simply for guaranteeing the that function
doesn't access global, mutable state.

- Jonathan M Davis
Apr 02 2018
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/2/18 4:04 AM, Shachar Shemesh wrote:

 On 02/04/18 10:45, Jonathan M Davis wrote:

 Honestly, I think at this point pure is easier to understand if you 
 think of
 it as  noglobal and don't think about functional purity at all.


 
 That's fine. My point was that the only optimizations possible are 
 possible on strictly pure functions (the immutable cast one included). 
 Weakly pure functions add nothing.
 
 But merely having them around means that when I annotate a function with 
 "pure", I do not promise any guarantees that the compiler can actually 
 use to perform optimizations.
 
 Shachar


This is a good article motivating the relaxed purity model we have: 
http://klickverbot.at/blog/2012/05/purity-in-d/
Apr 02 2018
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 02.04.2018 10:04, Shachar Shemesh wrote:

 On 02/04/18 10:45, Jonathan M Davis wrote:

 Honestly, I think at this point pure is easier to understand if you 
 think of
 it as  noglobal and don't think about functional purity at all.


 
 That's fine. My point was that the only optimizations possible are 
 possible on strictly pure functions (the immutable cast one included). 
 Weakly pure functions add nothing.
 ...


That point is wrong. It would be wrong even if your next point was true.


 But merely having them around means that when I annotate a function with 
 "pure", I do not promise any guarantees that the compiler can actually 
 use to perform optimizations.
 
 Shachar


Yes, actually you do.

For example, if you have:

void foo(int[] x)pure;

void main(){
     auto x = [1];
     auto y = x.dup;
     foo(x);
     foo(y);
}

This can be transformed into:

void main(){
     auto x = [1];
     foo(x);
     auto y = x.dup;
}

Pure functions have deterministic side-effects.
Apr 02 2018
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 02.04.2018 08:56, Shachar Shemesh wrote:

 On 01/04/18 03:32, H. S. Teoh wrote:

 The one nagging question I've been having about pure is: how much are we
 actually taking advantage of the guarantees provided by pure?


 
 My problem is that pure do not provide many guarantees.
 ...


It guarantees that no global state is accessed.


  We have
 developed very clever ways of extending the traditional definition of
 pure and invented creative ways of making more things pure, which is all
 great.


 
 Can anyone explain to me what good are the guarantees provided by a 
 function that is pure but not strictly pure? I couldn't find them.
 ...


You can use weakly pure functions to compose strongly pure functions.


  But AFAIK the only place where it's actually taken advantage of
 is to elide some redundant function calls inside a single expression.


 
 You cannot even do that unless the function is strictly pure. For all 
 D's extension of the pure concept, it weakened, rather than enhanced, 
 what it means.
 ...


There is no such weakening.


 And perhaps infer uniqueness in some cases for implicit casting to
 immutable.


 
 Can you expand on that one?
 
 Shachar
Apr 02 2018
prev sibling next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, March 31, 2018 17:32:10 H. S. Teoh via Digitalmars-d wrote:

 On Sat, Mar 31, 2018 at 07:38:06PM -0400, Andrei Alexandrescu via
 Digitalmars-d wrote: [...]


 Once we have that, we can encapsulate desirable abstractions (such as
  nogc safe collections that work in pure code), regardless of how
 difficult their implementations might be. It seems that currently
 this(this) does not allow us to do that.


 What exactly is it about this(this) that blocks us from doing that?

 Removing this(this) is going to be a huge breaking change far bigger
 than, say, removing autodecoding ever will be.


Well, as far const goes, you have the inherent problem that the object you
have in this(this) has already been initialized, and you can't mutate const.
I think that other attribute issues are mostly implementation issues.

And as far as auto-decoding goes, removing postblit constructors would
actually be far easier, because it's simply a matter of deprecating a
particular function. Some traits that check for postblits would have to be
adjusted, but all those really care about is that there's extra code beyond
the default copying code, and they could easily be made to work with both
postblits and whatever the new solution is at the same time until postblits
are actually gone.

Auto-decoding on the other hand, affects far more than just the functions in
std.range.primitives, and there is no clear deprecation path. So, while it
_might_ be true that deprecating postblits would break more code (and I'm
honestly not convinced that it would), postblit constructors would actually
have a clean deprecation path. So, while it breaks code, it does so in a way
that can easily be managed, whereas the removal of auto-decoding is not
straightforward at all. It could be done by just flipping the switch
so-to-speak, but AFAIK, no one has yet presented a workable deprecation
path. And ultimately, I think that _that_ is what's preventing us from
fixing auto-decoding. If a clean deprecation path were found, then we could
discuss whether the resulting breakage would be worth it, but without a
clean deprecation path, I don't see how we could ever do it.

Also, unless we can fix postblit constructors (and the evidence thus far is
that if we can, it's way too complicated - e.g. Kenji's attempt several
years ago was rejected because it was way too complicated), if we don't
switch to a differnt solution, we're talking about permanently not
supporting copying const types that require user-defined copying.


 A lot of us here have essentially given up on const except for a few
 very narrow cases.  The transitive nature of const makes it extremely
 difficult to work with in the general case, even though the simplest use
 cases are workable.

 One of the biggest stumbling blocks is that whenever ranges are
 involved, const is practically out of the question, because even though
 it can be made to work for most cases, there will almost always be that
 one pathological case where it's impossible / too hard to work around,
 and that ruins it for everything else, so that it's much simpler to just
 avoid it altogether.


I don't think that it's even the case that it can be made to work in most
cases - or if it can, it involves a lot of static ifs. The range API does
not require const for _anything_ (and really can't due to how restricted
const is), so _no_ generic range-based code can assume that even something
like length or empty can be called if the range is const or inout. As such,
the only ranges that can mark anything with const are ones that either
aren't generic or which have a bunch of static ifs presenting const and
non-const versions depending on the template arguments, and IMHO, that's
just not workable. I've done it before, and it's a mess. As such, ranges and
const really don't work together at all. const really only works when you're
dealing with a very constrainted set of types where you can actually
guarantee that they work with const.


 * pure is difficult


 [...]

 The one nagging question I've been having about pure is: how much are we
 actually taking advantage of the guarantees provided by pure?  We have
 developed very clever ways of extending the traditional definition of
 pure and invented creative ways of making more things pure, which is all
 great.  But AFAIK the only place where it's actually taken advantage of
 is to elide some redundant function calls inside a single expression.
 And perhaps infer uniqueness in some cases for implicit casting to
 immutable.

 While these are indisputably useful, they seem so far to be only
 relatively minor benefits.  If pure is indeed so difficult to support
 generically, it begs the question, is it worth the effort just to gain
 these niggling benefits?  Whatever happened to larger-scale benefits
 conferred by purity?


Honestly, I think that the main benefit of pure is that if you know that a
function is pure, you know that it doesn't access any global, mutable state
except through its arguments. The secondary benefit is that it can allow for
functions which construct immutable objects using mutable state and without
casts (though such functions are usually small in scale and don't require
that large portions of the code base be pure). I think that the idea that
pure is going to result in compiler optimizations is mostly a joke. Not only
can it only work with strongly pure functions (which most pure functions
aren't and can't be), but the way it's currently implemented, it can only
elide calls within a single statement (and it might actually be a single
expression - I'm not sure which). Going farther than that requires code-flow
analysis, which Walter is almost always against, so it's almost certainly
not happening. And even if it did, I don't think that it would help much.
How often do you call the same function with the same arguments within a
single function body? I expect that that's pretty rare. The place that it
would likely be of the most benefit would be math code, and even there, I'm
not sure that it happens much.

So, I think that the only large-scale benefit thet exists for pure and
really can exist for pure is the fact that you know that the function
doesn't access global, mutable state. Everything else it does is just gravy
and too limited to be a "large-scale" benefit. Certainly, optimizations are
clearly _not_ the main benefit of pure, since they almost don't exist. But
over time, we have managed to add more gravy here and there as we've figured
out assumptions that can be made based on pure (like the case where we can
convert the result of a pure function to immutable).

- Jonathan M Davis
Mar 31 2018
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/18 9:01 PM, Jonathan M Davis wrote:

 And as far as auto-decoding goes


Let's keep this about construction and not about auto-decoding. Thanks.
Mar 31 2018
prev sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Sunday, 1 April 2018 at 01:01:24 UTC, Jonathan M Davis wrote:

 So, I think that the only large-scale benefit thet exists for 
 pure and really can exist for pure is the fact that you know 
 that the function doesn't access global, mutable state. 
 Everything else it does is just gravy and too limited to be a 
 "large-scale" benefit. Certainly, optimizations are clearly 
 _not_ the main benefit of pure, since they almost don't exist. 
 But over time, we have managed to add more gravy here and there 
 as we've figured out assumptions that can be made based on pure 
 (like the case where we can convert the result of a pure 
 function to immutable).

 - Jonathan M Davis


Great write-up.

Why keep pure when the benefits obviously don't outweight the 
costs?
Here is what I interpret reading this: "lost productivity".
Apr 01 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, April 01, 2018 11:37:21 Guillaume Piolat via Digitalmars-d wrote:

 On Sunday, 1 April 2018 at 01:01:24 UTC, Jonathan M Davis wrote:

 So, I think that the only large-scale benefit thet exists for
 pure and really can exist for pure is the fact that you know
 that the function doesn't access global, mutable state.
 Everything else it does is just gravy and too limited to be a
 "large-scale" benefit. Certainly, optimizations are clearly
 _not_ the main benefit of pure, since they almost don't exist.
 But over time, we have managed to add more gravy here and there
 as we've figured out assumptions that can be made based on pure
 (like the case where we can convert the result of a pure
 function to immutable).

 - Jonathan M Davis


 Great write-up.

 Why keep pure when the benefits obviously don't outweight the
 costs?
 Here is what I interpret reading this: "lost productivity".


Personally, I don't think that the costs outweigh the benefits. Yes, if your
primary goal is optimizations, then pure falls flat on its face, but
personally, I think that the fact that pure allows you to prove that a
function doesn't access any global, mutable state except through its
arguments is quite valuable. There are still some things in druntime and
Phobos which don't work with pure like they should, but in general, I
haven't found that it's really a big problem. Some code can't be pure, and
that's life, but a _lot_ can be. If anything, I find that const is the
attribute that causes problems, not pure.

But if you don't want to use pure, then don't use pure. Nothing is forcing
you to use it, and if someone else insists on making their code work with
pure, it's only a problem for you if they're trying to call your code, and
they complain about the fact that your code doesn't work with pure.
Certainly, code that isn't pure has no problems calling code that is, so if
other folks go to the effort of making their code work with pure, it won't
cause you problems.

- Jonathan M Davis
Apr 01 2018
prev sibling next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 01/04/2018 11:38 AM, Andrei Alexandrescu wrote:

 We need to have a simple recipe on how to define a canonical object. 
 That would include requirements such as:
 
 * should work with mutable, const, immutable, and shared
 * the right way to define constructor(s)
 * the right way to define copying
 * the right way to define destructor(s)


We will also need to make this consistent for classes.

				***

So lets go back to the basics.

 From the primordial goo that is My Own Little Language (MOLL heavily WIP):

__userType [Identifier:name] {
	(Declaration|Definition)*
}

Is a value type, acts pretty much like struct.
Why do I not call it a struct? Because it shouldn't be used in user 
code. This exists only in mental models for D users currently.

So something like:

__userType Foo {
	int x;
	
	void __constructor(int x) {
		this.x = x;
	}

	void __postblit() {
		this.x = x+1;
	}

	void __deconstructor() {
		// free?
	}
}

Is comparable to D code of:

struct Foo {
	int x;

	this(this) {
		this.x = x + 1;
	}

	~this() {
		// free?
	}
}

So what does a class look like?

__userType Foo {
	enum __IS_CLASS = true;
	static TypeInfo_Class TypeInfo = TypeInfo_Class(...);

	static void*[][] __vtables = [
		[&TypeInfo, ...],
		[&TypeInfo, ...]
	];

	void*[] __vtable;
	void* __ptr;

	auto __cast(T)() if(T.__IS_CLASS) {
		// ...
	}
}

Why is this important? Because it shows that our current lifetime 
management strategies for ref counting with structs can be applied 
directly to classes.

					***

At this point in time, lets look at how each qualifier is perceived:

mutable: default, good, everything else is a pain
const: library code and the author doesn't want you to go touchy
immutable: in binary only, quite yucky
shared: a message from the library author and not much else

I do not believe we can continue this conversation with the above 
perceptions.

First and foremost qualifiers are there to tell other developers (and 
yourself in the future) what you intend on doing with a given bit of 
memory. Now we do need a way to express head-const. Because it allows us 
to say that we personally can't modify a bit of memory but we can call a 
method that does (from it).

Good engineering requires us to consider psychology on these issues. To 
create a single unified view of this, will be tricky I think, but well 
worth it.
Mar 31 2018
prev sibling next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, March 31, 2018 19:38:06 Andrei Alexandrescu via Digitalmars-d 
wrote:

 We need to have a simple recipe on how to define a canonical object.
 That would include requirements such as:

 * should work with mutable, const, immutable, and shared
 * the right way to define constructor(s)
 * the right way to define copying
 * the right way to define destructor(s)

 I've asked my student Razvan to document the behavior of this(this). His
 work:

 https://github.com/dlang/dmd/pull/8055
 https://github.com/dlang/dlang.org/pull/2281
 https://github.com/dlang/dlang.org/pull/2299

 ... reveals a puzzling array of behaviors. Sometimes the typechecking is
 wrong, too.

 I think it's very important for us to have a simple, correct, and
 canonical way of defining structs in the D language that work with the
 language features: qualifiers, pure, safe, and nogc.

 Once we have that, we can encapsulate desirable abstractions (such as
  nogc safe collections that work in pure code), regardless of how
 difficult their implementations might be. It seems that currently
 this(this) does not allow us to do that.

 Eduard, another student I work with, has made steps toward a collections
 library that rant into difficulties:

 *  safe is achievable with relative ease

 * immutable and const are very difficult, but we have an attack
 (assuming copy construction gets taken care of)

 *  nogc is doable with a couple of conventions for allocators

 * pure is difficult

 * making them work together is very difficult

 We need to offer features, tools, and guidance toward creating simple
 encapsulated types that work well with D's own abstractions. Once we
 have that we can build libraries to work satisfactorily for any domain.

 I think the way to move forward is to deprecate this(this) entirely and
 create a DIP that allows people to define truly encapsulated structs.
 This is important, urgent, and of huge impact.

 I am looking for folks to assist me in creating a DIP for that. There
 will be a _lot_ of work involved, so don't take it lightly.


Another potential issue is whether any of this does or should relate to

https://github.com/dlang/DIPs/pull/109

and it's solution for hooking into to moves. I'm not at all sure that what
happens with that needs to be related to this at all, but it might.

- Jonathan M Davis
Mar 31 2018
next sibling parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Sunday, 1 April 2018 at 01:56:40 UTC, Jonathan M Davis wrote:

 Another potential issue is whether any of this does or should 
 relate to

 https://github.com/dlang/DIPs/pull/109

 and it's solution for hooking into to moves. I'm not at all 
 sure that what happens with that needs to be related to this at 
 all, but it might.

 - Jonathan M Davis


And before we think about `opMove` we should, IMO, make the 
compiler pass by move in more cases, for instance, in range 
constructors such as

this(Source source)
{
     this.source = source; // last occurrence of `source` can be 
moved
}

I'd be happy to help out with adding this in dmd. Andrei has 
already showed interest in this idea.
Apr 01 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, April 01, 2018 12:23:29 Per Nordlöw via Digitalmars-d wrote:

 On Sunday, 1 April 2018 at 01:56:40 UTC, Jonathan M Davis wrote:

 Another potential issue is whether any of this does or should
 relate to

 https://github.com/dlang/DIPs/pull/109

 and it's solution for hooking into to moves. I'm not at all
 sure that what happens with that needs to be related to this at
 all, but it might.

 - Jonathan M Davis


 And before we think about `opMove` we should, IMO, make the
 compiler pass by move in more cases, for instance, in range
 constructors such as

 this(Source source)
 {
      this.source = source; // last occurrence of `source` can be
 moved
 }

 I'd be happy to help out with adding this in dmd. Andrei has
 already showed interest in this idea.


I don't see what that has to do with opMove other than the fact that in such
cases, you'd then end up with opMove being called instead of the postblit
constructor. It seems to me that whether opMove is part of the language or
not when such improvements are made is irrelevant. They should be able to be
done independently of one other.

As I understand it, the motivations behind opMove really have nothing to do
with how often moves are made vs copying. It has to do with types that have
serious issues if they're moved. The Weka guys (and probably others) have
use cases where the fact that an object is moved without any way for them to
detect it and do stuff like update pointers to the object is a serious
problem. If anything, their use case might be better off right now if _no_
moves were ever done (at least until something like opMove is in the
language), since the moves result in bugs.

- Jonathan M Davis
Apr 01 2018
prev sibling parent Shachar Shemesh <shachar weka.io> writes:
On 01/04/18 04:56, Jonathan M Davis wrote:

 Another potential issue is whether any of this does or should relate to
 
 https://github.com/dlang/DIPs/pull/109
 
 and it's solution for hooking into to moves. I'm not at all sure that what
 happens with that needs to be related to this at all, but it might.
 
 - Jonathan M Davis


I was actually going to start a new thread about it.

On the one hand, nothing in the opMove DIP is directly affected by this. 
On the other, if we're moving away from "copy and then fix" mentality, 
then opMove should also reflect this.

The problem is that the alternative solution has a much bigger impact on 
backward compatibility. I'm really tempted to try and push this DIP as 
is, only renaming "opMove" to "opPostMove". This way, we can push the 
simpler version as is, and when (and it will take a while) "this(this)" 
deprecation finally comes, we can implement "opMove" as an in-process 
user hook.

Shachar
Apr 04 2018
prev sibling next sibling parent reply Johannes Loher <johannes.loher fg4f.de> writes:
Am 01.04.2018 um 01:38 schrieb Andrei Alexandrescu:

 We need to have a simple recipe on how to define a canonical object.
 That would include requirements such as:
 
 * should work with mutable, const, immutable, and shared
 * the right way to define constructor(s)
 * the right way to define copying
 * the right way to define destructor(s)
 
 I've asked my student Razvan to document the behavior of this(this). His
 work:
 
 https://github.com/dlang/dmd/pull/8055
 https://github.com/dlang/dlang.org/pull/2281
 https://github.com/dlang/dlang.org/pull/2299
 
 ... reveals a puzzling array of behaviors. Sometimes the typechecking is
 wrong, too.
 
 I think it's very important for us to have a simple, correct, and
 canonical way of defining structs in the D language that work with the
 language features: qualifiers, pure, safe, and nogc.
 
 Once we have that, we can encapsulate desirable abstractions (such as
  nogc safe collections that work in pure code), regardless of how
 difficult their implementations might be. It seems that currently
 this(this) does not allow us to do that.
 
 Eduard, another student I work with, has made steps toward a collections
 library that rant into difficulties:
 
 *  safe is achievable with relative ease
 
 * immutable and const are very difficult, but we have an attack
 (assuming copy construction gets taken care of)
 
 *  nogc is doable with a couple of conventions for allocators
 
 * pure is difficult
 
 * making them work together is very difficult
 
 We need to offer features, tools, and guidance toward creating simple
 encapsulated types that work well with D's own abstractions. Once we
 have that we can build libraries to work satisfactorily for any domain.
 
 I think the way to move forward is to deprecate this(this) entirely and
 create a DIP that allows people to define truly encapsulated structs.
 This is important, urgent, and of huge impact.
 
 I am looking for folks to assist me in creating a DIP for that. There
 will be a _lot_ of work involved, so don't take it lightly.
 
 
 Thanks,
 
 Andrei


This seems really sudden, april fool's joke? Not really sure, as there
are real problems with this(this)...
Apr 01 2018
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/1/18 6:04 AM, Johannes Loher wrote:

 This seems really sudden, april fool's joke? Not really sure, as there
 are real problems with this(this)...


I'm glad I've sent it yesterday then at least in my time zone :o).

This looks sudden but isn't. Eduard and I have been blocked by this 
problem seriously whilst working on the collections library.


Andrei
Apr 01 2018
prev sibling parent reply bachmeier <no spam.net> writes:
On Sunday, 1 April 2018 at 10:04:04 UTC, Johannes Loher wrote:

 This seems really sudden, april fool's joke? Not really sure, 
 as there are real problems with this(this)...


What I was wondering too. I mean, breaking changes just don't 
happen to this language. Now there will be, without even an 
indication of how existing code would have to be rewritten, or 
how this large-scale breakage is different than the breakages 
that just can't happen because reasons. I guess that's why 
there's always the disclaimer, "We'll only break code if there's 
a really good reason." That reason is "in case we want to".
Apr 01 2018
next sibling parent reply Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Sunday, 1 April 2018 at 10:49:22 UTC, bachmeier wrote:

 On Sunday, 1 April 2018 at 10:04:04 UTC, Johannes Loher wrote:

 This seems really sudden, april fool's joke? Not really sure, 
 as there are real problems with this(this)...


 What I was wondering too. I mean, breaking changes just don't 
 happen to this language. Now there will be, without even an 
 indication of how existing code would have to be rewritten, or 
 how this large-scale breakage is different than the breakages 
 that just can't happen because reasons. I guess that's why 
 there's always the disclaimer, "We'll only break code if 
 there's a really good reason." That reason is "in case we want 
 to".


Nothing has been lay out yet and people are already freaking out. 
No wonder nothing gets done anymore.
Apr 01 2018
parent reply bachmeier <no spam.net> writes:
On Sunday, 1 April 2018 at 11:17:38 UTC, Patrick Schluter wrote:


 What I was wondering too. I mean, breaking changes just don't 
 happen to this language. Now there will be, without even an 
 indication of how existing code would have to be rewritten, or 
 how this large-scale breakage is different than the breakages 
 that just can't happen because reasons. I guess that's why 
 there's always the disclaimer, "We'll only break code if 
 there's a really good reason." That reason is "in case we want 
 to".


 Nothing has been lay out yet and people are already freaking 
 out. No wonder nothing gets done anymore.


No, the reason nothing gets done is because "that would break 
code" is used to kill every proposal that comes along. Someone 
that only responds to proposals with "write a DIP" proceeds to 
announce a major piece of the language will be deprecated without 
writing a DIP himself. Corporate leadership doesn't work with an 
open source project. I could have gotten more involved long ago, 
but I'd rather not jump on a ship that's sailing in circles. From 
the many comments I've seen, I'm not the only one.
Apr 02 2018
parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 2 April 2018 at 13:55:25 UTC, bachmeier wrote:


 No, the reason nothing gets done is because "that would break 
 code" is used to kill every proposal that comes along. Someone 
 that only responds to proposals with "write a DIP" proceeds to 
 announce a major piece of the language will be deprecated 
 without writing a DIP himself. Corporate leadership doesn't 
 work with an open source project. I could have gotten more 
 involved long ago, but I'd rather not jump on a ship that's 
 sailing in circles. From the many comments I've seen, I'm not 
 the only one.


Andrei wrote in the message


I am looking for folks to assist me in creating a DIP for that.
There will be a _lot_ of work involved, so don't take it lightly.


So, let's keep the discussion factual. I'm pretty sure that every 
aspect will be taken in account and pondered prior to a decision.

I'm +1 on major breaking changes if they drive D towards a better 
shape.

/Paolo
Apr 02 2018
parent reply bachmeier <no spam.net> writes:
On Monday, 2 April 2018 at 15:30:23 UTC, Paolo Invernizzi wrote:


 Andrei wrote in the message


I am looking for folks to assist me in creating a DIP for that.
There will be a _lot_ of work involved, so don't take it 
lightly.


 So, let's keep the discussion factual. I'm pretty sure that 
 every aspect will be taken in account and pondered prior to a 
 decision.

 I'm +1 on major breaking changes if they drive D towards a 
 better shape.


Andrei is asking others to write a DIP to formalize a decision he 
has already made. Yet when Manu posts here, he responds:


 The good news is there is a way to ensure your proposal gets a 
 fair shake of the stick: write a DIP.



 Filing a DIP is like filing a police report: once it's in the 
 system, we're obligated to work on it. There's a guarantee of a 
 response. In the case of acceptance, we commit to implementing 
 the proposal. In the case of rejection, we give a clear 
 motivation of the reasons we had. In the case we ask for 
 further review, we provide clear feedback of what would take 
 the DIP through another iteration.



 Forum discussions are the equivalent of complaining loudly in a 
 bar to people you know and also to strangers within earshot 
 that your house was broken into. Until you file a report, the 
 police will not look into it.


There's not even an attempt made to pretend there's symmetry. The 
only way for Manu (and basically anyone else) to propose a change 
is to write a DIP. Andrei won't even participate in discussions 
without a DIP. That's probably a good idea. What's not a good 
idea is to make unilateral decisions about major breaking 
changes, posting in the forum, and then asking others to write 
the DIP. That's corporate software development, and it's very 
discouraging to potential contributors.
Apr 02 2018
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/02/2018 12:00 PM, bachmeier wrote:

 On Monday, 2 April 2018 at 15:30:23 UTC, Paolo Invernizzi wrote:
 

 Andrei wrote in the message


 I am looking for folks to assist me in creating a DIP for that.
 There will be a _lot_ of work involved, so don't take it lightly.


 So, let's keep the discussion factual. I'm pretty sure that every 
 aspect will be taken in account and pondered prior to a decision.

 I'm +1 on major breaking changes if they drive D towards a better shape.


 
 Andrei is asking others to write a DIP to formalize a decision he has 
 already made. Yet when Manu posts here, he responds:


Apologies for the misunderstanding. I'll be the first author of the DIP 
and plan to dedicate a lot of time to it. I was just asking for others 
to join me in this important and urgent effort.

Andrei
Apr 02 2018
prev sibling parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 2 April 2018 at 16:00:11 UTC, bachmeier wrote:

 On Monday, 2 April 2018 at 15:30:23 UTC, Paolo Invernizzi wrote:



 Andrei wrote in the message





I am looking for folks to assist me in creating a DIP for that.
There will be a _lot_ of work involved, so don't take it 
lightly.







 Andrei is asking others to write a DIP to formalize a decision


<snip>


 There's not even an attempt made to pretend there's symmetry. 
 The only way for Manu (and basically anyone else) to propose a 
 change is to write a DIP. Andrei won't even participate in 
 discussions without a DIP. That's probably a good idea. What's 
 not a good idea is to make unilateral decisions about major 
 breaking changes, posting in the forum, and then asking others 
 to write the DIP. That's corporate software development, and 
 it's very discouraging to potential contributors.


I think you are plain wrong on this: the 'P' in a DIP stands for 
Proposal, so any decision is not taken yet. And I'll bet:

- Andrei will be the main author or he will partecipate in the 
writing.
- the DIP will follow the usual proceeding, exactly like the 
others.

/Paolo
Apr 02 2018
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/02/2018 12:14 PM, Paolo Invernizzi wrote:

 On Monday, 2 April 2018 at 16:00:11 UTC, bachmeier wrote:

 On Monday, 2 April 2018 at 15:30:23 UTC, Paolo Invernizzi wrote:


 

 Andrei wrote in the message




 

 I am looking for folks to assist me in creating a DIP for that.
 There will be a _lot_ of work involved, so don't take it lightly.






 

 Andrei is asking others to write a DIP to formalize a decision


 
 <snip>
 

 There's not even an attempt made to pretend there's symmetry. The only 
 way for Manu (and basically anyone else) to propose a change is to 
 write a DIP. Andrei won't even participate in discussions without a 
 DIP. That's probably a good idea. What's not a good idea is to make 
 unilateral decisions about major breaking changes, posting in the 
 forum, and then asking others to write the DIP. That's corporate 
 software development, and it's very discouraging to potential 
 contributors.


 
 I think you are plain wrong on this: the 'P' in a DIP stands for 
 Proposal, so any decision is not taken yet. And I'll bet:
 
 - Andrei will be the main author or he will partecipate in the writing.
 - the DIP will follow the usual proceeding, exactly like the others.


Affirmative. No need to discuss this further, it's a simple 
misunderstanding. I'd agree with bachmeier if his perception was correct.
Apr 02 2018
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/1/2018 3:49 AM, bachmeier wrote:

 What I was wondering too. I mean, breaking changes just don't happen to this 
 language. Now there will be, without even an indication of how existing code 
 would have to be rewritten, or how this large-scale breakage is different than 
 the breakages that just can't happen because reasons. I guess that's why
there's 
 always the disclaimer, "We'll only break code if there's a really good
reason." 
 That reason is "in case we want to".


The idea is not to introduce a breaking change. Postblits will remain. The copy 
constructor will be an additional construct, and if one is defined, it will be 
preferred.

Eventually (years later) postblits will be slowly deprecated and eventually
removed.
Apr 04 2018
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 4 April 2018 at 22:30:39 UTC, Walter Bright wrote:

 On 4/1/2018 3:49 AM, bachmeier wrote:

 What I was wondering too. I mean, breaking changes just don't 
 happen to this language. Now there will be, without even an 
 indication of how existing code would have to be rewritten, or 
 how this large-scale breakage is different than the breakages 
 that just can't happen because reasons. I guess that's why 
 there's always the disclaimer, "We'll only break code if 
 there's a really good reason." That reason is "in case we want 
 to".


 The idea is not to introduce a breaking change. Postblits will 
 remain. The copy constructor will be an additional construct, 
 and if one is defined, it will be preferred.

 Eventually (years later) postblits will be slowly deprecated 
 and eventually removed.


Could you describe how a copy constructor differs from postblit ?
Apr 04 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, April 04, 2018 22:41:39 Stefan Koch via Digitalmars-d wrote:

 On Wednesday, 4 April 2018 at 22:30:39 UTC, Walter Bright wrote:

 On 4/1/2018 3:49 AM, bachmeier wrote:

 What I was wondering too. I mean, breaking changes just don't
 happen to this language. Now there will be, without even an
 indication of how existing code would have to be rewritten, or
 how this large-scale breakage is different than the breakages
 that just can't happen because reasons. I guess that's why
 there's always the disclaimer, "We'll only break code if
 there's a really good reason." That reason is "in case we want
 to".


 The idea is not to introduce a breaking change. Postblits will
 remain. The copy constructor will be an additional construct,
 and if one is defined, it will be preferred.

 Eventually (years later) postblits will be slowly deprecated
 and eventually removed.


 Could you describe how a copy constructor differs from postblit ?


With a postblit, all of the members are copied _before_ the postblit
constructor is called, meaning that the struct was blitted, and then the
postblit constructors were called for each member variable that has a
postblit constructor, before the struct's postblit constructor is called.
So, inside the postblit constructor, you're mutating an already copied
object. It's doing work post the copy - hence postblit.

With a copy constructor, the copy is being directly constructed. Each member
will be initialized exactly once rather than being initialized and then
mutated. Ideally, each member that was not explicitly initialized by the
copy constructor would be automatically copied from the original so that you
wouldn't have to explicitly copy each member, so syntactically, it may look
very much like a postblit constructor, but the key difference is that with a
copy constructor, the copy is directly constructed with whatever changes are
done by the copy constructor as opposed to having it copied and then
mutated.

- Jonathan M Davis
Apr 04 2018
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Apr 04, 2018 at 03:30:39PM -0700, Walter Bright via Digitalmars-d wrote:

 On 4/1/2018 3:49 AM, bachmeier wrote:

 What I was wondering too. I mean, breaking changes just don't happen
 to this language. Now there will be, without even an indication of
 how existing code would have to be rewritten, or how this
 large-scale breakage is different than the breakages that just can't
 happen because reasons. I guess that's why there's always the
 disclaimer, "We'll only break code if there's a really good reason."
 That reason is "in case we want to".


 
 The idea is not to introduce a breaking change. Postblits will remain.
 The copy constructor will be an additional construct, and if one is
 defined, it will be preferred.


This may lead to trouble, unless we explicitly make it illegal for
something to have both a postblit and a copy ctor. Having the two
interact with each other seems likely to lead to a lot of pathological,
hard-to-understand results.



 Eventually (years later) postblits will be slowly deprecated and
 eventually removed.


Actually, if it's possible to have them coexist and there are no
disadvantages to doing so, why not just leave both of them in?  Then
cases where postblits give the best performance could just be left
as-is, and cases that require a copy ctor can use a copy ctor.


T

-- 
Klein bottle for rent ... inquire within. -- Stephen Mulraney
Apr 04 2018
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, April 04, 2018 15:51:25 H. S. Teoh via Digitalmars-d wrote:

 On Wed, Apr 04, 2018 at 03:30:39PM -0700, Walter Bright via Digitalmars-d 


wrote:

 On 4/1/2018 3:49 AM, bachmeier wrote:

 What I was wondering too. I mean, breaking changes just don't happen
 to this language. Now there will be, without even an indication of
 how existing code would have to be rewritten, or how this
 large-scale breakage is different than the breakages that just can't
 happen because reasons. I guess that's why there's always the
 disclaimer, "We'll only break code if there's a really good reason."
 That reason is "in case we want to".


 The idea is not to introduce a breaking change. Postblits will remain.
 The copy constructor will be an additional construct, and if one is
 defined, it will be preferred.


 This may lead to trouble, unless we explicitly make it illegal for
 something to have both a postblit and a copy ctor. Having the two
 interact with each other seems likely to lead to a lot of pathological,
 hard-to-understand results.


+1


 Eventually (years later) postblits will be slowly deprecated and
 eventually removed.


 Actually, if it's possible to have them coexist and there are no
 disadvantages to doing so, why not just leave both of them in?  Then
 cases where postblits give the best performance could just be left
 as-is, and cases that require a copy ctor can use a copy ctor.


The big disadvantage is that you would have to deal with two distinct ways
to copy objects and would have to explain them both to everyone learnig D.
AFAIK, the only real advantage to a postblit constructor is that you don't
have to explicitly copy everything like you do when declaring a copy
constructor in C++ - you just change the parts that need to be changed after
the copy is made in order to make it a deep copy or do whatever you need to
do to finish making the copy what it should be. As such, if we define copy
constructors in a way that only requires listing the members that are
actually going to be different from the default copy, then I really don't
see an advantage to postblit constructors.

Hmmm. And actually, thinking about that, I almost wonder if we could just
change this(this) to be a copy constructor. Assume for a moment that inside
this(this), we provide a new symbol that is the this pointer/reference for
the original. We then have make it so that any member variable in the newly
constructed object which is read before it is assigned is default-copied. As
such,

this(this)
{
}

would just default-copy everything basically as it does now. And if you
currently had

this(this)
{
    _foo = _foo.dup;
}

then after the change, it would still have the same semantics, because the
_foo is used before it is assigned, so it must be default-copied. However,
if we provide the object being copied via an invisible ref (like this)
called orig, doing

this(this)
{
    _foo = orig._foo;
}

then that would directly initialize _foo without it being default-copied. As
such, existing postblit constructors should continue to work, but it would
be straightforward to turn them into copy constructors, and most of the
changes would be underneath the hood.

I don't know if orig is the best name (and making it a keyword would be
really annoying, since it's a name that I use all the time), but on the
surface at least, the idea seems sound to me. And assuming that there isn't
a big implementation issue that makes it a serious problem, it should
actually make transitioning from postblit constructors to copy constructors
very clean and negate the need for any kind of deprecation process.

- Jonathan M Davis
Apr 04 2018
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Apr 04, 2018 at 05:08:26PM -0600, Jonathan M Davis via Digitalmars-d
wrote:
[...]

 AFAIK, the only real advantage to a postblit constructor is that you
 don't have to explicitly copy everything like you do when declaring a
 copy constructor in C++ - you just change the parts that need to be
 changed after the copy is made in order to make it a deep copy or do
 whatever you need to do to finish making the copy what it should be.
 As such, if we define copy constructors in a way that only requires
 listing the members that are actually going to be different from the
 default copy, then I really don't see an advantage to postblit
 constructors.


That's a very interesting concept.  It could be a nice way of bridging
current semantics over to the new semantics.  Some of the existing
postblit code might even be usable as-is, when reinterpreted in this
way!



 Hmmm. And actually, thinking about that, I almost wonder if we could
 just change this(this) to be a copy constructor. Assume for a moment
 that inside this(this), we provide a new symbol that is the this
 pointer/reference for the original. We then have make it so that any
 member variable in the newly constructed object which is read before
 it is assigned is default-copied. As such,
 
 this(this)
 {
 }
 
 would just default-copy everything basically as it does now.


Why even bother with declaring an empty this(this)?  Just leave it out,
and the compiler assumes default-copy.



 And if you currently had
 
 this(this)
 {
     _foo = _foo.dup;
 }
 
 then after the change, it would still have the same semantics, because
 the _foo is used before it is assigned, so it must be default-copied.


That's a pretty neat way of transitioning from current semantics to new
semantics.  I like it.



 However, if we provide the object being copied via an invisible ref
 (like this) called orig, doing
 
 this(this)
 {
     _foo = orig._foo;
 }
 
 then that would directly initialize _foo without it being
 default-copied. As such, existing postblit constructors should
 continue to work, but it would be straightforward to turn them into
 copy constructors, and most of the changes would be underneath the
 hood.


[...]

I like this idea.  Except the syntax could be improved:

	this(this orig)		// <-- N.B.
	{
		_foo = orig._foo;
	}

Let the user specify the symbol, instead of introducing yet another
implicit magic identifier. :-)


T

-- 
"I suspect the best way to deal with procrastination is to put off the
procrastination itself until later. I've been meaning to try this, but
haven't gotten around to it yet. " -- swr
Apr 05 2018
parent Nick Treleaven <nick geany.org> writes:
On Thursday, 5 April 2018 at 18:46:25 UTC, H. S. Teoh wrote:

 I like this idea.  Except the syntax could be improved:

 	this(this orig)		// <-- N.B.
 	{


It's already valid syntax, equivalent to `this(typeof(this) 
orig)`. It can be called explicitly, but is ignored for implicit 
construction. Perhaps we will just enable `this(typeof(this) 
orig)` for implicit construction too, if there's no postblit 
defined.
Apr 11 2018
prev sibling next sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Saturday, 31 March 2018 at 23:38:06 UTC, Andrei Alexandrescu 
wrote:

 * immutable and const are very difficult, but we have an attack 
 (assuming copy construction gets taken care of)

 * pure is difficult


How about removing pure, const and immutable?
Apr 01 2018
prev sibling next sibling parent Shachar Shemesh <shachar weka.io> writes:
By sheer coincidence, I've just stumbled upon another limitation of 
this(this). I am not sure whether it is already documented.

Let's define struct S1 with no copying allowed, and put it as a member 
of struct S2. Under C++ as well as under D, this automatically makes S2 
also non-copyable.
Under C++, however, I can do this:

struct S1 {
     S1() {}

     // Make S1 non-copyable
     S1(const S1 &that) = delete;
     S1 &operator=(const S1 &that) = delete;
};

struct S2 {
     int a;
     S1 s;

     S2(int _a) : a(_a) {
     }

     S2(const S2 &that) : a(that.a) {
     }
};

int main() {
     S2 s(17);
     S2 v(s); // This compiles, invoking S2's copy ctor
}


In other words, I can tell the compiler that I know how to copy S2 
without having to copy the member S1.

Under D, this simply doesn't work. If S1 has  disable this(this), any 
struct that has S1 as a member will be uncopyable, and this is not 
overridable.

Shachar
Apr 02 2018
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Saturday, 31 March 2018 at 23:38:06 UTC, Andrei Alexandrescu 
wrote:

 * should work with mutable, const, immutable, and shared


The problem is that postblit is not overloadable, so make it 
overloadable, and problems with overloading will be solved.


 * immutable and const are very difficult, but we have an attack 
 (assuming copy construction gets taken care of)


Collections must be filled somehow, so they are inherently 
mutable, immutable collections need a whole different design 
approach, it doesn't look specific to postblit.


 * pure is difficult


Purity depends on written code. Running impure code in copy 
constructor won't make it pure.
Apr 02 2018
prev sibling next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Saturday, 31 March 2018 at 23:38:06 UTC, Andrei Alexandrescu 
wrote:

 [snip]

 * immutable and const are very difficult, but we have an attack 
 (assuming copy construction gets taken care of)


Would it be easier if the const/immutable containers were 
considered separate types? For instance, in the code below, there 
is InoutFoo and then Foo takes InoutFoo as an alias this (you 
could do the same thing with immutable, but then you’d have to 
include two get functions). This would be like inheriting the 
InoutFoo.

With better syntax, InoutFoo would be something like inout(Foo) 
and the compiler could recognize that the mutable constructor is 
also defined and to call that when appropriate.

struct InoutFoo
{
     int a;

     this(int b) inout
     {
         this.a = b;
     }

     int get() inout
     {
         return a;
     }
}

struct Foo
{
     InoutFoo inoutfoo;
     alias inoutfoo this;

     this(int b)
     {
         a = b;
     }

     void set(int b)
     {
         a = b;
     }
}


void main()
{
     auto x = immutable InoutFoo(1);
     auto y = Foo(1);

     assert(is(typeof(y) : typeof(x)));

     //x.a++; //not allowed
     y.a++;

     assert(x.a == 1);
     assert(y.a == 2);

     assert(x.get == 1);
     y.set(3);
     assert(y.get == 3);
}
Apr 03 2018
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, April 05, 2018 11:46:25 H. S. Teoh via Digitalmars-d wrote:

 On Wed, Apr 04, 2018 at 05:08:26PM -0600, Jonathan M Davis via
 Digitalmars-d wrote: [...]

 Hmmm. And actually, thinking about that, I almost wonder if we could
 just change this(this) to be a copy constructor. Assume for a moment
 that inside this(this), we provide a new symbol that is the this
 pointer/reference for the original. We then have make it so that any
 member variable in the newly constructed object which is read before
 it is assigned is default-copied. As such,

 this(this)
 {
 }

 would just default-copy everything basically as it does now.


 Why even bother with declaring an empty this(this)?  Just leave it out,
 and the compiler assumes default-copy.


You wouldn't declare it. I was just talking about it to discuss the
semantics.


 However, if we provide the object being copied via an invisible ref
 (like this) called orig, doing

 this(this)
 {

     _foo = orig._foo;

 }

 then that would directly initialize _foo without it being
 default-copied. As such, existing postblit constructors should
 continue to work, but it would be straightforward to turn them into
 copy constructors, and most of the changes would be underneath the
 hood.


 [...]

 I like this idea.  Except the syntax could be improved:

   this(this orig)     // <-- N.B.
   {
       _foo = orig._foo;
   }

 Let the user specify the symbol, instead of introducing yet another
 implicit magic identifier. :-)


Well, that might be better in that it allows the user to choose the name,
but it then opens up the question of whether this(this) should be
deprecated, and part of the point was to try and just do it all in-place and
simply turn postblit constructors into copy constructors - though it could
be argued that this(this) was just a copy constructor where you didn't
bother to use the original, and anyone actually wanting to take advantage of
copy constructors is going to need to make changes anyway. But if this(this)
simply becomes a copy constructor (whether a default name is used or
something like this(this orig) is used to define the name), then we
hopefully would be able to avoid actually deprecating anything and just make
more code work and making it so that only minor tweaks are necessary to
actually take full advantage of it being a copy constructor instead of
mutating the copy in the copy constructor.

The other issue that occurred to me after posting about this was the
question of what to do about const. Ideally, this(this) or this(this orig)
would work just fine with const (assuming that it didn't do anything that
would mutate the object), but someone might want to actually require const
or overload the copy constructor for the const case, in which case, I was
thinking making this(const this), but with your suggestion, I guess that it
would become either this(const this orig) or this(this const orig). I don't
know. As usual, adding const into the mix makes things messier, but part of
the point of all of this is to make it so that we can copy arbitrarily
complex const objects.

In any case, I think that the idea of trying to simply turn postblit
constructors into copy constructors has merit. I certainly don't think that
we need to be doing anything like

struct S
{
    this(ref S orig)
    {
        ...
    }
}

where everyone will have to convert their postblits over time in order to
avoid the deprecation and where it actually risks conflicting with existing
constructors.

- Jonathan M Davis
Apr 05 2018