www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - mutable, const, immutable guidelines

reply "Daniel Davidson" <nospam spam.com> writes:
I'm reviewing Ali's insightful presentation from 2013 DConf. I 
wonder has he or anyone else followed up on the concepts or 
formalized some guidelines that could achieve consensus. I 
definitely agree it would be helpful to have a 50 Ways To Improve 
Your D. The first thing I'd like to see is a set of guidelines on 
mutability along the lines he discussed in his talk. But as it 
stands, I don't know if there was any finalization/consensus. For 
instance, from the first two guidelines at the end (after several 
iterative reconstructions of those guidelines):

1. If a variable is never mutated, make it const, not immutable.
2. Make the parameter reference to immutable if that is how you 
will use it anyway. It is fine to ask a favor from the caller.
...

If you follow (1) exclusively, why the need for immutable in the 
language at all?

Maybe it is a philosophical question, but where does immutability 
really come from? Is it an aspect of some piece of data or is it 
a promise that function will not change it? Or is it a 
requirement by a function that data passed not be changed by 
anyone else?

The two keywords cover all in some sense.

I found the end of the video amusing, when one gentleman looking 
at a rather sophisticated "canonical" struct with three overloads 
for 'this(...)' and two overloads for opAssign, asked if all 
those methods were required. I think there was back and forth and 
head-scratching. Another asked if the language designers were 
happy with the resultant complexity. Naturally the answer was yes 
- it is a good mix. If that is the case I wonder if the reason is 
they don't write software in D like Ali was looking to develop. I 
imagine standard library code is much more functional than OO by 
its very nature. My money says you will not find a struct S 
constructed like Ali's in the wild - it is just too much 
boilerplate. But surely they have their own guidelines/approaches.

By posting this I am not looking for a single answer to the 
simple question above as it is just one of many questions in the 
set of "how do you choose among the options in general". If you 
have such guidelines - please post them.

Thanks
Dan
Oct 02 2013
next sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 10/02/2013 06:09 AM, Daniel Davidson wrote:


 I'm reviewing Ali's insightful presentation from 2013 DConf. I
 wonder has he or anyone else followed up on the concepts or
 formalized some guidelines that could achieve consensus.


I have not followed up on those concepts.


 1. If a variable is never mutated, make it const, not immutable.
 2. Make the parameter reference to immutable if that is how you will use
 it anyway. It is fine to ask a favor from the caller.
 ...

 If you follow (1) exclusively, why the need for immutable in the
 language at all?


For one, immutable is thread-safe. Also, an object can safely hold on to 
a piece of data and trust that it will never change. A file name is a 
good example: An object need not copy a string that is given to it as a 
constructor parameter. That string will be immutable as long as that 
reference is valid.


 Maybe it is a philosophical question, but where does
 immutability really come from? Is it an aspect of some piece of
 data or is it a promise that function will not change it? Or is
 it a requirement by a function that data passed not be changed
 by anyone else?


The two concepts are necessarily conflated in languages like C++ because 
for those languages there is only the 'const' keyword.

D's immutable is the last point you made: a requirement by a function 
that data passed not be changed by anyone else?


 I found the end of the video amusing, when one gentleman looking at a
 rather sophisticated "canonical" struct with three overloads for
 'this(...)' and two overloads for opAssign, asked if all those methods
 were required.


I think you are referring to Ben Gertzfield's question. He later told me 
that he was sorry that he asked the wrong question at the wrong time. :D

To be honest, that slide was added more as an after thought. I suspect 
that that canonical struct is simpler today after changes made to dmd 
since the conference. (I will look at it later.) For example, there are 
no 'pure' keywords around in that code. I think the (ability to) use of 
that keyword will simplify matters.

However, I should have answered Ben's question by the following:

* If the struct is simply a value type or has no mutable indirections, 
no member function is really necessary. D takes care of it automatically.

* Sometimes post-blit will be necessary for correctness. For example, we 
may not want two objects share the same internal buffer, File, etc.

* As noted in the presentation, the default behavior of struct 
assignment in D is exception-safe: first copy then swap. In some cases 
that automatic behavior is less than optimal e.g. when an object's 
existing buffer can be reused; no need to "copy then swap (which 
implicitly destroys)" in that case. (The spec or Andrei's book has that 
exact case as an example.) So, opAssign should be for optimization 
reasons only.

(I will look at these again later.)


 By posting this I am not looking for a single answer to the simple
 question above as it is just one of many questions in the set of "how do
 you choose among the options in general". If you have such guidelines -
 please post them.

 Thanks
 Dan


Thanks for asking these questions.

Ali
Oct 02 2013
next sibling parent "Daniel Davidson" <nospam spam.com> writes:
On Wednesday, 2 October 2013 at 17:07:55 UTC, Ali Çehreli wrote:

 On 10/02/2013 06:09 AM, Daniel Davidson wrote:


 1. If a variable is never mutated, make it const, not


 immutable.

 2. Make the parameter reference to immutable if that is how


 you will use

 it anyway. It is fine to ask a favor from the caller.
 ...

 If you follow (1) exclusively, why the need for immutable in


 the

 language at all?


 For one, immutable is thread-safe.


Ok - then new guidelines should be added to the effect that, "if 
you want a guarantee of read-only that can be counted on across 
threads, use immutable". But, I'm not sure that is a sensible 
guideline or when you know up-front to deal with it. Often 
threading is an afterthought. My point is not that immutable is 
not useful, but that if you never make an lvalue immutable in the 
guidelines (via guideline 1), it is hard to see them ever being 
used at all. The case in the slides where it was used was to when 
a method required it on a parameter type with mutable aliasing. 
In this case the client was required to copy beforehand. In other 
words, I think even the summary simple guidelines are not so 
simple, maybe because they are incomplete. I would love to help 
complete them.


 Also, an object can safely hold on to a piece of data and trust 
 that it will never change.


I think for this to be true in general, the piece of data must be 
immutable, not just tail-immutable. If the class holds onto a 
string (i.e. only tail-immutable), the class can add to the 
string - just not change the existing characters.


 A file name is a good example: An object need not copy a string 
 that is given to it as a constructor parameter. That string 
 will be immutable as long as that reference is valid.


IMHO file name and ultimately string are the *worst* example :-) 
The reason is they are a special case because they provide *safe 
sharing*. String encapsulates its contiguous data and does not 
allow mutation of elements. But it does allow concatenation that 
by design detail can not be seen by other references to the same 
data because of copy-on-write. Granted, if you have a handle to  
immutable(T)[] you know *it* (i.e. the complete string) will not 
change (unless you change it). The problem is it is just a 
byproduct of the implementation details of immutable(T)[]. From 
the referrer's point of view it is immutable.

For example, the following comparable associative array does not 
have the same benefit as string since two references to the same 
underlying data *do see changes*:

     import std.stdio;

     struct V {
       string s = "I'm a value";
     }

     alias immutable(V)[int] Map;

     void main() {
       Map m = [ 1:V(), 2:V("bar") ];
       Map m2 = m;
       m2[3] = V("moo");
       writeln(m);
       writeln(m2);
     }

The reason I think string is a bad example is one might 
incorrectly assume tail-immutable is good enough for true 
read/only thread-safe data. And it is for string or any 
immutable(T)[], but not in general.



 Maybe it is a philosophical question, but where does
 immutability really come from? Is it an aspect of some piece


 of

 data or is it a promise that function will not change it? Or


 is

 it a requirement by a function that data passed not be changed
 by anyone else?


 The two concepts are necessarily conflated in languages like 
 C++ because for those languages there is only the 'const' 
 keyword.

 D's immutable is the last point you made: a requirement by a 
 function that data passed not be changed by anyone else?


 I found the end of the video amusing, when one gentleman


 looking at a

 rather sophisticated "canonical" struct with three overloads


 for

 'this(...)' and two overloads for opAssign, asked if all


 those methods

 were required.


 I think you are referring to Ben Gertzfield's question. He 
 later told me that he was sorry that he asked the wrong 
 question at the wrong time. :D


I can understand - I would not want to put you on the spot on 
stage either. But, I have no qualms doing it in the news groups 
:-)


 To be honest, that slide was added more as an after thought. I 
 suspect that that canonical struct is simpler today after 
 changes made to dmd since the conference. (I will look at it 
 later.) For example, there are no 'pure' keywords around in 
 that code. I think the (ability to) use of that keyword will 
 simplify matters.

 However, I should have answered Ben's question by the following:

 * If the struct is simply a value type or has no mutable 
 indirections, no member function is really necessary. D takes 
 care of it automatically.


Which I wonder if is a good idea. Immediately from that comes the 
problem that "D takes care of it" breaks when you go from no 
mutable aliasing to some mutable aliasing. That mutable aliasing 
could be deep down in the composition chain. I know you can feel 
this issue since your slides point out cases where code may break 
when well encapsulated structs make changes that should not 
impact client code but will break them.


 * Sometimes post-blit will be necessary for correctness. For 
 example, we may not want two objects share the same internal 
 buffer, File, etc.


Agreed - but post-blits are easier said than done with nested 
data. For instance, if you have a member that is a struct that 
does not provide a post-blit (i.e. the original author did not 
concern himself with sharing) then you have to effectively write 
his post-blit for him in your post-blit. This is why I lobby for 
language supported generalized dup.


 * As noted in the presentation, the default behavior of struct 
 assignment in D is exception-safe: first copy then swap. In 
 some cases that automatic behavior is less than optimal e.g. 
 when an object's existing buffer can be reused; no need to 
 "copy then swap (which implicitly destroys)" in that case. (The 
 spec or Andrei's book has that exact case as an example.) So, 
 opAssign should be for optimization reasons only.

 (I will look at these again later.)


I look forward to it and would be glad to help get to a good set 
of guidelines. For instance filling out this, which only covers 
variable declarations - not parameter passing, with a robust 
"when to use".

| context         | T | C(T) | I(T) | I(T)[] |
|-----------------+---+------+------+--------|
| local stack     |   |      |      |        |
| local heap      |   |      |      |        |
| global          |   |      |      |        |
| instance member |   |      |      |        |
| static member   |   |      |      |        |

You could actually have two tables - one for T1 with no mutable 
aliasing and one for T2 with mutable aliasing. But relying on 
that in the decision matrix may lead to issues.

Thanks,
Dan
Oct 02 2013
prev sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 10/02/2013 10:07 AM, Ali Çehreli wrote:


 On 10/02/2013 06:09 AM, Daniel Davidson wrote:

  > I'm reviewing Ali's insightful presentation from 2013 DConf. I
  > wonder has he or anyone else followed up on the concepts or
  > formalized some guidelines that could achieve consensus.

 I have not followed up on those concepts.

  > 1. If a variable is never mutated, make it const, not immutable.
  > 2. Make the parameter reference to immutable if that is how you 


will use

  > it anyway. It is fine to ask a favor from the caller.
  > ...


I have to report that I am not finding much time to go deeper in these 
issues. However, I spent some time this weekend, gaining a hint of 
enlightenment.

immutable is a requirement on two concepts, second of which is new to me:

1) As it is commonly known, it is a requirement that the data does not 
mutate.

2) It is also a requirement on the type that the variables of that type 
will always be usable as immutable. I have not formalized "be usable" 
yet but what I mean is that the type cannot be modified in the future in 
a way that inhibits its use in existing code.

To look at just one usage example, the following line carries two 
requirements:

     auto a = T();
     immutable b = a;

1) b will be an immutable copy of a.

2) T will always be usable as in that fashion.

If T appears on an API, it is the responibility of the user to ensure 
whether they are allowed to treat T in that way. Otherwise, they risk 
maintainability if the module decides to change T in any way that fits 
the module's needs. If they have not yet advertised that T can be used 
as immutable, it should not be.

So, the question is how can a module expose a type that can always be 
used as immutable? Can the module guarantee that its future needs will 
never disallow T's use as immutable (e.g. by inserting mutable 
references into the type)? Can the library always guarantee that T can 
be used as immutable by improving its definition, e.g. by adding 
necessary opCast, opEquals, 'alias this', constructors, etc?

I don't know these answers... :/ (yet! :p)

Ali
Oct 07 2013
parent reply "qznc" <qznc web.de> writes:
On Monday, 7 October 2013 at 17:57:11 UTC, Ali Çehreli wrote:

 To look at just one usage example, the following line carries 
 two requirements:

     auto a = T();
     immutable b = a;

 1) b will be an immutable copy of a.

 2) T will always be usable as in that fashion.

 If T appears on an API, it is the responibility of the user to 
 ensure whether they are allowed to treat T in that way. 
 Otherwise, they risk maintainability if the module decides to 
 change T in any way that fits the module's needs. If they have 
 not yet advertised that T can be used as immutable, it should 
 not be.


I do not agree with you, that the user has the responsibility. 
Rather I think the provider of T has the responsibility to 
maintain backwards compatibility. My principle is "anything is 
allowed, unless explicitly forbidden". This includes immutable 
type constructing. It is great that we get a type error, if 
backwards compatibility is broken.

Nevertheless, the question of responsibility seems to be 
subjective. Are there any clear technical reasons for either way?

Unfortunately, there is no way for the provider to allow or 
disallow immutable(T). Maybe there should be a UDA or something 
for this?
Oct 08 2013
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 10/08/2013 03:12 PM, qznc wrote:


 On Monday, 7 October 2013 at 17:57:11 UTC, Ali Çehreli wrote:

 To look at just one usage example, the following line carries two
 requirements:

     auto a = T();
     immutable b = a;

 1) b will be an immutable copy of a.

 2) T will always be usable as in that fashion.

 If T appears on an API, it is the responibility of the user to ensure
 whether they are allowed to treat T in that way. Otherwise, they risk
 maintainability if the module decides to change T in any way that fits
 the module's needs. If they have not yet advertised that T can be used
 as immutable, it should not be.


 I do not agree with you, that the user has the responsibility.


I have difficulty agreeing with myself as well. :) However, the power of 
immutable makes me think so. Interestingly, once a user creates an 
immutable variable of a type, that type must support that use.


 Rather I
 think the provider of T has the responsibility to maintain backwards
 compatibility.


Agreed but I don't know how. Here is challenge: Let's start with the 
following program:

// Library type
struct MyInt
{
     int i;
}

void main()
{
     // User code
     auto a = MyInt(1);
     immutable b = a;
}

Let's assume that the library adds a private dynamic array of ints to 
that type:

// Library type
struct MyInt
{
     int i;
     private int[] history;    // <-- Added
}

void main()
{
     // User code
     auto a = MyInt(1);
     immutable b = a;          // <-- Existing code breaks
}

Error: cannot implicitly convert expression (a) of type MyInt to 
immutable(MyInt)

Apparently, the library made a change that made its type 
non-immutable-able. :p What is missing in MyInt? How should it be 
defined instead of simply adding 'history'?


 My principle is "anything is allowed, unless explicitly
 forbidden". This includes immutable type constructing. It is great that
 we get a type error, if backwards compatibility is broken.


Agreed. What is the solution?


 Nevertheless, the question of responsibility seems to be subjective. Are
 there any clear technical reasons for either way?


Technically, MyInt is in a breaking state because the user used it as 
immutable. One way to look at this situation is to observe that the user 
took some freedom of the library just by using its type as immutable.


 Unfortunately, there is no way for the provider to allow or disallow
 immutable(T). Maybe there should be a UDA or something for this?


I think the language is lacking tools to support the use case above.

Ali
Oct 08 2013
next sibling parent "Daniel Davidson" <nospam spam.com> writes:
On Wednesday, 9 October 2013 at 04:31:55 UTC, Ali Çehreli wrote:

 On 10/08/2013 03:12 PM, qznc wrote:


 On Monday, 7 October 2013 at 17:57:11 UTC, Ali Çehreli wrote:

 To look at just one usage example, the following line




 carries two

 requirements:

     auto a = T();
     immutable b = a;

 1) b will be an immutable copy of a.

 2) T will always be usable as in that fashion.

 If T appears on an API, it is the responibility of the user




 to ensure

 whether they are allowed to treat T in that way. Otherwise,




 they risk

 maintainability if the module decides to change T in any way




 that fits

 the module's needs. If they have not yet advertised that T




 can be used

 as immutable, it should not be.


 I do not agree with you, that the user has the responsibility.


 I have difficulty agreeing with myself as well. :) However, the 
 power of immutable makes me think so. Interestingly, once a 
 user creates an immutable variable of a type, that type must 
 support that use.


 Rather I
 think the provider of T has the responsibility to maintain


 backwards

 compatibility.


 Agreed but I don't know how. Here is challenge: Let's start 
 with the following program:

 // Library type
 struct MyInt
 {
     int i;
 }

 void main()
 {
     // User code
     auto a = MyInt(1);
     immutable b = a;
 }

 Let's assume that the library adds a private dynamic array of 
 ints to that type:

 // Library type
 struct MyInt
 {
     int i;
     private int[] history;    // <-- Added
 }

 void main()
 {
     // User code
     auto a = MyInt(1);
     immutable b = a;          // <-- Existing code breaks
 }

 Error: cannot implicitly convert expression (a) of type MyInt 
 to immutable(MyInt)

 Apparently, the library made a change that made its type 
 non-immutable-able. :p What is missing in MyInt? How should it 
 be defined instead of simply adding 'history'?


The example is great. I would say it like "Apparently the library 
made a change that added mutable aliasing and therefore broke the 
special logic the compiler applies to types without aliasing". I 
think the root cause is the language special casing structs with 
no mutable aliasing by allowing the "immutable b = a" to work in 
the first place. One step toward a solution would be the 
existence of a generalized dup. Then calls to "immutable b = a" 
with mutable aliasing could lower to a call to the generalized 
dup, causing a deep copy and therefore still allowing assignment 
from non-immutable to immutable. This has its own set of problems 
- like what if the library designer wants sharing and not deep 
copies (perhaps they employ copy on write semantics on their own).

What are other solutions or ways to prevent this jump from no 
aliasing to mutable aliasing breaking code? Maybe assume mutable 
aliasing on all your structs from the start, then those 
statements like "immutable b = a" won't appear in the first place.

struct T {
   ...
   version(unittest) int[] _justToAddMutableAliasing;
}

Now you know that assignment of "immutable b = a" will fail in 
unittests. But this is crazy and it would only help your users if 
they run unittests.


 Technically, MyInt is in a breaking state because the user used 
 it as immutable. One way to look at this situation is to 
 observe that the user took some freedom of the library just by 
 using its type as immutable.


I don't quite understand the terminology "using it's type as 
immutable". I think the breaking state comes from copying in 
general having two flavors - shallow and deep. Transitive deep 
copy always offers the option of immutable source and target. 
Shallow copy also offers the option of immutable source and 
target, as long as the type has no mutable aliasing, simply 
because a shallow copy is also a deep copy if no aliasing. And, 
shallow copy does not offer the option of immutable in the face 
of types with mutable aliasing. It is when types go from having 
no mutable aliasing to having mutable aliasing (and the reverse) 
that the rules of the compiler change the game.


 Unfortunately, there is no way for the provider to allow or


 disallow

 immutable(T). Maybe there should be a UDA or something for


 this?


One way might be to disallow copy and assignment (is this a D 
capability?) and provide pure factory that only returns 
immutable(T).


 I think the language is lacking tools to support the use case 
 above.


I think if the default assignment and copy constructor for 
structs with aliasing were automatic transitive deep copy this 
issue would disappear. Other issues would come up. Also, I'm 
pretty sure Walter is not a fan of this concept because he does 
not advocate copying any data in postblits.


 Ali
Oct 09 2013
prev sibling parent reply "qznc" <qznc web.de> writes:
On Wednesday, 9 October 2013 at 04:31:55 UTC, Ali Çehreli wrote:

 On 10/08/2013 03:12 PM, qznc wrote:

 On Monday, 7 October 2013 at 17:57:11 UTC, Ali Çehreli wrote:

 To look at just one usage example, the following line




 carries two

 requirements:

     auto a = T();
     immutable b = a;

 1) b will be an immutable copy of a.

 2) T will always be usable as in that fashion.

 If T appears on an API, it is the responibility of the user




 to ensure

 whether they are allowed to treat T in that way. Otherwise,




 they risk

 maintainability if the module decides to change T in any way




 that fits

 the module's needs. If they have not yet advertised that T




 can be used

 as immutable, it should not be.


 I do not agree with you, that the user has the responsibility.


 I have difficulty agreeing with myself as well. :) However, the 
 power of immutable makes me think so. Interestingly, once a 
 user creates an immutable variable of a type, that type must 
 support that use.


 Rather I
 think the provider of T has the responsibility to maintain


 backwards

 compatibility.


 Agreed but I don't know how. Here is challenge: Let's start 
 with the following program:

 // Library type
 struct MyInt
 {
     int i;
 }

 void main()
 {
     // User code
     auto a = MyInt(1);
     immutable b = a;
 }

 Let's assume that the library adds a private dynamic array of 
 ints to that type:

 // Library type
 struct MyInt
 {
     int i;
     private int[] history;    // <-- Added
 }

 void main()
 {
     // User code
     auto a = MyInt(1);
     immutable b = a;          // <-- Existing code breaks
 }

 Error: cannot implicitly convert expression (a) of type MyInt 
 to immutable(MyInt)


Maybe the fact that D allows this implicit copy to immutable is 
the problem? If one could require the use of a specific function, 
this function could be overridden with working behavior. The 
following code works.

import std.exception: assumeUnique;

struct MyInt
{
   int i;
   private int[] history;    // <-- Added
}

// idup for creating immutable MyInts
immutable(MyInt) idup(const MyInt mi) pure nothrow  trusted {
   MyInt cpy = MyInt(mi.i);
   return cast(immutable) cpy;
}

// special version for performance
immutable(MyInt) idup(immutable MyInt mi) pure nothrow  trusted {
   return mi;
}

unittest {
   auto a = MyInt(1);
   immutable b = a.idup;     // <-- Code does not break
}

D could either remove the implicit-copy-to-immutable or provide a 
special copy-constructor for immutable structs.
Oct 10 2013
parent "Daniel Davidson" <nospam spam.com> writes:
On Thursday, 10 October 2013 at 23:06:23 UTC, qznc wrote:

 Maybe the fact that D allows this implicit copy to immutable is 
 the problem? If one could require the use of a specific 
 function, this function could be overridden with working 
 behavior. The following code works.


Yes - the issue arises because the language has different 
behavior for structs with mutable aliasing and structs without. 
For structs without mutable aliasing a copy is safe so crossing 
bounds between any of (mutable, const, immutable) is easily 
achieved.


 import std.exception: assumeUnique;

 struct MyInt
 {
   int i;
   private int[] history;    // <-- Added
 }

 // idup for creating immutable MyInts
 immutable(MyInt) idup(const MyInt mi) pure nothrow  trusted {
   MyInt cpy = MyInt(mi.i);
   return cast(immutable) cpy;
 }

 // special version for performance
 immutable(MyInt) idup(immutable MyInt mi) pure nothrow  trusted 
 {
   return mi;
 }

 unittest {
   auto a = MyInt(1);
   immutable b = a.idup;     // <-- Code does not break
 }

 D could either remove the implicit-copy-to-immutable or provide 
 a special copy-constructor for immutable structs.


See this discussion: 
http://forum.dlang.org/thread/fmhkvogowjlduqercsjn forum.dlang.org
Oct 10 2013
prev sibling next sibling parent reply "qznc" <qznc web.de> writes:
On Wednesday, 2 October 2013 at 13:09:34 UTC, Daniel Davidson 
wrote:

 1. If a variable is never mutated, make it const, not immutable.
 2. Make the parameter reference to immutable if that is how you 
 will use it anyway. It is fine to ask a favor from the caller.
 ...


I think guideline 1 should be about "arguments" not "variables". 
Functions should take const arguments, so they can be called with 
mutable and immutable values.

I would rephrase the second guideline as: "Never dup or idup an 
argument to make it mutable or immutable, but require the caller 
to do this (might be able to avoid it)".

For more guidelines:

3. Return value, which are freshly created, should never be 
const. They should be mutable or immutable.

Basically, I cannot come up with a case, where const would be 
preferable to mutable and immutable. Maybe someone is able to 
find a good example? Of course, this does not hold for returned 
values, which are retrieved from somewhere (array,container,etc). 
In this case, I have no general advice.

4. Data structures should not restrict themselves to be mutable, 
const, or immutable.

This is a noble goal, which in reality is probably either trivial 
or impossible. So I am not sure, if it is worthy to be called a 
guideline.
Oct 08 2013
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 10/08/2013 03:03 PM, qznc wrote:


 On Wednesday, 2 October 2013 at 13:09:34 UTC, Daniel Davidson wrote:

 1. If a variable is never mutated, make it const, not immutable.
 2. Make the parameter reference to immutable if that is how you will
 use it anyway. It is fine to ask a favor from the caller.
 ...


 I think guideline 1 should be about "arguments" not "variables".
 Functions should take const arguments, so they can be called with
 mutable and immutable values.


As demonstrated in the presentation, const erases the actual type. So, 
the function must take a copy if the argument is to be used in an 
immutable context.

Instead, if the immutable appears on the interface, that copy will be 
avoided.


 I would rephrase the second guideline as: "Never dup or idup an argument
 to make it mutable or immutable, but require the caller to do this
 (might be able to avoid it)".


Agreed. But it means you agree with me that immutable should indeed 
appear on function signatures as needed.


 For more guidelines:

 3. Return value, which are freshly created, should never be const. They
 should be mutable or immutable.


Agreed. I say that it should be mutable if the function is pure, as such 
a return values is automatically convertible to immutable.


 Basically, I cannot come up with a case, where const would be preferable
 to mutable and immutable. Maybe someone is able to find a good example?


You mean on the return type? Perhaps if the freshly created object 
carries references to existing data that you don't want mutated. Making 
the head const makes tail const; so the members would all be protected.

However, it is always possible to return a mutable object of a type that 
has const members:

struct Result
{
     const(int)[] reference_to_precious_existing_data;
}

Result freshly_created_value()
{
     // ...
}


 Of course, this does not hold for returned values, which are retrieved
 from somewhere (array,container,etc). In this case, I have no general
 advice.

 4. Data structures should not restrict themselves to be mutable, const,
 or immutable.


What is the template of a struct that can be used as such? Providing 
simple values seems to be insufficient:

struct MyInt
{
     int i;
     private int[] history;
}

What else should the struct above have to make it usable as mutable, 
const, immutable?


 This is a noble goal, which in reality is probably either trivial or
 impossible. So I am not sure, if it is worthy to be called a guideline.


We have to nail this down already. We have this great addition of 
immutable in the language but we either don't know how to use it or the 
language is lacking hopefully trivial things to make it work.

Ali
Oct 08 2013
next sibling parent "qznc" <qznc web.de> writes:
On Wednesday, 9 October 2013 at 04:41:35 UTC, Ali Çehreli wrote:

 4. Data structures should not restrict themselves to be


 mutable, const,

 or immutable.


 What is the template of a struct that can be used as such? 
 Providing simple values seems to be insufficient:

 struct MyInt
 {
     int i;
     private int[] history;
 }

 What else should the struct above have to make it usable as 
 mutable, const, immutable?


 This is a noble goal, which in reality is probably either


 trivial or

 impossible. So I am not sure, if it is worthy to be called a


 guideline.

 We have to nail this down already. We have this great addition 
 of immutable in the language but we either don't know how to 
 use it or the language is lacking hopefully trivial things to 
 make it work.


For this case, there are basically two approaches: a) prevent the 
use as immutable in the first place or b) make it work as 
immutable by force.

For version a) we could have a UDA like this:

    NeverImmutable struct MyInt { int i; }

   auto x = immutable(MyInt)(42);  // compile error
   immutable y = MyInt(42);        // compile error

However, the library writer would have to anticipate the private 
dynamic array, which probably does not happen. Defensive coding 
like putting  NeverImmutable on everything would defeat the 
purpose as well.

For version b) we would relax the type system, which opens an 
ugly can of worms of unknown size. Basically, immutable data 
would be allowed to have mutable private data, if the author 
sufficiently asserts everything is correct. Since there are lots 
of interesting compiler optimizations with immutable things, I 
can imagine there will be lots of subtle errors until one can 
make something like this work.

At this point, I cannot imagine that there is a solution which 
makes this case work, so I think we have to accept this breakage.
Oct 09 2013
prev sibling parent reply "Daniel Davidson" <nospam spam.com> writes:
On Wednesday, 9 October 2013 at 04:41:35 UTC, Ali Çehreli wrote:

 On 10/08/2013 03:03 PM, qznc wrote:


 On Wednesday, 2 October 2013 at 13:09:34 UTC, Daniel Davidson


 wrote:

 1. If a variable is never mutated, make it const, not




 immutable.

 2. Make the parameter reference to immutable if that is how




 you will

 use it anyway. It is fine to ask a favor from the caller.
 ...


 I think guideline 1 should be about "arguments" not


 "variables".

 Functions should take const arguments, so they can be called


 with

 mutable and immutable values.


 As demonstrated in the presentation, const erases the actual 
 type. So, the function must take a copy if the argument is to 
 be used in an immutable context.

 Instead, if the immutable appears on the interface, that copy 
 will be avoided.


True. But what does it mean to be "used in an immutable context". 
In your example you pass a variable to a function `void 
usefulFunction(string s)`. The problem with transitivity is once 
you start making rules regarding it, those rules must also be 
transitive (or turtles all the way down as you say). The point 
is, the desire in the example wants to use the guideline only at 
the top level and break it at the next level. If usefulFunction 
were following the guideline it would have a signature `void 
usefulFunction(const(char)[] s)` instead.

If the guideline were: "if a function parameter is not mutated, 
make it const" and it were applied that way everywhere, the 
problem you suggest would not be hit. But we can't go changing 
any `standardFunction(string s)` to 
`standardFunction(const(char)[] s)`. Using const on parameters 
seems to be the prevalent guideline - even though passing strings 
breaks it.

In regards to parameters, when, if ever, is there benefit to 
immutable over const (if you assume all called functions from 
that function have const parameters/signatures). The only reason 
I can think is if a *member* function wants to hold onto a 
reference or copy for future use and the designer wants to 
guarantee that member is truly immutable. An example would be a 
struct that initializes with an immutable(T) that the struct 
wants to then use throughout its life.

struct S { this(ref immutable(T) t) { _t = t; }  immutable(T) _t; 
}

This is all very confusing and to me unsettled. I've brought up 
questions like this many times and have yet to hear Walter or 
Andrei weigh in. Either we (you and I) are missing something and 
making things much more difficult than necessary or it is just 
difficult. I think a third likely possibility is that composition 
in the face of structs with mutable aliasing is not heavily used 
by them.


 I would rephrase the second guideline as: "Never dup or idup


 an argument

 to make it mutable or immutable, but require the caller to do


 this

 (might be able to avoid it)".


 Agreed. But it means you agree with me that immutable should 
 indeed appear on function signatures as needed.


I don't really agree. Naturally, if const were used all the way 
down there would be no need for copies. It is the introduction of 
immutable (via string in your example) that breaks the rule and 
causes the pain. But then if the rule were adopted globally - 
immutable is never used on parameters in a signature, only const 
is used, when would you start seeing any benefit to the immutable 
keyword?


 For more guidelines:

 3. Return value, which are freshly created, should never be


 const. They

 should be mutable or immutable.


 Agreed. I say that it should be mutable if the function is 
 pure, as such a return values is automatically convertible to 
 immutable.


 Basically, I cannot come up with a case, where const would be


 preferable

 to mutable and immutable. Maybe someone is able to find a


 good example?


I don't know if these are good examples, but they are examples:

 From zip.d:
         const(void)[] compress(const(void)[] buf)

 From vibe:
  	const(Json) opIndex(string key) const {


[snip]


 4. Data structures should not restrict themselves to be


 mutable, const,

 or immutable.


 What is the template of a struct that can be used as such? 
 Providing simple values seems to be insufficient:

 struct MyInt
 {
     int i;
     private int[] history;
 }

 What else should the struct above have to make it usable as 
 mutable, const, immutable?


I'm confused by the term "usable as mutable, const, immutable"? 
Isn't it true that types are not mutable, const, immutable in 
their definition but only in their context?

[snip]


 We have to nail this down already. We have this great addition 
 of immutable in the language but we either don't know how to 
 use it or the language is lacking hopefully trivial things to 
 make it work.


Amen, brother!

Either it is a great addition and usable, or it is not so great 
an addition. At times I find myself questioning even the basic 
benefit of immutable in the context of sharing. I've heard that 
immutable is great because it means the data can be shared across 
threads and you have confidence it will never change. This sounds 
good. But then you have to choose signatures:

void foo(const(MutableType) mt);
void foo(immutable(MutableType) mt);

Naturally the inclination is to choose the second as it is a 
stronger guarantee that no threads are changing the data. Cool. 
But wait, the first one still probably requires the same 
guarantee, even if it does not state it. If in the const case 
another thread changes the state of mt during the function foo 
then it will fail. foo actually requires that mt not be change by 
other threads during its operation - that is the only sane way of 
using the data in the function.

Take, for example, LinearCongruentialEngine from random.d. It has 
a function:

     bool opEquals(ref const LinearCongruentialEngine rhs) const

Why is it using const here instead of immutable? Does it not care 
about other threads? No - it just is not smart enough to deal 
with it. It assumes other threads won't be changing it or if they 
are caveat developer. So when do you use immutable as a signal 
that not only will this function not change it, no thread will 
change it either? Probably when you know you have to deal with 
threading. But that then would be coupling two maybe orthogonal 
decisions - the threading of a program and the signature of 
functions.

Where is Scott Myers when you need him :-)



 Ali
Oct 09 2013
next sibling parent reply "qznc" <qznc web.de> writes:
On Wednesday, 9 October 2013 at 15:50:55 UTC, Daniel Davidson 
wrote:

 void foo(const(MutableType) mt);
 void foo(immutable(MutableType) mt);

 Naturally the inclination is to choose the second as it is a 
 stronger guarantee that no threads are changing the data. Cool. 
 But wait, the first one still probably requires the same 
 guarantee, even if it does not state it. If in the const case 
 another thread changes the state of mt during the function foo 
 then it will fail. foo actually requires that mt not be change 
 by other threads during its operation - that is the only sane 
 way of using the data in the function.


Why "the first one still probably requires the same guarantee"?
Why would foo fail if "in the const case  another thread changes 
the state of mt"?
If foo is not thread-safe, then it should require immutable.


 Take, for example, LinearCongruentialEngine from random.d. It 
 has a function:

     bool opEquals(ref const LinearCongruentialEngine rhs) const

 Why is it using const here instead of immutable? Does it not 
 care about other threads? No - it just is not smart enough to 
 deal with it. It assumes other threads won't be changing it or 
 if they are caveat developer. So when do you use immutable as a 
 signal that not only will this function not change it, no 
 thread will change it either? Probably when you know you have 
 to deal with threading. But that then would be coupling two 
 maybe orthogonal decisions - the threading of a program and the 
 signature of functions.


Hm, that actually looks like a bug. If LinearCongruentialEngine 
is instantiated with a UIntType larger than size_t, it is not 
thread-safe, but this seems to be claimed in the section header.

Why are those two decisions orthogonal? The signature of a 
function is a contract between caller and callee. If an argument 
is const, it means the callee says he can handle others changing 
the state concurrently.
Oct 09 2013
next sibling parent "Daniel Davidson" <nospam spam.com> writes:
On Wednesday, 9 October 2013 at 23:05:27 UTC, qznc wrote:

 On Wednesday, 9 October 2013 at 15:50:55 UTC, Daniel Davidson 
 wrote:

 void foo(const(MutableType) mt);
 void foo(immutable(MutableType) mt);

 Naturally the inclination is to choose the second as it is a 
 stronger guarantee that no threads are changing the data. 
 Cool. But wait, the first one still probably requires the same 
 guarantee, even if it does not state it. If in the const case 
 another thread changes the state of mt during the function foo 
 then it will fail. foo actually requires that mt not be change 
 by other threads during its operation - that is the only sane 
 way of using the data in the function.


 Why "the first one still probably requires the same guarantee"?
 Why would foo fail if "in the const case  another thread 
 changes the state of mt"?
 If foo is not thread-safe, then it should require immutable.


 Take, for example, LinearCongruentialEngine from random.d. It 
 has a function:

    bool opEquals(ref const LinearCongruentialEngine rhs) const

 Why is it using const here instead of immutable? Does it not 
 care about other threads? No - it just is not smart enough to 
 deal with it. It assumes other threads won't be changing it or 
 if they are caveat developer. So when do you use immutable as 
 a signal that not only will this function not change it, no 
 thread will change it either? Probably when you know you have 
 to deal with threading. But that then would be coupling two 
 maybe orthogonal decisions - the threading of a program and 
 the signature of functions.


 Hm, that actually looks like a bug. If LinearCongruentialEngine 
 is instantiated with a UIntType larger than size_t, it is not 
 thread-safe, but this seems to be claimed in the section header.

 Why are those two decisions orthogonal? The signature of a 
 function is a contract between caller and callee. If an 
 argument is const, it means the callee says he can handle 
 others changing the state concurrently.


I don't think that is how people interpret and use const. The 
general guideline is, for function parameters use const because 
it is most accepting - it takes mutable, immutable and const. If 
there are multiple threads developer beware.

Threading is or can be somewhat orthogonal to function signatures 
simply because you can write a whole lot of code with the const 
guideline above. Then with proper serialized access use that 
thread unaware code in multi-threaded context.
Oct 10 2013
prev sibling parent reply =?UTF-8?B?Q2hyaXN0aWFuIEvDtnN0bGlu?= <christian.koestlin gmail.com> writes:
On 10/10/13 1:05 , qznc wrote:
Very interesting discussion!


 contract between caller and callee. If an argument is const, it means
 the callee says he can handle others changing the state concurrently.


i think what the usual understanding of const for an argument to callee 
is, what is written at http://dlang.org/const3.html. that means for me, 
that callee promises not to change the data itself. perhaps a bettr 
description would be readonly.

usually you would think that no one else should change the data while 
callee runs. but at least with c++ i could imagine running callee in a 
thread with a reference to a const thing which changes underneath and 
while callee is running.
Oct 10 2013
parent reply "qznc" <qznc web.de> writes:
On Thursday, 10 October 2013 at 18:39:32 UTC, Christian Köstlin 
wrote:

 On 10/10/13 1:05 , qznc wrote:
 Very interesting discussion!


 contract between caller and callee. If an argument is const,


 it means

 the callee says he can handle others changing the state


 concurrently.
 i think what the usual understanding of const for an argument 
 to callee is, what is written at http://dlang.org/const3.html. 
 that means for me, that callee promises not to change the data 
 itself. perhaps a bettr description would be readonly.


The linked page clearly says "It may, however, be changed by 
another reference to that same data."


 usually you would think that no one else should change the data 
 while callee runs. but at least with c++ i could imagine 
 running callee in a thread with a reference to a const thing 
 which changes underneath and while callee is running.


A const argument gives information to both sides. For the caller 
it means callee does not modify the data. For the callee it means 
somebody else (another thread) may modify the data at any time.

An immutable argument means the same for the caller, but gives an 
additional guarantee to the callee that nobody modifies the data.
Oct 10 2013
parent reply "Daniel Davidson" <nospam spam.com> writes:
On Thursday, 10 October 2013 at 23:12:24 UTC, qznc wrote:

 The linked page clearly says "It may, however, be changed by 
 another reference to that same data."


 usually you would think that no one else should change the 
 data while callee runs. but at least with c++ i could imagine 
 running callee in a thread with a reference to a const thing 
 which changes underneath and while callee is running.


 A const argument gives information to both sides. For the 
 caller it means callee does not modify the data. For the callee 
 it means somebody else (another thread) may modify the data at 
 any time.

 An immutable argument means the same for the caller, but gives 
 an additional guarantee to the callee that nobody modifies the 
 data.


That is probably a reasonable interpretation... but I think it 
will only get you pain. The fact is, regardless of your 
interpretation of "const" arguments - the general guideline is 
"prefer const because immutables and mutables can be passed in". 
Think of it this way: you are writing a very generic function 
that requires inputs that you do not mutate. If you choose 
immutable you are making your life easier, I guess, since you 
know data is not being changed on you. But you are also 
preventing your function from being called by any arguments that 
are already mutable (assuming the type has mutable aliasing) 
because crossing the mutable to immutable divide is not part of 
the language. So, the only way then for mutable types to make it 
into your function is to copy them and this may or may not even 
be possible. And if you truly know no code is mutating your data 
it is a tougher sell.

On the other hand if you go with const parameters, your data 
could in fact change on you. Unfortunately, it can cause terrible 
angst when the only options make you wish for other options.

Have you actually written nested D data types with mutable 
aliasing (not just slices but assoc arrays) and used immutable in 
the signatures? I have tried and not succeeded.

I guess this just highlights the need for guidelines.
Oct 10 2013
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 11/10/13 01:43, Daniel Davidson wrote:

 That is probably a reasonable interpretation... but I think it will only get
you
 pain. The fact is, regardless of your interpretation of "const" arguments - the
 general guideline is "prefer const because immutables and mutables can be
passed
 in".


Which makes much sense for code that is intended to be generically useful.


 Think of it this way: you are writing a very generic function that requires
 inputs that you do not mutate. If you choose immutable you are making your life
 easier, I guess, since you know data is not being changed on you. But you are
 also preventing your function from being called by any arguments that are
 already mutable (assuming the type has mutable aliasing) because crossing the
 mutable to immutable divide is not part of the language. So, the only way then
 for mutable types to make it into your function is to copy them and this may or
 may not even be possible. And if you truly know no code is mutating your data
it
 is a tougher sell.


If you are insisting that your function receive immutable inputs, you are in 
turn constraining the user of that code.  If you really want your function to
be 
"generic", isn't the appropriate thing to allow the user the choice of whether 
they want to pass it immutable data, or to take the risk of passing it mutable
data?

Surely there's a tradeoff -- you may well feel the resulting safety is worth
it, 
and there may be other benefits, such as optimizations, that you can derive
from 
having only immutable data.

But for the standard library of a multiparadigm language, it's not appropriate 
to force users to use immutable data.


 On the other hand if you go with const parameters, your data could in fact
 change on you. Unfortunately, it can cause terrible angst when the only options
 make you wish for other options.


Well, that depends.  If you go with const parameters but you pass it immutable 
data, you know it won't change on you.  You, the user, are in control.


 Have you actually written nested D data types with mutable aliasing (not just
 slices but assoc arrays) and used immutable in the signatures? I have tried and
 not succeeded.


Yes, I tried it, and ran into this problem.  But generally these days I use
"in" 
to mark arguments that are not intended to be modified.  I personally prefer 
that -- it has a semantic meaning, and how it translates into code is for the 
compiler.


 I guess this just highlights the need for guidelines.


Guidelines are always nice. :-)
Oct 10 2013
prev sibling next sibling parent "qznc" <qznc web.de> writes:
On Wednesday, 9 October 2013 at 15:50:55 UTC, Daniel Davidson 
wrote:

 I would rephrase the second guideline as: "Never dup or idup


 an argument

 to make it mutable or immutable, but require the caller to do


 this

 (might be able to avoid it)".


 Agreed. But it means you agree with me that immutable should 
 indeed appear on function signatures as needed.


 I don't really agree. Naturally, if const were used all the way 
 down there would be no need for copies. It is the introduction 
 of immutable (via string in your example) that breaks the rule 
 and causes the pain. But then if the rule were adopted globally 
 - immutable is never used on parameters in a signature, only 
 const is used, when would you start seeing any benefit to the 
 immutable keyword?


The benefit of immutable is that nobody else can change it. 
Functions that would be non-thread-safe with const arguments are 
thread-safe with immutable arguments.

That relationship with thread-safety almost seems to be a good 
guideline.

5. If a function is not thread-safe due to const arguments, 
consider making it immutable. Otherwise you must explicitly 
document the lack of thread-safety.



 For more guidelines:

 3. Return value, which are freshly created, should never be


 const. They

 should be mutable or immutable.


 Agreed. I say that it should be mutable if the function is 
 pure, as such a return values is automatically convertible to 
 immutable.


 Basically, I cannot come up with a case, where const would be


 preferable

 to mutable and immutable. Maybe someone is able to find a


 good example?


 I don't know if these are good examples, but they are examples:

 From zip.d:
         const(void)[] compress(const(void)[] buf)


Looking it up (in zlib.d actually) it could just as well return 
an immutable array, since it is freshly allocated.


 From vibe:
  	const(Json) opIndex(string key) const {


Not sure where exactly this comes from. Seems to be some kind of 
map? In this case the value is not "freshly created", so the 
guideline does not apply.


 I'm confused by the term "usable as mutable, const, immutable"? 
 Isn't it true that types are not mutable, const, immutable in 
 their definition but only in their context?


MyInt is a type, probably mutable.
immutable(MyInt) is also a type, but immutable.
const(MyInt) is also a type, but const.

These are example of immutable and const being used as type 
constructors. They create a new type from an existing one (MyInt).
Oct 09 2013
prev sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 09/10/13 17:50, Daniel Davidson wrote:

 Take, for example, LinearCongruentialEngine from random.d. It has a function:

      bool opEquals(ref const LinearCongruentialEngine rhs) const

 Why is it using const here instead of immutable?


Among other things, because no LinearCongruentialEngine will ever, ever be 
immutable -- because it won't work if it's immutable.  Whereas ref const I 
believe is simply saying, "You can't mutate it via this reference."  I'm not 
sure if it ought technically to be auto ref const.

In any case, I think the general reason is in order to be agnostic about type 
qualifications.  Try:

     BigInt a = BigInt(5);
     immutable BigInt b = immutable(BigInt)(5);
     writeln(a.opEquals(b));

... which works.  BigInt.opEquals takes input by an auto ref const.
Oct 10 2013
prev sibling parent reply "Daniel Davidson" <nospam spam.com> writes:
On Wednesday, 2 October 2013 at 13:09:34 UTC, Daniel Davidson 
wrote:

 I'm reviewing Ali's insightful presentation from 2013 DConf. I 
 wonder has he or anyone else followed up on the concepts or 
 formalized some guidelines that could achieve consensus. I 
 definitely agree it would be helpful to have a 50 Ways To 
 Improve Your D. The first thing I'd like to see is a set of 
 guidelines on mutability along the lines he discussed in his 
 talk. But as it stands, I don't know if there was any 
 finalization/consensus. For instance, from the first two 
 guidelines at the end (after several iterative reconstructions 
 of those guidelines):

 1. If a variable is never mutated, make it const, not immutable.
 2. Make the parameter reference to immutable if that is how you 
 will use it anyway. It is fine to ask a favor from the caller.
 ...

 If you follow (1) exclusively, why the need for immutable in 
 the language at all?

 Maybe it is a philosophical question, but where does 
 immutability really come from? Is it an aspect of some piece of 
 data or is it a promise that function will not change it? Or is 
 it a requirement by a function that data passed not be changed 
 by anyone else?

 The two keywords cover all in some sense.

 I found the end of the video amusing, when one gentleman 
 looking at a rather sophisticated "canonical" struct with three 
 overloads for 'this(...)' and two overloads for opAssign, asked 
 if all those methods were required. I think there was back and 
 forth and head-scratching. Another asked if the language 
 designers were happy with the resultant complexity. Naturally 
 the answer was yes - it is a good mix. If that is the case I 
 wonder if the reason is they don't write software in D like Ali 
 was looking to develop. I imagine standard library code is much 
 more functional than OO by its very nature. My money says you 
 will not find a struct S constructed like Ali's in the wild - 
 it is just too much boilerplate. But surely they have their own 
 guidelines/approaches.

 By posting this I am not looking for a single answer to the 
 simple question above as it is just one of many questions in 
 the set of "how do you choose among the options in general". If 
 you have such guidelines - please post them.

 Thanks
 Dan


After trying for several days to use immutable with types 
containing some mutable aliasing I have come to the conclusion 
that maybe rule number one should be:

If you have a type that has now or may ever have in the future 
any mutable aliasing (e.g. inclusion of T[] or T1[T1] where Ts 
are mutable) do not ever use the immutable keyword in any context 
as things just break down.

If you have had more success with a immutable with types 
containing mutable aliasing and can share your success story that 
would be great.

As D is growing are there any out there yet offering support 
services?

Thanks,
Dan
Oct 16 2013
next sibling parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 10/16/2013 10:23 AM, Daniel Davidson wrote:


 On Wednesday, 2 October 2013 at 13:09:34 UTC, Daniel Davidson wrote:



 guidelines):

 1. If a variable is never mutated, make it const, not immutable.
 2. Make the parameter reference to immutable if that is how you will
 use it anyway. It is fine to ask a favor from the caller.





 After trying for several days to use immutable with types containing
 some mutable aliasing I have come to the conclusion that maybe rule
 number one should be:

 If you have a type that has now or may ever have in the future any
 mutable aliasing (e.g. inclusion of T[] or T1[T1] where Ts are mutable)
 do not ever use the immutable keyword in any context as things just
 break down.


I think this topic should be carried to the main newsgroup already. I am 
convinced that this is a language issue. :-/


 Thanks,
 Dan


Ali
Oct 16 2013
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Oct 16, 2013 at 07:23:24PM +0200, Daniel Davidson wrote:

 On Wednesday, 2 October 2013 at 13:09:34 UTC, Daniel Davidson wrote:


[...]

Maybe it is a philosophical question, but where does immutability
really come from? Is it an aspect of some piece of data or is it a
promise that function will not change it? Or is it a requirement
by a function that data passed not be changed by anyone else?




I think it helps to realize that D's const system is different from
C++'s.

Immutable means the data will never change, ever. It means you can put
that data in read-only memory, maybe burned into a ROM chip or something
like that.

Const means *you* can't change the data, but somebody else may be able
to.

Therefore:


[...]

 After trying for several days to use immutable with types containing
 some mutable aliasing I have come to the conclusion that maybe rule
 number one should be:


If you have mutable aliasing, that means the data cannot be immutable.
Use const.



 If you have a type that has now or may ever have in the future any
 mutable aliasing (e.g. inclusion of T[] or T1[T1] where Ts are
 mutable) do not ever use the immutable keyword in any context as
 things just break down.


Yes, because immutable means nothing, no one, can change the data after
it's constructed, ever. If you want mutable aliasing, what you want is
const, not immutable.



 If you have had more success with a immutable with types containing
 mutable aliasing and can share your success story that would be
 great.


[...]

Maybe it's helpful to understand how D's const system works. The
following diagram may help (please excuse the ASCII graphics):

	       const
	      /     \
	mutable     immutable

What this means is that const subsumes mutable and immutable. A mutable
type can be implicitly converted to a const type (the receiver of the
const can't modify the data, which is fine since the code holding the
mutable reference can still mutate it), and so can immutable (immutable
cannot be modified, ever, and const doesn't let you modify it either, so
it's OK to make a const reference to immutable data). However, you
cannot implicitly convert between mutable and immutable, unless you're
copying the data by value.

So if you have immutable data and want to make changes, you have to
first make a copy of the data, then mutate it at will.

What's the use of immutable, you ask? Immutable makes hard guarantees
about the non-changeability of some piece of data. This makes it useful
for implementing strings -- in fact, the 'string' type in D is just an
alias for immutable(char)[]. You can take substrings (slices) of any
given string freely, and be assured that your copy of the (sub)string
will never unexpectedly change its value from somewhere else in the
code. This saves the need for a lot of copying, which can be costly.

One interesting subtlety here is that 'string' is immutable(char)[], but
not immutable(char[]). The latter would mean that the string itself can
never be changed -- you couldn't assign to it, you couldn't append to
it, etc., which would make strings a lot less useful than they are. But
by making strings a *mutable* array of *immutable* chars, you allow the
string to be appended to, substring'd, etc., all while guaranteeing that
the underlying bytes themselves will never change. So you can have the
best of both worlds: you can append to strings, take substrings, assign
strings to each other, etc., yet at the same time be assured that the
list of intermediate substrings you stored somewhere during the process
will continue to retain the values you assigned to them, because the
underlying bytes they point to are immutable, and therefore guaranteed
never to change.


T

-- 
Just because you survived after you did it, doesn't mean it wasn't stupid!
Oct 16 2013
next sibling parent reply "Daniel Davidson" <nospam spam.com> writes:
On Wednesday, 16 October 2013 at 17:55:14 UTC, H. S. Teoh wrote:

 On Wed, Oct 16, 2013 at 07:23:24PM +0200, Daniel Davidson wrote:

 On Wednesday, 2 October 2013 at 13:09:34 UTC, Daniel Davidson 
 wrote:


 [...]

Maybe it is a philosophical question, but where does 
immutability
really come from? Is it an aspect of some piece of data or is 
it a
promise that function will not change it? Or is it a 
requirement
by a function that data passed not be changed by anyone else?




 I think it helps to realize that D's const system is different 
 from
 C++'s.

 Immutable means the data will never change, ever. It means you 
 can put
 that data in read-only memory, maybe burned into a ROM chip or 
 something
 like that.

 Const means *you* can't change the data, but somebody else may 
 be able
 to.

 Therefore:


 [...]

 After trying for several days to use immutable with types 
 containing
 some mutable aliasing I have come to the conclusion that maybe 
 rule
 number one should be:


 If you have mutable aliasing, that means the data cannot be 
 immutable.
 Use const.



 If you have a type that has now or may ever have in the future 
 any
 mutable aliasing (e.g. inclusion of T[] or T1[T1] where Ts are
 mutable) do not ever use the immutable keyword in any context 
 as
 things just break down.


 Yes, because immutable means nothing, no one, can change the 
 data after
 it's constructed, ever. If you want mutable aliasing, what you 
 want is
 const, not immutable.


I think the term "mutable aliasing" and "what you want" don't 
work together. "mutable aliasing" is not about context of usage 
or what you want, it is a compile time attribute of a struct. I 
want to use associative arrays in composition because I think 
they are the way I view and use my data. `struct T { 
string[string] i; }` has mutable aliasing, like it or not. So, it 
is not so much that I want mutable aliasing - in fact I fear it. 
But once you use AAs it is a fact of life.


 If you have had more success with a immutable with types 
 containing
 mutable aliasing and can share your success story that would be
 great.


 [...]

 Maybe it's helpful to understand how D's const system works. The
 following diagram may help (please excuse the ASCII graphics):

 	       const
 	      /     \
 	mutable     immutable

 What this means is that const subsumes mutable and immutable. A 
 mutable
 type can be implicitly converted to a const type (the receiver 
 of the
 const can't modify the data, which is fine since the code 
 holding the
 mutable reference can still mutate it), and so can immutable 
 (immutable
 cannot be modified, ever, and const doesn't let you modify it 
 either, so
 it's OK to make a const reference to immutable data). However, 
 you
 cannot implicitly convert between mutable and immutable, unless 
 you're
 copying the data by value.


yes - it requires transitive deep copy.


 So if you have immutable data and want to make changes, you 
 have to
 first make a copy of the data, then mutate it at will.

 What's the use of immutable, you ask? Immutable makes hard 
 guarantees
 about the non-changeability of some piece of data. This makes 
 it useful
 for implementing strings -- in fact, the 'string' type in D is 
 just an
 alias for immutable(char)[]. You can take substrings (slices) 
 of any
 given string freely, and be assured that your copy of the 
 (sub)string
 will never unexpectedly change its value from somewhere else in 
 the
 code. This saves the need for a lot of copying, which can be 
 costly.


I agree that string behaves as you say. I don't quite agree that 
immutable alone is the reason it does. I think it is an byproduct 
of the way immutable(T)[] is implemented. It is an implementation 
detail and relying on that could lead to bad deduction. We 
covered that here in this thread:
http://forum.dlang.org/post/jfjudswamyxlttgsdwva forum.dlang.org



 One interesting subtlety here is that 'string' is 
 immutable(char)[], but
 not immutable(char[]). The latter would mean that the string 
 itself can
 never be changed -- you couldn't assign to it, you couldn't 
 append to
 it, etc., which would make strings a lot less useful than they 
 are. But
 by making strings a *mutable* array of *immutable* chars, you 
 allow the
 string to be appended to, substring'd, etc., all while 
 guaranteeing that
 the underlying bytes themselves will never change. So you can 
 have the
 best of both worlds: you can append to strings, take 
 substrings, assign
 strings to each other, etc., yet at the same time be assured 
 that the
 list of intermediate substrings you stored somewhere during the 
 process
 will continue to retain the values you assigned to them, 
 because the
 underlying bytes they point to are immutable, and therefore 
 guaranteed
 never to change.


 T


Agreed with the description of the behavior. But disagree on why. 
It works that way because T[] is modeled as contiguous memory and 
the api associated with slice of type immutable(T)[] means there 
is no unsafe sharing. So, `struct T { string[string] i; }` and 
`struct T { immutable(S)[string] }` do not have the same 
properties because they have a different layout model.

Is the suggestion here: use immutable in any composition context 
where you have slices and you want to not "worry about mutable 
aliasing". So, do like string does: any case where you have T[] 
as a member, prefer immutable(T)[] since then you don't have to 
worry about sharing. Well, not sure that works in general because 
the T in string is char and has no mutable aliasing itself. 
Suppose T itself has mutable aliasing, then what? Or, suppose it 
is a struct with no mutable aliasing *now*. Who's to say it won't 
change.

So, where does all this leave us w.r.t. a good set of guidelines?
Oct 16 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Oct 16, 2013 at 08:49:51PM +0200, Daniel Davidson wrote:

 On Wednesday, 16 October 2013 at 17:55:14 UTC, H. S. Teoh wrote:

On Wed, Oct 16, 2013 at 07:23:24PM +0200, Daniel Davidson wrote:




[...]

If you have a type that has now or may ever have in the future any
mutable aliasing (e.g. inclusion of T[] or T1[T1] where Ts are
mutable) do not ever use the immutable keyword in any context as
things just break down.


Yes, because immutable means nothing, no one, can change the data
after it's constructed, ever. If you want mutable aliasing, what you
want is const, not immutable.


 
 I think the term "mutable aliasing" and "what you want" don't work
 together. "mutable aliasing" is not about context of usage or what
 you want, it is a compile time attribute of a struct. I want to use
 associative arrays in composition because I think they are the way I
 view and use my data. `struct T { string[string] i; }` has mutable
 aliasing, like it or not. So, it is not so much that I want mutable
 aliasing - in fact I fear it. But once you use AAs it is a fact of
 life.


[...]

 Is the suggestion here: use immutable in any composition context
 where you have slices and you want to not "worry about mutable
 aliasing". So, do like string does: any case where you have T[] as a
 member, prefer immutable(T)[] since then you don't have to worry
 about sharing. Well, not sure that works in general because the T in
 string is char and has no mutable aliasing itself. Suppose T itself
 has mutable aliasing, then what? Or, suppose it is a struct with no
 mutable aliasing *now*. Who's to say it won't change.
 
 So, where does all this leave us w.r.t. a good set of guidelines?


Sorry, I kinda barged into this conversation, so I'm not sure what
exactly you're trying to achieve here. What kind of composition contexts
do you have in mind? Maybe you could help me understand what you're
trying to do?

Keeping in mind that AA's, as they are currently implemented, leaves a
lot of room for improvement, to say the least. So you might be running
into some AA-related issues, not the type system proper.


T

-- 
Creativity is not an excuse for sloppiness.
Oct 16 2013
parent "Daniel Davidson" <nospam spam.com> writes:
On Wednesday, 16 October 2013 at 19:01:59 UTC, H. S. Teoh wrote:

 On Wed, Oct 16, 2013 at 08:49:51PM +0200, Daniel Davidson wrote:

 On Wednesday, 16 October 2013 at 17:55:14 UTC, H. S. Teoh 
 wrote:

On Wed, Oct 16, 2013 at 07:23:24PM +0200, Daniel Davidson 
wrote:




 [...]

If you have a type that has now or may ever have in the 
future any
mutable aliasing (e.g. inclusion of T[] or T1[T1] where Ts 
are
mutable) do not ever use the immutable keyword in any 
context as
things just break down.


Yes, because immutable means nothing, no one, can change the 
data
after it's constructed, ever. If you want mutable aliasing, 
what you
want is const, not immutable.


 
 I think the term "mutable aliasing" and "what you want" don't 
 work
 together. "mutable aliasing" is not about context of usage or 
 what
 you want, it is a compile time attribute of a struct. I want 
 to use
 associative arrays in composition because I think they are the 
 way I
 view and use my data. `struct T { string[string] i; }` has 
 mutable
 aliasing, like it or not. So, it is not so much that I want 
 mutable
 aliasing - in fact I fear it. But once you use AAs it is a 
 fact of
 life.


 [...]

 Is the suggestion here: use immutable in any composition 
 context
 where you have slices and you want to not "worry about mutable
 aliasing". So, do like string does: any case where you have 
 T[] as a
 member, prefer immutable(T)[] since then you don't have to 
 worry
 about sharing. Well, not sure that works in general because 
 the T in
 string is char and has no mutable aliasing itself. Suppose T 
 itself
 has mutable aliasing, then what? Or, suppose it is a struct 
 with no
 mutable aliasing *now*. Who's to say it won't change.
 
 So, where does all this leave us w.r.t. a good set of 
 guidelines?


 Sorry, I kinda barged into this conversation, so I'm not sure 
 what
 exactly you're trying to achieve here. What kind of composition 
 contexts
 do you have in mind? Maybe you could help me understand what 
 you're
 trying to do?


Thanks. I appreciate any help I can get. Here is a small sample: 
http://pastebin.com/TeiQ9DYa

Other samples have at least 5 levels of composition. I'm 
generating the structure from json schema so I'm looking for 
something that scales and is immune to changes in deeply nested 
members.


 Keeping in mind that AA's, as they are currently implemented, 
 leaves a
 lot of room for improvement, to say the least. So you might be 
 running
 into some AA-related issues, not the type system proper.


I understand. It is a shame. If the data source is rich json data 
- I think AAs are a necessity.
Oct 16 2013
prev sibling parent reply "qznc" <qznc web.de> writes:
On Wednesday, 16 October 2013 at 17:55:14 UTC, H. S. Teoh wrote:

 Maybe it's helpful to understand how D's const system works. The
 following diagram may help (please excuse the ASCII graphics):

 	       const
 	      /     \
 	mutable     immutable


I think people in this thread know how const works, but some 
think it is broken. Scenario is this:

Library code:

struct Foo { int x; }

User code:

Foo f;
immutable f2 = f;

This works, even though the library writer might not have 
anticipated that someone makes Foo immutable. However, now the 
library writer obliviously releases a new version of the library, 
which extends it like this:

struct Foo {
   int x;
   private int[] history;
}

Unfortunately, now the user code is broken due to the freshly 
introduced mutable aliasing. Personally, I think is fine. Upon 
compilation the user code gives a  error message and user 
developer can adapt to the code to the new library version. Some 
think the library writer should have a possibility to make this 
work.
Oct 16 2013
next sibling parent reply "Daniel Davidson" <nospam spam.com> writes:
On Wednesday, 16 October 2013 at 18:52:23 UTC, qznc wrote:

 On Wednesday, 16 October 2013 at 17:55:14 UTC, H. S. Teoh wrote:

 Maybe it's helpful to understand how D's const system works. 
 The
 following diagram may help (please excuse the ASCII graphics):

 	       const
 	      /     \
 	mutable     immutable


 I think people in this thread know how const works, but some 
 think it is broken. Scenario is this:


Thanks.


 Library code:

 struct Foo { int x; }

 User code:

 Foo f;
 immutable f2 = f;

 This works, even though the library writer might not have 
 anticipated that someone makes Foo immutable. However, now the 
 library writer obliviously releases a new version of the 
 library, which extends it like this:

 struct Foo {
   int x;
   private int[] history;
 }

 Unfortunately, now the user code is broken due to the freshly 
 introduced mutable aliasing. Personally, I think is fine. Upon 
 compilation the user code gives a  error message and user 
 developer can adapt to the code to the new library version. 
 Some think the library writer should have a possibility to make 
 this work.


I don't understand how it could be fine. As code grows it would 
lead to people not adding useful members like history just 
because of the huge repercussions.

struct User {
    immutable(Foo) foos;
}

How can I as a user adapt to that change? Before the change 
assignment worked equally well among all of Mutable, Immutable, 
Const. After that change any `foos ~= createFoo(...)` would 
require change. And it is not clear what the change would be.
Oct 16 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 16 October 2013 at 19:06:06 UTC, Daniel Davidson 
wrote:

 I don't understand how it could be fine. As code grows it would 
 lead to people not adding useful members like history just 
 because of the huge repercussions.

 struct User {
    immutable(Foo) foos;
 }

 How can I as a user adapt to that change? Before the change 
 assignment worked equally well among all of Mutable, Immutable, 
 Const. After that change any `foos ~= createFoo(...)` would 
 require change. And it is not clear what the change would be.


I think any usage of immutable with types/entities not initially 
designed for immutability is an potential mistake and in that 
sense it is good that change has broken the user code. Same goes 
for operating on immutable entity in generic code as if it is a 
value type without actually checking it via introspection.
Oct 16 2013
parent reply "Daniel Davidson" <nospam spam.com> writes:
On Wednesday, 16 October 2013 at 19:12:48 UTC, Dicebot wrote:

 On Wednesday, 16 October 2013 at 19:06:06 UTC, Daniel Davidson 
 wrote:

 I don't understand how it could be fine. As code grows it 
 would lead to people not adding useful members like history 
 just because of the huge repercussions.

 struct User {
   immutable(Foo) foos;
 }

 How can I as a user adapt to that change? Before the change 
 assignment worked equally well among all of Mutable, 
 Immutable, Const. After that change any `foos ~= 
 createFoo(...)` would require change. And it is not clear what 
 the change would be.


 I think any usage of immutable with types/entities not 
 initially designed for immutability is an potential mistake and 
 in that sense it is good that change has broken the user code. 
 Same goes for operating on immutable entity in generic code as 
 if it is a value type without actually checking it via 
 introspection.


I don't disagree. Now, what does it mean to "initially design for 
immutability" and what are the guidelines. I thought that was 
kind of what we were talking about here. How to effectively use 
const and immutable. If the rule is - don't use immutable unless 
you have immutability in mind for your design, ok. Maybe take it 
a step further...

How about this - show a good use of immutable in any context 
where mutable aliasing (e.g. AAs) are in the mix. If the response 
is there are none, it is just too hairy of a proposition then 
something is wrong.

One general idea that was presented by Ali is that an immutable 
parameter means that not only are you guaranteeing that you will 
not change your data, but also that no one else, even in another 
thread will. That sounds appealing. After all, who doesn't want 
the data they are using in a function to not change from 
underneath them? Suppose you have highly structured, deeply 
nested reference data you read from a nosql DB. Surely there is 
opportunity and some benefit to use immutable? The result of the 
query is just a read only data source. But then that data will be 
consumed - presumably by functions with either immutable or const 
parameters. If all signatures use const then when can you benefit 
from the fact that the data is really immutable (by the time it 
gets to a function with const parm the fact that it really was/is 
immutable is lost.
Oct 16 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Oct 16, 2013 at 09:45:09PM +0200, Daniel Davidson wrote:

 On Wednesday, 16 October 2013 at 19:12:48 UTC, Dicebot wrote:


[...]

I think any usage of immutable with types/entities not initially
designed for immutability is an potential mistake and in that
sense it is good that change has broken the user code. Same goes
for operating on immutable entity in generic code as if it is a
value type without actually checking it via introspection.


 
 I don't disagree. Now, what does it mean to "initially design for
 immutability" and what are the guidelines. I thought that was kind
 of what we were talking about here. How to effectively use const and
 immutable.


I'd say that user code should use const, not immutable (unless the
library provides a way of constructing immutable instances of the type,
of course), because immutable makes assumptions about implementation
details, which should not be known unless you break encapsulation. As a
user of a properly encapsulated type, I can't make any guarantees about
its immutability; the best I can do is to promise I don't change it
myself -- i.e., const.

Immutable should be used in library code to provide strong guarantees to
the user: since you're the one responsible for implementing the type,
you're in the position to make guarantees about its uniqueness (and
hence, immutability).


[...]

 One general idea that was presented by Ali is that an immutable
 parameter means that not only are you guaranteeing that you will not
 change your data, but also that no one else, even in another thread
 will. That sounds appealing. After all, who doesn't want the data
 they are using in a function to not change from underneath them?


I'm actually wary of this view, to be honest. An immutable parameter
means you expect your *caller* to provide you with a value that cannot
be changed, not by you, nor by anybody else, ever. Sure, it's nice to
have, but that imposes a rather high bar on your callers. They have to
be responsible to guarantee that whatever they hand to you cannot be
changed by anything or anyone else, at any time. If they can do this,
then great; if not, they won't be able to call your function.



 Suppose you have highly structured, deeply nested reference data you
 read from a nosql DB. Surely there is opportunity and some benefit
 to use immutable? The result of the query is just a read only data
 source. But then that data will be consumed - presumably by
 functions with either immutable or const parameters. If all
 signatures use const then when can you benefit from the fact that
 the data is really immutable (by the time it gets to a function with
 const parm the fact that it really was/is immutable is lost.


I'm of the view that code should only require the minimum of assumptions
it needs to actually work. If your code can work with mutable types,
then let it take a mutable (unqualified) type. If your code works
without modifying input data, then let it take const. Only if your code
absolutely will not work correctly unless the data is guaranteed to
never change, ever, by anyone, should it take immutable.

I'm not sure what "benefits" you get from requiring immutable when the
code doesn't really need to depend on immutability. It just makes the
code harder to use (you have to make sure whatever you pass to it is
immutable, and sometimes that's not easy to guarantee, and would require
a lot of copying). Immutable only benefits you when the code *can't*
work correctly unless the data is guaranteed never to change, ever. Hash
table keys come to mind -- if you compute the hash value of the key and
use that to determine which slot to put the data into, it would be very
bad if somebody else mutated that key via a mutable reference after the
fact -- now your AA is broken because the hash value no longer matches
the key.  By requiring an immutable key, you ensure that this never
happens.

Note that in D, everything is thread-local by default unless explicitly
made shared, so using immutable to guarantee other threads won't mutate
the data isn't really necessary.


T

-- 
What do you mean the Internet isn't filled with subliminal messages? What about
all those buttons marked "submit"??
Oct 16 2013
next sibling parent reply "qznc" <qznc web.de> writes:
On Wednesday, 16 October 2013 at 20:33:23 UTC, H. S. Teoh wrote:

 I'm of the view that code should only require the minimum of 
 assumptions
 it needs to actually work. If your code can work with mutable 
 types,
 then let it take a mutable (unqualified) type. If your code 
 works
 without modifying input data, then let it take const. Only if 
 your code
 absolutely will not work correctly unless the data is 
 guaranteed to
 never change, ever, by anyone, should it take immutable.


Sounds reasonable. Which one of the following would you prefer?

foo1(const T t) {
   takes_immutable(t.idup);
}

foo2(immutable(T) t) {
   takes_immutable(t);
}

I'd say foo2, which puts the burden of the copy on the caller. 
However, if the caller already has an immutable value an 
unecessary copy is avoided.
Oct 16 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Oct 17, 2013 at 08:56:08AM +0200, qznc wrote:

 On Wednesday, 16 October 2013 at 20:33:23 UTC, H. S. Teoh wrote:

I'm of the view that code should only require the minimum of
assumptions it needs to actually work. If your code can work with
mutable types, then let it take a mutable (unqualified) type. If your
code works without modifying input data, then let it take const. Only
if your code absolutely will not work correctly unless the data is
guaranteed to never change, ever, by anyone, should it take
immutable.


 
 Sounds reasonable. Which one of the following would you prefer?
 
 foo1(const T t) {
   takes_immutable(t.idup);
 }
 
 foo2(immutable(T) t) {
   takes_immutable(t);
 }
 
 I'd say foo2, which puts the burden of the copy on the caller.
 However, if the caller already has an immutable value an unecessary
 copy is avoided.


Well, that's a bit too generic to really decide one way or the other.
What do foo1 and foo2 do? I don't think it's fair to say one or the
other is superior without more context. It depends on what you're trying
to achieve.


T

-- 
Too many people have open minds but closed eyes.
Oct 17 2013
prev sibling next sibling parent reply "qznc" <qznc web.de> writes:
On Wednesday, 16 October 2013 at 20:33:23 UTC, H. S. Teoh wrote:

 I'm of the view that code should only require the minimum of 
 assumptions
 it needs to actually work. If your code can work with mutable 
 types,
 then let it take a mutable (unqualified) type. If your code 
 works
 without modifying input data, then let it take const. Only if 
 your code
 absolutely will not work correctly unless the data is 
 guaranteed to
 never change, ever, by anyone, should it take immutable.


What about functions, which are not thread-safe?

auto percentageDistribution(const int[] percentages) {
   [...]
   assert (sum == 100);
   [...]
}

Not a realistic example, but it should do. This function checks 
that the sum of all percentages equals 100, but a concurrent 
modification might break this. You could require an immutable 
array to avoid that. Alternatively, the absence of "shared" could 
be considered enough to express this.
Oct 17 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Oct 17, 2013 at 09:08:16AM +0200, qznc wrote:

 On Wednesday, 16 October 2013 at 20:33:23 UTC, H. S. Teoh wrote:

I'm of the view that code should only require the minimum of
assumptions it needs to actually work. If your code can work with
mutable types, then let it take a mutable (unqualified) type. If your
code works without modifying input data, then let it take const. Only
if your code absolutely will not work correctly unless the data is
guaranteed to never change, ever, by anyone, should it take
immutable.


 
 What about functions, which are not thread-safe?
 
 auto percentageDistribution(const int[] percentages) {
   [...]
   assert (sum == 100);
   [...]
 }
 
 Not a realistic example, but it should do. This function checks that
 the sum of all percentages equals 100, but a concurrent modification
 might break this. You could require an immutable array to avoid that.
 Alternatively, the absence of "shared" could be considered enough to
 express this.


In D, variables are thread-local by default, and only shared if
explicitly stated so. If this function took shared(const(int)[]), or you
pass it something that's __gshared, then you might have a point. But in
normal D code I'd recommend against doing that.

Currently, D has some issues in the area of shared, requiring a cast to
strip shared from the type. So basically the onus is on the caller to
ensure that when shared is stripped from the type it's actually safe
from concurrent modification. In which case the above code should be OK
as-is.


T

-- 
Guns don't kill people. Bullets do.
Oct 17 2013
prev sibling parent "Daniel Davidson" <nospam spam.com> writes:
On Wednesday, 16 October 2013 at 20:33:23 UTC, H. S. Teoh wrote:

 On Wed, Oct 16, 2013 at 09:45:09PM +0200, Daniel Davidson wrote:

 On Wednesday, 16 October 2013 at 19:12:48 UTC, Dicebot wrote:


 [...]

I think any usage of immutable with types/entities not 
initially
designed for immutability is an potential mistake and in that
sense it is good that change has broken the user code. Same 
goes
for operating on immutable entity in generic code as if it is 
a
value type without actually checking it via introspection.


 
 I don't disagree. Now, what does it mean to "initially design 
 for
 immutability" and what are the guidelines. I thought that was 
 kind
 of what we were talking about here. How to effectively use 
 const and
 immutable.




Thanks - this is most helpful.


 I'd say that user code should use const, not immutable (unless 
 the
 library provides a way of constructing immutable instances of 
 the type,
 of course), because immutable makes assumptions about 
 implementation
 details, which should not be known unless you break 
 encapsulation. As a
 user of a properly encapsulated type, I can't make any 
 guarantees about
 its immutability; the best I can do is to promise I don't 
 change it
 myself -- i.e., const.


The distinction between user code and the rest (non-user code?) 
is not enough for me to get clear guidance. I am writing user 
code, the vast majority will be in libraries because I find that 
helpful. I would hate to have a separate set of rules for my code 
and the code I use. We should strive for guidelines that work in 
general - as in this is a good way to do D.

But leaving "user code" aside, I like the argument and the vote 
for const over immutable.



 Immutable should be used in library code to provide strong 
 guarantees to
 the user: since you're the one responsible for implementing the 
 type,
 you're in the position to make guarantees about its uniqueness 
 (and
 hence, immutability).


I don't see the benefit of separating usage of types, usually via 
functions where mutability guarantees adorn the types, and 
implementation guarantees in a world with turtles all the way 
down. Maybe if those guarantees are totally hidden from user code 
via encapsulation then the impact of immutable and the 
difficulties of dealing with it are lessened. But then is the 
immutable guarantee for the user or just the developer of the 
encapsulated non-user code?

How about this... here is a plea for ideas on a good use of 
immutable for a type with potentially mutable aliasing. String I 
think is a great use of immutable but char will likely never have 
aliasing introduced to it. At this stage I want the information 
for educational purposes, because based on this thread and my 
experimentation - Ali's presentation guidelines aside - I am 
about to abandon immutable altogether. The fight is too hard and 
my skills too weakened.


 [...]

 One general idea that was presented by Ali is that an immutable
 parameter means that not only are you guaranteeing that you 
 will not
 change your data, but also that no one else, even in another 
 thread
 will. That sounds appealing. After all, who doesn't want the 
 data
 they are using in a function to not change from underneath 
 them?


 I'm actually wary of this view, to be honest. An immutable 
 parameter
 means you expect your *caller* to provide you with a value that 
 cannot
 be changed, not by you, nor by anybody else, ever. Sure, it's 
 nice to
 have, but that imposes a rather high bar on your callers. They 
 have to
 be responsible to guarantee that whatever they hand to you 
 cannot be
 changed by anything or anyone else, at any time. If they can do 
 this,
 then great; if not, they won't be able to call your function.


I now agree with you on this, especially since it goes with my 
new guideline of don't use immutable.


 Suppose you have highly structured, deeply nested reference 
 data you
 read from a nosql DB. Surely there is opportunity and some 
 benefit
 to use immutable? The result of the query is just a read only 
 data
 source. But then that data will be consumed - presumably by
 functions with either immutable or const parameters. If all
 signatures use const then when can you benefit from the fact 
 that
 the data is really immutable (by the time it gets to a 
 function with
 const parm the fact that it really was/is immutable is lost.


 I'm of the view that code should only require the minimum of 
 assumptions
 it needs to actually work. If your code can work with mutable 
 types,
 then let it take a mutable (unqualified) type. If your code 
 works
 without modifying input data, then let it take const. Only if 
 your code
 absolutely will not work correctly unless the data is 
 guaranteed to
 never change, ever, by anyone, should it take immutable.


I don't have the instincts yet to really buy this. Perhaps a 
specific example would be helpful.

I think most of the time `foo(ref const(T) t)` is written such 
that it is assumed t is never changed during the span of foo and 
the compiler helps. While it is possible to, in a single thread 
call out to code that also has a handle to shared state so t 
could be accidentally or purposely modified elsewhere, it is 
probably a rare design goal.

Take the query example. A big mongo db query result comes back 
and is deserialized into a large web of json like data - lists, 
dictionaries at many levels. Is that a good reason then for 
`foo(ref immutable(QueryResult) qr)`. Surely while you are 
working on query result you don't want it to change. I am 
ambivalent here because immutable sounds appealing. Any iteration 
over a hash in the data set must be immutable. I think this is 
the expectation most have with all const or immutable types taken 
in a function. Sticking with const just makes life easier.



 I'm not sure what "benefits" you get from requiring immutable 
 when the
 code doesn't really need to depend on immutability. It just 
 makes the
 code harder to use (you have to make sure whatever you pass to 
 it is
 immutable, and sometimes that's not easy to guarantee, and 
 would require
 a lot of copying). Immutable only benefits you when the code 
 *can't*
 work correctly unless the data is guaranteed never to change, 
 ever. Hash
 table keys come to mind -- if you compute the hash value of the 
 key and
 use that to determine which slot to put the data into, it would 
 be very
 bad if somebody else mutated that key via a mutable reference 
 after the
 fact -- now your AA is broken because the hash value no longer 
 matches
 the key.  By requiring an immutable key, you ensure that this 
 never
 happens.


I like the AA example - since it sounds like a good use for 
immutable.


 Note that in D, everything is thread-local by default unless 
 explicitly
 made shared, so using immutable to guarantee other threads 
 won't mutate
 the data isn't really necessary.


 T
Oct 17 2013
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Oct 16, 2013 at 09:06:05PM +0200, Daniel Davidson wrote:

 On Wednesday, 16 October 2013 at 18:52:23 UTC, qznc wrote:


[...]

Library code:

struct Foo { int x; }

User code:

Foo f;
immutable f2 = f;

This works, even though the library writer might not have
anticipated that someone makes Foo immutable. However, now the
library writer obliviously releases a new version of the library,
which extends it like this:

struct Foo {
  int x;
  private int[] history;
}

Unfortunately, now the user code is broken due to the freshly
introduced mutable aliasing. Personally, I think is fine. Upon
compilation the user code gives a  error message and user
developer can adapt to the code to the new library version. Some
think the library writer should have a possibility to make this
work.


 
 I don't understand how it could be fine. As code grows it would lead
 to people not adding useful members like history just because of the
 huge repercussions.
 
 struct User {
    immutable(Foo) foos;
 }
 
 How can I as a user adapt to that change? Before the change
 assignment worked equally well among all of Mutable, Immutable,
 Const. After that change any `foos ~= createFoo(...)` would require
 change. And it is not clear what the change would be.


The root of the problem is reliance on assignment between mutable /
immutable / const. This reliance breaks encapsulation because you're
making an assumption about the assignability of a presumedly opaque
library type to immutable / const. In D, immutable is *physical*
immutability, not logical immutability; by writing immutable(Foo) you're
saying that you wish to have physically-immutable instances of Foo.
However, whether this is possible depends on the implementation details
of Foo, which, if Foo is supposed to be an opaque type, breaks
encapsulation. Without knowing how Foo is implemented (and user code
shouldn't know that), you can't reliably go around and claim Foo can be
made immutable from a mutable instance. The fact that you're relying on
Foo being implicitly convertible to immutable(Foo) means you're already
depending on implementation details of Foo, and should be prepared to
change code when Foo's implementation changes.

If you want to say that User cannot modify the Foo's it contains, you
should use const rather than immutable. It is safe to use const because
anything is implicitly convertible to const, so it doesn't introduce any
reliance upon implementational details of Foo.

If you insist on being able to append to immutable(Foo)[], then you'll
need a createFoo method that returns immutable instances of Foo:

	struct User {
		immutable(Foo)[] foos;
	}
	immutable(Foo) createFoo(...) { ... }

	User u;
	u.foos ~= createFoo(...); // now this works

The problem with this, of course, is that it unnecessarily restricts
createFoo(): if you want *mutable* instances of Foo, then you can't use
this version of createFoo(), but have to create another function that
probably does exactly the same thing. So an alternative solution is to
use Phobos' assumeUnique template:

	struct User {
		immutable(Foo)[] foos;
	}
	Foo createFoo(...) { ... }

	User u;
	u.foos ~= assumeUnique(createFoo(...));

The assumeUnique template basically does a cast from mutable to
immutable, but explicitly documents the purpose of this cast in the
code. It places the onus on the user to ensure that the Foo returned by
createFoo is actually unique. If not, you break the type system and the
immutability guarantee may no longer hold.

To illustrate why adding mutable aliases to Foo *should* break code,
consider this:

	/* This is what Foo looked like before:
	struct OriginalFoo {
		int x;
	}
	*/

	/* This is what Foo looks like now */
	struct Foo {
		int x;
		private int[] history;
		void changeHistory() { history[0]++; }
	}

	Foo createFoo(int x) {
		Foo f;
		f.x = x;
		f.history = [1];
	}

	Foo f = createFoo();

	immutable(Foo) g = f; // doesn't compile, but suppose it does
	f.changeHistory();    // oops, g.history has mutated, so it's
	                      // *not* immutable after all

That's why assigning f to g must be made illegal, since it breaks
immutability guarantees. OTOH, if you absolutely have to do it, you can
document your intent thus:

	Foo f = createFoo();
	immutable(Foo) g = assumeUnique(f);

	// Now if you use f to mutate g, it's your own problem: you
	// claimed that g was unique but actually it isn't. So it's your
	// own fault when your supposedly-immutable Foo mutates.
	// If you *don't* do stupid things, OTOH, this lets your code
	// continue to work when the library writer decides to change
	// Foo's implementation to contain mutable aliases.


T

-- 
Do not reason with the unreasonable; you lose by definition.
Oct 16 2013
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Oct 16, 2013 at 08:52:22PM +0200, qznc wrote:

 On Wednesday, 16 October 2013 at 17:55:14 UTC, H. S. Teoh wrote:

Maybe it's helpful to understand how D's const system works. The
following diagram may help (please excuse the ASCII graphics):

	       const
	      /     \
	mutable     immutable


 
 I think people in this thread know how const works, but some think
 it is broken. Scenario is this:
 
 Library code:
 
 struct Foo { int x; }
 
 User code:
 
 Foo f;
 immutable f2 = f;
 
 This works, even though the library writer might not have
 anticipated that someone makes Foo immutable. However, now the
 library writer obliviously releases a new version of the library,
 which extends it like this:
 
 struct Foo {
   int x;
   private int[] history;
 }
 
 Unfortunately, now the user code is broken due to the freshly
 introduced mutable aliasing. Personally, I think is fine. Upon
 compilation the user code gives a  error message and user developer
 can adapt to the code to the new library version. Some think the
 library writer should have a possibility to make this work.


Well, I think here the type system is working as advertised. Since the
original version of Foo has no mutable aliasing, implicit conversion to
immutable is OK (you're making a new binary copy of the data). But in
the second case, it will obviously violate immutability guarantees,
because in D, immutable is transitive, so immutable(Foo) in the second
case is equivalent to:

	struct Foo {
		int x;
		private immutable(int[]) history;
	}

You can't implicitly convert int[] to immutable(int[]) because whoever
holds the original int[] reference can use it to change the data, which
breaks the immutable guarantee of immutable(int[]).

The only way this could work is if you made a copy of the int[]. So the
library writer would have to provide a method for creating an immutable
copy of the struct (like an .idup method or something).

You may argue that this is bad because now user code is broken and you
need to rewrite it to use .idup, but I'd argue that relying on implicit
conversion to immutable already introduces a dependency on
implementation details of Foo, which should be avoided in the first
place if you want to have a clean encapsulation.

I mean, given a Foo type exported by a library, if that type is meant to
be an opaque type, then my code should make no assumptions about whether
it can implicitly convert to immutable. I should rather require the
library writer to provide an .idup method for creating an immutable
instance of the struct than to blindly write the code to implicitly
convert to immutable (thus introducing a hidden reliance on how Foo is
implemented) and then have it broken later when the library writer
changes its implementation.

OTOH, if it's not an opaque type, then it's no surprise that changing
its implementation should also require changing the code (that depends
on its implementation details).

So I don't see anything wrong with this particular scenario. What other
scenarios were being considered?


T

-- 
He who does not appreciate the beauty of language is not worthy to bemoan its
flaws.
Oct 16 2013