www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Array literals MUST be immutable.

reply Don <nospam nospam.com> writes:
This is for me the last remaining D2 issue.
I've mentioned this several times before. Mutable array 'literals' are 
wrong on so many levels. Here are some of the issues, listed in 
decreasing importance.

(1) The language has no syntax for immutable array literals.
   [1,2,3] is mutable. You can created an immutable literal using the 
awful 'static const' syntax, but that only works for named literals.

(2) Concurrency issues make (1) even more important. It ought to 
possible to pass an array (defined at compile time) as a message.

(3) Performance of the existing array 'literals' is absolutely 
appalling. I doubled the speed of my entire app by changing ONE array 
declaration from 'immutable [] xxx' into 'static const [] xxx'!!!
Right now, if an array literal appears in your code, chances are it's a 
performance bug. It's deceptive that there's a hidden heap allocation 
happening in every array literal. It's got no business being in a 
systems language IMHO.

(4) It is unintuitive. "hello" is immutable, but ['h','e','l','l','o'] 
is not???

(5) It's trivial (a one-liner) to make a function to create a mutable 
array literal. We don't need it in the language.

Mutable arrays literals are common in toy examples, but they've got very 
little real-world relevance. By contrast, immutable array literals are 
used in lookup tables all the time. They are extremely common. We need 
to be able to express them.

This is the only remaining thing in D2 which I'm certain is a big 
mistake. Of course there's ugliness in various places, but it's possible 
to ignore them or work around them. This is a primitive, 
bread-and-butter issue with no workaround.
Feb 17 2010
next sibling parent BCS <none anon.com> writes:
Hello Don,


 This is the only remaining thing in D2 which I'm certain is a big
 mistake. Of course there's ugliness in various places, but it's
 possible to ignore them or work around them. This is a primitive,
 bread-and-butter issue with no workaround.


Sounds like you have a good point. Vote++;
 
-- 
... <IXOYE><
Feb 17 2010
prev sibling next sibling parent "Simen kjaeraas" <simen.kjaras gmail.com> writes:
Don <nospam nospam.com> wrote:


 This is for me the last remaining D2 issue.
 I've mentioned this several times before. Mutable array 'literals' are  
 wrong on so many levels. Here are some of the issues, listed in  
 decreasing importance.

 (1) The language has no syntax for immutable array literals.
    [1,2,3] is mutable. You can created an immutable literal using the  
 awful 'static const' syntax, but that only works for named literals.

 (2) Concurrency issues make (1) even more important. It ought to  
 possible to pass an array (defined at compile time) as a message.

 (3) Performance of the existing array 'literals' is absolutely  
 appalling. I doubled the speed of my entire app by changing ONE array  
 declaration from 'immutable [] xxx' into 'static const [] xxx'!!!
 Right now, if an array literal appears in your code, chances are it's a  
 performance bug. It's deceptive that there's a hidden heap allocation  
 happening in every array literal. It's got no business being in a  
 systems language IMHO.

 (4) It is unintuitive. "hello" is immutable, but ['h','e','l','l','o']  
 is not???

 (5) It's trivial (a one-liner) to make a function to create a mutable  
 array literal. We don't need it in the language.

 Mutable arrays literals are common in toy examples, but they've got very  
 little real-world relevance. By contrast, immutable array literals are  
 used in lookup tables all the time. They are extremely common. We need  
 to be able to express them.

 This is the only remaining thing in D2 which I'm certain is a big  
 mistake. Of course there's ugliness in various places, but it's possible  
 to ignore them or work around them. This is a primitive,  
 bread-and-butter issue with no workaround.


Absolutely. This needs fixin'.

-- 
Simen
Feb 17 2010
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Wed, 17 Feb 2010 11:47:00 +0300, Don <nospam nospam.com> wrote:


 This is for me the last remaining D2 issue.
 I've mentioned this several times before. Mutable array 'literals' are  
 wrong on so many levels. Here are some of the issues, listed in  
 decreasing importance.

 (1) The language has no syntax for immutable array literals.
    [1,2,3] is mutable. You can created an immutable literal using the  
 awful 'static const' syntax, but that only works for named literals.

 (2) Concurrency issues make (1) even more important. It ought to  
 possible to pass an array (defined at compile time) as a message.

 (3) Performance of the existing array 'literals' is absolutely  
 appalling. I doubled the speed of my entire app by changing ONE array  
 declaration from 'immutable [] xxx' into 'static const [] xxx'!!!
 Right now, if an array literal appears in your code, chances are it's a  
 performance bug. It's deceptive that there's a hidden heap allocation  
 happening in every array literal. It's got no business being in a  
 systems language IMHO.

 (4) It is unintuitive. "hello" is immutable, but ['h','e','l','l','o']  
 is not???

 (5) It's trivial (a one-liner) to make a function to create a mutable  
 array literal. We don't need it in the language.

 Mutable arrays literals are common in toy examples, but they've got very  
 little real-world relevance. By contrast, immutable array literals are  
 used in lookup tables all the time. They are extremely common. We need  
 to be able to express them.

 This is the only remaining thing in D2 which I'm certain is a big  
 mistake. Of course there's ugliness in various places, but it's possible  
 to ignore them or work around them. This is a primitive,  
 bread-and-butter issue with no workaround.


Absolutely agree!
Feb 17 2010
prev sibling next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Don wrote:

 This is for me the last remaining D2 issue.


That would make it difficult to do things like:

   int*[] foo(int *p)
   {
     return [p, p + 1];
   }

as all the elements of the literal would also have to be immutable. I 
think you've made a good case, but there is also this issue.
Feb 17 2010
next sibling parent "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
Walter Bright wrote:

 Don wrote:

 This is for me the last remaining D2 issue.


 
 That would make it difficult to do things like:
 
   int*[] foo(int *p)
   {
     return [p, p + 1];
   }
 
 as all the elements of the literal would also have to be immutable. I 
 think you've made a good case, but there is also this issue.



Note his point (5).  It's trivial to add a function for creating mutable 
arrays to std.array.

   import std.array;

   int*[] foo(int* p)
   {
       return array(p, p+1);
   }

I think Don makes an excellent case. This needs to be fixed.

-Lars
Feb 17 2010
prev sibling next sibling parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Wed, 17 Feb 2010 12:17:09 +0300, Walter Bright  
<newshound1 digitalmars.com> wrote:


 Don wrote:

 This is for me the last remaining D2 issue.


 That would make it difficult to do things like:

    int*[] foo(int *p)
    {
      return [p, p + 1];
    }

 as all the elements of the literal would also have to be immutable. I  
 think you've made a good case, but there is also this issue.


IIRC, it was discussed before and the following solution was suggested:

T[] toArray(T)(T[] values...)
{
     return values.dup; // not sure why dup is needed here, but DMD2.039  
complains about escaping reference
}

int main()
{
     int* p = null;
     int*[] array = toArray(p, p + 1);
     assert(array.length == 2);
     assert(array[0] is p);
     assert(array[1] is (p + 1));

     return 0;
}

In my opinion, this is a minor use case that has graceful solution, in no  
way it is a show stopper.
Feb 17 2010
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Denis Koroskin (2korden gmail.com)'s article

 IIRC, it was discussed before and the following solution was suggested:
 T[] toArray(T)(T[] values...)
 {
      return values.dup; // not sure why dup is needed here, but DMD2.039
 complains about escaping reference


When using this kind of variadics, an implementation is allowed to allocate the
array on the stack for performance.  Therefore, you were trying to escape
references to a stack-allocated array.
Feb 17 2010
parent "Denis Koroskin" <2korden gmail.com> writes:
On Wed, 17 Feb 2010 16:58:35 +0300, dsimcha <dsimcha yahoo.com> wrote:


 == Quote from Denis Koroskin (2korden gmail.com)'s article

 IIRC, it was discussed before and the following solution was suggested:
 T[] toArray(T)(T[] values...)
 {
      return values.dup; // not sure why dup is needed here, but DMD2.039
 complains about escaping reference


 When using this kind of variadics, an implementation is allowed to  
 allocate the
 array on the stack for performance.  Therefore, you were trying to escape
 references to a stack-allocated array.


Oh, great! Didn't know the fact, thanks!
Feb 17 2010
prev sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-02-17 04:17:09 -0500, Walter Bright <newshound1 digitalmars.com> said:


 Don wrote:

 This is for me the last remaining D2 issue.


 
 That would make it difficult to do things like:
 
    int*[] foo(int *p)
    {
      return [p, p + 1];
    }
 
 as all the elements of the literal would also have to be immutable. I 
 think you've made a good case, but there is also this issue.


One thing I like to do with array literals is use them for appending 
several elements at once:

	array ~= [1, a+1, b+1];

This way I avoid multiple array appends so it should be faster, but if 
that temporary array literal gets allocated on the heap then it's 
rather counterproductive for me to use array literals for this.

Could cases like this, where the array never escape, be allocated on the stack?

Just like local variables get allocated on the heap when they escape 
through a closure, array literals could be allocated on the heap when 
they escape through a return or some other mean. They could also be 
part of the program's static data when the literal expression is 
constant and assigned to an immutable(T)[].

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Feb 17 2010
parent Walter Bright <newshound1 digitalmars.com> writes:
Michel Fortin wrote:

 Could cases like this, where the array never escape, be allocated on the 
 stack?


Yes, the compiler can do that.
Feb 17 2010
prev sibling next sibling parent reply grauzone <none example.net> writes:
IMHO array literals should be static arrays, which are value types. No 
issues with heap allocation or immutability requirements.

Also, even immutable values can change at runtime:

void foo(immutable int x) {
	auto array = [x];
}

array would have needed to be heap allocated even if they are changed to 
be immutable.

Don wrote:

 (2) Concurrency issues make (1) even more important. It ought to 
 possible to pass an array (defined at compile time) as a message.


This sounds very special. Any example where you'd pass an array known at 
compile time as message?


 (3) Performance of the existing array 'literals' is absolutely 
 appalling. I doubled the speed of my entire app by changing ONE array 
 declaration from 'immutable [] xxx' into 'static const [] xxx'!!!
 Right now, if an array literal appears in your code, chances are it's a 
 performance bug. It's deceptive that there's a hidden heap allocation 
 happening in every array literal. It's got no business being in a 
 systems language IMHO.


Delegates also can have hidden allocation (basically, you have to guess 
the compiler's guess whether a delegate is a real closure). There are 
several language elements that may or may not trigger additional memory 
allocations (like setting .length or the ~= operator).

And the underlying problem is the bad GC that is torturing us all, isn't it?
Feb 17 2010
parent reply Don <nospam nospam.com> writes:
grauzone wrote:

 IMHO array literals should be static arrays, which are value types. No 
 issues with heap allocation or immutability requirements.


That still wouldn't solve the performance problems.


 Don wrote:

 (2) Concurrency issues make (1) even more important. It ought to 
 possible to pass an array (defined at compile time) as a message.


 
 This sounds very special. Any example where you'd pass an array known at 
 compile time as message?


There's one in Andrei's concurrency chapter in TDPL!


 (3) Performance of the existing array 'literals' is absolutely 
 appalling. I doubled the speed of my entire app by changing ONE array 
 declaration from 'immutable [] xxx' into 'static const [] xxx'!!!
 Right now, if an array literal appears in your code, chances are it's 
 a performance bug. It's deceptive that there's a hidden heap 
 allocation happening in every array literal. It's got no business 
 being in a systems language IMHO.


 
 Delegates also can have hidden allocation (basically, you have to guess 
 the compiler's guess whether a delegate is a real closure). There are 
 several language elements that may or may not trigger additional memory 
 allocations (like setting .length or the ~= operator).
 
 And the underlying problem is the bad GC that is torturing us all, isn't 
 it?


No, this one has nothing to do with the GC. The problem is that there's 
no way of saying 'this array literal is not a variable'.
Feb 17 2010
parent grauzone <none example.net> writes:
Don wrote:

 grauzone wrote:

 IMHO array literals should be static arrays, which are value types. No 
 issues with heap allocation or immutability requirements.


 
 That still wouldn't solve the performance problems.


You mean even for a static array on the stack, the time to construct it 
would impose a performance problem? If that's the case, I'd say the 
optimizer should be able to tell that the array data actually is 
constant, and compile the code as if the array was immutable. I assume 
the array being a value type on the stack makes it much easier for the 
optimizer than a heap allocated array with unknown length.

Is that impossible?
Feb 17 2010
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Don wrote:

 This is for me the last remaining D2 issue.
 I've mentioned this several times before. Mutable array 'literals' are 
 wrong on so many levels. Here are some of the issues, listed in 
 decreasing importance.


[snip]

I agree. How do you see we should change the language to fix things?

Andrei
Feb 17 2010
parent reply Michael Rynn <michaelrynn optusnet.com.au> writes:
On Wed, 17 Feb 2010 07:49:09 -0600, Andrei Alexandrescu wrote:


 Don wrote:

 This is for me the last remaining D2 issue. I've mentioned this several
 times before. Mutable array 'literals' are wrong on so many levels.
 Here are some of the issues, listed in decreasing importance.


 [snip]
 
 I agree. How do you see we should change the language to fix things?
 
 Andrei


I am trying to improve my understanding here by reasoning some of it out a
bit, and trying to figure out what is everyone elses probably implicitly
understands.

Start looking at DPL C heritage, what is the behaviour of array literals
in C?  What does a good C compiler do?

Whenever we put something like this in code,

Elem* = [ Piece of literal data ]

The compiler has created a representation of the data which of these? 1.
pointer to protected read - only memory? (Probably would declare const) 2.
returned a pointer to a writeable static memory location in the code- data
image, which is start of the array data. If so then the data can be
changed but not extended in size without reallocation.

3. allocated a copy of the original data, location known only to compiler,
to heap or stack allocated memory block, and assigned to the pointer
variable.

How does this vary with static and stack context?

I would like to (wishfully) think that what happens is that it looks like
the code text in my editor.  The value of the literal in my source code
after being assigned is not destroyed by further manipulation of Elem*. It
still exists in the original source, so it should still exist in the
binary image, and can be recalled again.  So thats what I would define as
the "immutable source".

This happens now for string type, they are source immutable. Changes can
be only done to a mutable copy.  I have got used to that.



So what would really break if this was true of all array literals? How
hard is it to do in the current D2 compiler? How does it affect the
runtime typeinfo system? (I am still getting used to that).

Has anyone tried this in an experimental version of D2?
  
I suppose we can always write code that can copy from immutable source
where necessary? Is it possible to be able make mutable copies more
easily?

Is there anything that is really awkward in D2 working with immutable
string type, that cannot be done without resorting to mutable character
literals?   If not, then the principle can be applied to all "immutable
source".

But for those who would like mutable static memory initialised for some ,
might there be a way of telling the compiler that we wish to abuse the
memory image?

char[] = cast(mutable(char)[]) "I can overwrite this initialization string
for all time"; // points to original binary image

char[] = "I can overwrite this source string".dup;

The example given :-
   int*[] foo(int *p)
   {
     return [p, p + 1];
   }

Why would you make such a function?
Why not make it locally without the function call overhead?

int *p = &something;
...
int*[] mypointers = [p, p+1];  // this should not be a literal, but memory
allocated like a if its on the stack filled in dynamically at point of 
execution, and not immutable?

p, p+1 are the initialisation values.

int*[] mypointers = [&static1, &static2];  // this looks like a literal, 
known at compile time. but could also be memory filled in dynamically at 
point of execution.  If mypointers is static then it is immutable?.

How does it work for struct?


Michael Rynn...
Feb 17 2010
parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-02-17 20:07:20 -0500, Michael Rynn <michaelrynn optusnet.com.au> said:


 Is there anything that is really awkward in D2 working with immutable
 string type, that cannot be done without resorting to mutable character
 literals?   If not, then the principle can be applied to all "immutable
 source".


Well, array literals are not always constants. Take this for instance:

	int[] array = [x, y, z];

This array literal does not have an "immutable source" as you call it.

Should it really require a ".dup" to get a mutable array? How efficient 
would it be?

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Feb 17 2010
parent reply Don <nospam nospam.com> writes:
Michel Fortin wrote:

 On 2010-02-17 20:07:20 -0500, Michael Rynn <michaelrynn optusnet.com.au> 
 said:
 

 Is there anything that is really awkward in D2 working with immutable
 string type, that cannot be done without resorting to mutable character
 literals?   If not, then the principle can be applied to all "immutable
 source".


 
 Well, array literals are not always constants. Take this for instance:
 
     int[] array = [x, y, z];
 
 This array literal does not have an "immutable source" as you call it.


This is the issue. The syntax sugar you get from not requiring an 
"immutable source" comes at a very high price.


 Should it really require a ".dup" to get a mutable array? How efficient 
 would it be?


What it does at the moment is insert a hidden .dup at all times. And the 
performance is terrible. Really, really terrible. Roughly 100 times 
worse than you expect.
At the moment, [1, 2, 3] is basically transformed into something like 
makeArray(1,2,3);
Feb 18 2010
parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-02-18 03:10:47 -0500, Don <nospam nospam.com> said:


 Well, array literals are not always constants. Take this for instance:
 
     int[] array = [x, y, z];
 
 This array literal does not have an "immutable source" as you call it.


 
 This is the issue. The syntax sugar you get from not requiring an 
 "immutable source" comes at a very high price.
 

 Should it really require a ".dup" to get a mutable array? How efficient 
 would it be?


 
 What it does at the moment is insert a hidden .dup at all times. And 
 the performance is terrible. Really, really terrible. Roughly 100 times 
 worse than you expect.
 At the moment, [1, 2, 3] is basically transformed into something like 
 makeArray(1,2,3);


I agree the performance is terrible because of the often unnecessary 
heap allocation. But assigning a literal to an immutable array should 
be easy to special-case in the compiler so that it doesn't do an 
unnecessary copy. In fact, forcing literals to be assigned to immutable 
arrays only is the same special case, except it disallows the other 
case where it needs to do a copy (assigning to mutable).

Non-constant array literals can also be optimized in some situations: 
when they don't escape the function's scope, they can be allocated on 
the stack. Better optimization could take advantage of that.

Also, they're quite useful. How would you write the code above without them?

	int[] array;
	array.length = 3;
	array[0] = x;
	array[1] = y;
	array[2] = z;

Not very attractive isn't it? And it wouldn't work when assigning to an 
array of immutable(int)[]. Also, I expect this'll make it harder for 
the compiler to optimize and allocate on the stack when it can.


-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Feb 18 2010
parent reply Don <nospam nospam.com> writes:
Michel Fortin wrote:

 On 2010-02-18 03:10:47 -0500, Don <nospam nospam.com> said:
 

 Well, array literals are not always constants. Take this for instance:

     int[] array = [x, y, z];

 This array literal does not have an "immutable source" as you call it.


 This is the issue. The syntax sugar you get from not requiring an 
 "immutable source" comes at a very high price.


 Should it really require a ".dup" to get a mutable array? How 
 efficient would it be?


 What it does at the moment is insert a hidden .dup at all times. And 
 the performance is terrible. Really, really terrible. Roughly 100 
 times worse than you expect.
 At the moment, [1, 2, 3] is basically transformed into something like 
 makeArray(1,2,3);


 
 I agree the performance is terrible because of the often unnecessary 
 heap allocation.


It's not just the heap allocation. The values are also recalculated.


 But assigning a literal to an immutable array should be 
 easy to special-case in the compiler so that it doesn't do an 
 unnecessary copy.


Yes, but that doesn't work in the general case. It cannot be done for 
anonymous literals.
Also you can't force an array literal to be CTFEd.


 Also, they're quite useful. How would you write the code above without 
 them?
 
     int[] array;
     array.length = 3;
     array[0] = x;
     array[1] = y;
     array[2] = z;


Including the library implementation:

T[] toArray(T)(T[] values...)
{
     return values.dup;
}

int[] array = toArray(x,y,z);

Mutable array literals achieve almost nothing.
Feb 18 2010
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 18 Feb 2010 08:00:55 -0500, Don <nospam nospam.com> wrote:


 Including the library implementation:

 T[] toArray(T)(T[] values...)
 {
      return values.dup;
 }

 int[] array = toArray(x,y,z);

 Mutable array literals achieve almost nothing.


I don't think this will work:

toArray(1, 2L, 3.0)

I'm not saying that this is a reason not to consider the change, but it  
does note a difference.  You can always simply do toArray!double(...) to  
get around the limitation.

-Steve
Feb 18 2010
parent "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
Steven Schveighoffer wrote:

 On Thu, 18 Feb 2010 08:00:55 -0500, Don <nospam nospam.com> wrote:
 

 Including the library implementation:

 T[] toArray(T)(T[] values...)
 {
      return values.dup;
 }

 int[] array = toArray(x,y,z);

 Mutable array literals achieve almost nothing.


 
 I don't think this will work:
 
 toArray(1, 2L, 3.0)
 
 I'm not saying that this is a reason not to consider the change, but it 
 does note a difference.  You can always simply do toArray!double(...) to 
 get around the limitation.
 
 -Steve



import std.traits: CommonType;

CommonType!(T)[] toArray(T...)(T elements) { ... }


-Lars
Feb 18 2010
prev sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-02-18 08:00:55 -0500, Don <nospam nospam.com> said:


 I agree the performance is terrible because of the often unnecessary 
 heap allocation.


 
 It's not just the heap allocation. The values are also recalculated.


Well, in the case of a constant literal expression they don't need to 
be recalculated, they could exist in the static data segment. If a 
constant array literal is assigned to an immutable array they don't 
even need to be copied. I trust the compiler will eventually do that 
(and sooner the better).



 But assigning a literal to an immutable array should be easy to 
 special-case in the compiler so that it doesn't do an unnecessary copy.


 
 Yes, but that doesn't work in the general case. It cannot be done for 
 anonymous literals.


I'm not sure what you mean by "anonymous literals". Aren't all literals 
anonymous?



 Also you can't force an array literal to be CTFEd.


I think that's a more general problem with CTFE. But you can force CTFE 
using an enum:

	enum value = ctfeFunction();

Wrap that in a template if you want more convenience:

	template ctfe(alias ctfeValue) {
		enum ctfe = ctfeValue;
	}

	auto variable = ctfe!(ctfeFunction());



 Also, they're quite useful. How would you write the code above without them?
 
     int[] array;
     array.length = 3;
     array[0] = x;
     array[1] = y;
     array[2] = z;


 
 Including the library implementation:
 
 T[] toArray(T)(T[] values...)
 {
      return values.dup;
 }
 
 int[] array = toArray(x,y,z);
 
 Mutable array literals achieve almost nothing.


But now, how can the compiler optimize this and stack-allocate your 
array when it detect doesn't escape your function? You're explicitly 
telling it to duplicate it, even the compiler could decide it's not 
necessary. (I know stack-allocation isn't implemented, but Walter said 
it could be done.)

Consider this case:

	int a, b, c;
	int[] array;
	array ~= [a, b, c];
	array ~= toArray(a, b, c);

Does it make sense to heap-allocate the mutable array? Hardly. With the 
literal, the compiler is free to optimize away the heap allocation, not 
so with toArray.


-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Feb 18 2010
next sibling parent Don <nospam nospam.com> writes:
Michel Fortin wrote:


 But assigning a literal to an immutable array should be easy to 
 special-case in the compiler so that it doesn't do an unnecessary copy.


 Yes, but that doesn't work in the general case. It cannot be done for 
 anonymous literals.


 
 I'm not sure what you mean by "anonymous literals". Aren't all literals 
 anonymous?


No. If they are function parameters, they are anonymous.
foo([1,2,3]);

As opposed to:

immutable int [] x = [1,2,3];

where they have a name, a storage class, and a chance to put an 
'immutable' in.


 Also you can't force an array literal to be CTFEd.


 
 I think that's a more general problem with CTFE. But you can force CTFE 
 using an enum:
 
     enum value = ctfeFunction();


Unfortunately, that doesn't work.
(1) Enum arrays aren't stored in the executable. You can't index them.
(currently you can due, to a compiler bug, and it generates wrong code).
(2) Using 'enum' doesn't work for the anonymous case.

 
 Wrap that in a template if you want more convenience:
 
     template ctfe(alias ctfeValue) {
         enum ctfe = ctfeValue;
     }


Expressions can be aliased. So I doubt that will work.


     auto variable = ctfe!(ctfeFunction());
 
 

 Also, they're quite useful. How would you write the code above 
 without them?

     int[] array;
     array.length = 3;
     array[0] = x;
     array[1] = y;
     array[2] = z;


 Including the library implementation:

 T[] toArray(T)(T[] values...)
 {
      return values.dup;
 }

 int[] array = toArray(x,y,z);

 Mutable array literals achieve almost nothing.


 
 But now, how can the compiler optimize this and stack-allocate your 
 array when it detect doesn't escape your function?


That's *exactly* the way it works now. Really, there's no difference.
The toArray function is just a single return statement, so it gets 
inlined perfectly. And .dup is a compiler built-in. So it's easy for the 
compiler. The compiler sees:
int [] array = [x, y, z].dup; (where [x, y, z] is the existing mutable 
array syntax).
Feb 18 2010
prev sibling parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Thu, 18 Feb 2010 18:42:16 +0300, Michel Fortin
<michel.fortin michelf.com> wrote:


 On 2010-02-18 08:00:55 -0500, Don <nospam nospam.com> said:


 I agree the performance is terrible because of the often unnecessary  
 heap allocation.


  It's not just the heap allocation. The values are also recalculated.


 Well, in the case of a constant literal expression they don't need to be  
 recalculated, they could exist in the static data segment. If a constant  
 array literal is assigned to an immutable array they don't even need to  
 be copied. I trust the compiler will eventually do that (and sooner the  
 better).



 But assigning a literal to an immutable array should be easy to  
 special-case in the compiler so that it doesn't do an unnecessary copy.


  Yes, but that doesn't work in the general case. It cannot be done for  
 anonymous literals.


 I'm not sure what you mean by "anonymous literals". Aren't all literals  
 anonymous?



 Also you can't force an array literal to be CTFEd.


 I think that's a more general problem with CTFE. But you can force CTFE  
 using an enum:

 	enum value = ctfeFunction();

 Wrap that in a template if you want more convenience:

 	template ctfe(alias ctfeValue) {
 		enum ctfe = ctfeValue;
 	}

 	auto variable = ctfe!(ctfeFunction());



 Also, they're quite useful. How would you write the code above without  
 them?
      int[] array;
     array.length = 3;
     array[0] = x;
     array[1] = y;
     array[2] = z;


  Including the library implementation:
  T[] toArray(T)(T[] values...)
 {
      return values.dup;
 }
  int[] array = toArray(x,y,z);
  Mutable array literals achieve almost nothing.


 But now, how can the compiler optimize this and stack-allocate your  
 array when it detect doesn't escape your function? You're explicitly  
 telling it to duplicate it, even the compiler could decide it's not  
 necessary. (I know stack-allocation isn't implemented, but Walter said  
 it could be done.)

 Consider this case:

 	int a, b, c;
 	int[] array;
 	array ~= [a, b, c];
 	array ~= toArray(a, b, c);

 Does it make sense to heap-allocate the mutable array? Hardly. With the  
 literal, the compiler is free to optimize away the heap allocation, not  
 so with toArray.


[a, b, c] could result in a static array. Then there wouldn't even be a  
need for toArray, just use more natural int[] arr = [a, b, c].dup; syntax
Feb 18 2010
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 18 Feb 2010 16:17:50 -0500, Denis Koroskin <2korden gmail.com>  
wrote:


 On Thu, 18 Feb 2010 18:42:16 +0300, Michel Fortin
 <michel.fortin michelf.com> wrote:


 Consider this case:

 	int a, b, c;
 	int[] array;
 	array ~= [a, b, c];
 	array ~= toArray(a, b, c);

 Does it make sense to heap-allocate the mutable array? Hardly. With the  
 literal, the compiler is free to optimize away the heap allocation, not  
 so with toArray.


 [a, b, c] could result in a static array. Then there wouldn't even be a  
 need for toArray, just use more natural int[] arr = [a, b, c].dup; syntax


That would be bad,  T[] is implicitly casted from T[N].  Consider that you  
could easily escape stack data using this.  In other words, the type  
system would allow the assignment without the dup.

-Steve
Feb 18 2010
parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Fri, 19 Feb 2010 00:46:05 +0300, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:


 On Thu, 18 Feb 2010 16:17:50 -0500, Denis Koroskin <2korden gmail.com>  
 wrote:


 On Thu, 18 Feb 2010 18:42:16 +0300, Michel Fortin
 <michel.fortin michelf.com> wrote:


 Consider this case:

 	int a, b, c;
 	int[] array;
 	array ~= [a, b, c];
 	array ~= toArray(a, b, c);

 Does it make sense to heap-allocate the mutable array? Hardly. With  
 the literal, the compiler is free to optimize away the heap  
 allocation, not so with toArray.


 [a, b, c] could result in a static array. Then there wouldn't even be a  
 need for toArray, just use more natural int[] arr = [a, b, c].dup;  
 syntax


 That would be bad,  T[] is implicitly casted from T[N].  Consider that  
 you could easily escape stack data using this.  In other words, the type  
 system would allow the assignment without the dup.

 -Steve


I don't think so. First of all, it is consistent with current behavior:

int[] foo()
{
	int[3] staticArray = [0, 1, 2];
	int[] dynamicArray = staticArray;

	return dynamicArray;
}

int[] a = foo();
writeln(a); // prints garbage

Second, it's very useful to prevent unnecessary heap allocations.

I would prefer static arrays not to cast to dynamic ones implicitly, but  
using opSlice syntax instead:

int[] arr = [0, 1, 2]; // error
int[] arr = [0, 1, 2][]; // fine, use on your own risk
Feb 18 2010
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 18 Feb 2010 20:47:16 -0500, Denis Koroskin <2korden gmail.com>  
wrote:


 On Fri, 19 Feb 2010 00:46:05 +0300, Steven Schveighoffer  
 <schveiguy yahoo.com> wrote:

 That would be bad,  T[] is implicitly casted from T[N].  Consider that  
 you could easily escape stack data using this.  In other words, the  
 type system would allow the assignment without the dup.

 -Steve


 I don't think so. First of all, it is consistent with current behavior:

 int[] foo()
 {
 	int[3] staticArray = [0, 1, 2];
 	int[] dynamicArray = staticArray;

 	return dynamicArray;
 }


Here is the case I'd say looks bad to me:

int[] foo()
{
    int[] dynamicArray = [0, 1, 2];
    return dynamicArray;
}

If [0, 1, 2] is a static array, then it escapes its scope and corruption  
ensues.  The implicit static array is the problem, not the assigning of a  
dynamic array from a static array.  The literal doesn't look like it's a  
static array.


 Second, it's very useful to prevent unnecessary heap allocations.


This is already available by using a static array explicitly.  The  
question is, what should be the default?  I think making it immutable and  
non-static is the right choice because it can always escape scope, and  
does not require an allocation.  From there you can do either a static  
array if you know it won't escape scope, or a dynamic array (via dup) if  
you need to modify it and it will escape scope.


 I would prefer static arrays not to cast to dynamic ones implicitly, but  
 using opSlice syntax instead:

 int[] arr = [0, 1, 2]; // error
 int[] arr = [0, 1, 2][]; // fine, use on your own risk


This is just an annoyance error.  What will happen is people will  
constantly just use the brackets to shut up the compiler without thinking  
about what it really means.

-Steve
Feb 19 2010
next sibling parent reply grauzone <none example.net> writes:
Steven Schveighoffer wrote:

 On Thu, 18 Feb 2010 20:47:16 -0500, Denis Koroskin <2korden gmail.com> 
 wrote:
 

 On Fri, 19 Feb 2010 00:46:05 +0300, Steven Schveighoffer 
 <schveiguy yahoo.com> wrote:

 That would be bad,  T[] is implicitly casted from T[N].  Consider 
 that you could easily escape stack data using this.  In other words, 
 the type system would allow the assignment without the dup.

 -Steve


 I don't think so. First of all, it is consistent with current behavior:

 int[] foo()
 {
     int[3] staticArray = [0, 1, 2];
     int[] dynamicArray = staticArray;

     return dynamicArray;
 }


 
 Here is the case I'd say looks bad to me:
 
 int[] foo()
 {
    int[] dynamicArray = [0, 1, 2];
    return dynamicArray;
 }
 
 If [0, 1, 2] is a static array, then it escapes its scope and corruption 
 ensues.  The implicit static array is the problem, not the assigning of 
 a dynamic array from a static array.  The literal doesn't look like it's 
 a static array.


SafeD will take care of it. As the compiler has to support SafeD anyway, 
the same mechanisms can be used to create warnings or even errors in 
normal D code.

Don wants to make array literals immutable; that means you couldn't just 
assign them to int[] anyway. In both cases, you'd have to use .dup to 
use them as normal, mutable arrays.

Array literals being static arrays would be very handy in other places. 
Wouldn't it be nice if you could just use them for small fixed size 
vectors (wasn't that one of the reasons to make static arrays value 
types)? Then you could write assert([1, 2] + [3, 4] == [4, 6]). No silly 
struct vector "classes" needed.


 Second, it's very useful to prevent unnecessary heap allocations.


 
 This is already available by using a static array explicitly.  The 


Static arrays still suck a bit, although making them value types 
improved the situation.

Inside functions, this still allocates an array on the heap and copies 
it into the static array:

int[2] array = [1, 2];

I don't quite know why it can't just behaves as if "array" was declared 
on module level (or as, um, the other "static").


 question is, what should be the default?  I think making it immutable 
 and non-static is the right choice because it can always escape scope, 
 and does not require an allocation.  From there you can do either a 
 static array if you know it won't escape scope, or a dynamic array (via 
 dup) if you need to modify it and it will escape scope.


How would you "do" a static array from an immutable array?

As I understood, immutable array literals would still require heap 
allocations sometimes. Because even runtime dependent values can be 
immutable.

Actually, if array literals only worked for compile time values, array 
literals could be both immutable and static arrays allocated in the data 
segment.


 I would prefer static arrays not to cast to dynamic ones implicitly, 
 but using opSlice syntax instead:

 int[] arr = [0, 1, 2]; // error
 int[] arr = [0, 1, 2][]; // fine, use on your own risk




I agree. I wonder how static arrays will be handled in SafeD. Maybe just 
disallow using them as normal arrays? Require an explicit .dup? Anyway, 
it will be safe.


 This is just an annoyance error.  What will happen is people will 
 constantly just use the brackets to shut up the compiler without 
 thinking about what it really means.


Not really an argument, when array literals will be immutable, you could 
as well say:

//the cast is only there to shut up the stupid dmd!
int[] arr = cast(int[])([0, 1, 2]);


 -Steve
Feb 19 2010
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 19 Feb 2010 10:22:15 -0500, grauzone <none example.net> wrote:


 Steven Schveighoffer wrote:

 On Thu, 18 Feb 2010 20:47:16 -0500, Denis Koroskin <2korden gmail.com>  
 wrote:


 On Fri, 19 Feb 2010 00:46:05 +0300, Steven Schveighoffer  
 <schveiguy yahoo.com> wrote:

 That would be bad,  T[] is implicitly casted from T[N].  Consider  
 that you could easily escape stack data using this.  In other words,  
 the type system would allow the assignment without the dup.

 -Steve


 I don't think so. First of all, it is consistent with current behavior:

 int[] foo()
 {
     int[3] staticArray = [0, 1, 2];
     int[] dynamicArray = staticArray;

     return dynamicArray;
 }


  Here is the case I'd say looks bad to me:
  int[] foo()
 {
    int[] dynamicArray = [0, 1, 2];
    return dynamicArray;
 }
  If [0, 1, 2] is a static array, then it escapes its scope and  
 corruption ensues.  The implicit static array is the problem, not the  
 assigning of a dynamic array from a static array.  The literal doesn't  
 look like it's a static array.


 SafeD will take care of it. As the compiler has to support SafeD anyway,  
 the same mechanisms can be used to create warnings or even errors in  
 normal D code.


I'm not sure where Andrei will draw the line for SafeD.  SafeD still has  
to be UsefulD ;)

But I think slicing a static array makes the compiler lose all knowledge  
that it is a scoped entity.  Until scope (or something equivalent) is a  
proper type constructor, we will not have full escape analysis, just  
simple tricks and hacks that prevent certain errors.

The other option is to disallow slicing static arrays in SafeD, which IMO  
is a mistake, but I can see validity in the argument behind it.

But besides this, I want non-safe D still to be as safe as it can be!


 Don wants to make array literals immutable; that means you couldn't just  
 assign them to int[] anyway. In both cases, you'd have to use .dup to  
 use them as normal, mutable arrays.


That's fine, if I don't specify dup by accident, the compiler would  
complain.  In Denis' proposal, the compiler doesn't complain it just  
silently compiles.


 Array literals being static arrays would be very handy in other places.  
 Wouldn't it be nice if you could just use them for small fixed size  
 vectors (wasn't that one of the reasons to make static arrays value  
 types)? Then you could write assert([1, 2] + [3, 4] == [4, 6]). No silly  
 struct vector "classes" needed.


Does this work with static arrays?  I thought you needed to use the  
slicing notation.  In any case, using literals in array operations would  
be nifty.  I don't think you need to have array literals be statically  
sized to accomplish this.


 Second, it's very useful to prevent unnecessary heap allocations.


  This is already available by using a static array explicitly.  The


 Static arrays still suck a bit, although making them value types  
 improved the situation.

 Inside functions, this still allocates an array on the heap and copies  
 it into the static array:

 int[2] array = [1, 2];


The allocation goes away if the literal is immutable.  I think we all  
agree the current situation is sucky.


 I don't quite know why it can't just behaves as if "array" was declared  
 on module level (or as, um, the other "static").


Essentially, that is what an immutable array is.  The difference is in  
type inference.


 question is, what should be the default?  I think making it immutable  
 and non-static is the right choice because it can always escape scope,  
 and does not require an allocation.  From there you can do either a  
 static array if you know it won't escape scope, or a dynamic array (via  
 dup) if you need to modify it and it will escape scope.


 How would you "do" a static array from an immutable array?

 As I understood, immutable array literals would still require heap  
 allocations sometimes. Because even runtime dependent values can be  
 immutable.


Requiring array literals to be immutable prevents you from including  
runtime values.  This would work fine and not allocate:

int[3] arr = [1, 2, 3];

But your point is well taken.  I remember you brought it up before, and I  
argued a while with someone about it to no avail (can't remember what or  
with whom I was arguing :)

If array literals become immutable, being able to initialize static arrays  
easily from runtime values *without* heap allocation would be a very nice  
feature to have.  Perhaps you could use variables in an array literal only  
if they are used to initialize statically sized arrays.  This problem  
needs to be discussed if we are to get immutable array literals.


 I would prefer static arrays not to cast to dynamic ones implicitly,  
 but using opSlice syntax instead:

 int[] arr = [0, 1, 2]; // error
 int[] arr = [0, 1, 2][]; // fine, use on your own risk




 I agree. I wonder how static arrays will be handled in SafeD. Maybe just  
 disallow using them as normal arrays? Require an explicit .dup? Anyway,  
 it will be safe.


I have a feeling that safeD will disallow taking the address of a local,  
which includes slicing a static array, but IMO this would be a huge  
deterrent from using safeD.  Might as well call it slowD at that point :)


 This is just an annoyance error.  What will happen is people will  
 constantly just use the brackets to shut up the compiler without  
 thinking about what it really means.


 Not really an argument, when array literals will be immutable, you could  
 as well say:

 //the cast is only there to shut up the stupid dmd!
 int[] arr = cast(int[])([0, 1, 2]);


Yes, but a cast is a red flag.  A simple appending of brackets is not.  It  
is well established that casts are to let you do things that in most  
contexts are not a good idea.  Brackets are not the same.

Beside all these points, if array literals are static, then obtaining an  
immutable array literal requires a dup or a global.  I don't think that's  
a good strategy.  Another point is IFTI and auto type detection -- what is  
most useful as an automatic type for array literals?  I'd say immutable  
arrays are more useful because they can be easily converted to all other  
types of arrays, and it is consistent with string literals.  The only  
caveat as you brought up is using runtime values inside a literal to  
initialize a static array.  I think this can be special-cased by the  
compiler, but I'm not sure.

-Steve
Feb 19 2010
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Steven Schveighoffer wrote:

 On Fri, 19 Feb 2010 10:22:15 -0500, grauzone <none example.net> wrote:

 SafeD will take care of it. As the compiler has to support SafeD 
 anyway, the same mechanisms can be used to create warnings or even 
 errors in normal D code.


 
 I'm not sure where Andrei will draw the line for SafeD.  SafeD still has 
 to be UsefulD ;)


Good point, but safe means safe. It's can't be a hint. Perhaps the 
biggest crisis between Walter and me consumed when he mentioned he 
wanted to make "safe" just a hint and still disable bounds checking in 
release mode, safe or not. He was kind enough to back off, which was 
very nice of him - I see no way in which I could accept a compromise on 
that particular matter. (By the way, much of the very fruitful 
collaboration between us has been caused by our ability to work out good 
compromises even though we have sometimes different viewpoints.)

Statically-allocated arrays are not a difficult safety issue. We have 
two solutions - (a) have the compiler just disable conversions for 
stack-allocated T[N] to T[] in safe mode, (b) have the compiler perform 
its own flow analysis and transparently switch to dynamic allocation 
when needed, just like in the case of closures.


 But I think slicing a static array makes the compiler lose all knowledge 
 that it is a scoped entity.


I thought the same, but then I figured this works:

int[10] a;
int[5] b = a[0 .. 5];

Well it's off-topic, I know.

I agree that the current situation needs addressing.


Andrei
Feb 19 2010
prev sibling next sibling parent grauzone <none example.net> writes:
Steven Schveighoffer wrote:

 Don wants to make array literals immutable; that means you couldn't 
 just assign them to int[] anyway. In both cases, you'd have to use 
 .dup to use them as normal, mutable arrays.


 
 That's fine, if I don't specify dup by accident, the compiler would 
 complain.  In Denis' proposal, the compiler doesn't complain it just 
 silently compiles.


True enough, but you have this problem _anyway_, as Denis pointed out. 
And the solution will remove those safety problems for array literals as 
well.

I don't know if the compiler has enough magic to detect the escaping 
reference by itself, but it would be great if that worked.


 Array literals being static arrays would be very handy in other 
 places. Wouldn't it be nice if you could just use them for small fixed 
 size vectors (wasn't that one of the reasons to make static arrays 
 value types)? Then you could write assert([1, 2] + [3, 4] == [4, 6]). 
 No silly struct vector "classes" needed.


 
 Does this work with static arrays?  I thought you needed to use the 
 slicing notation.  In any case, using literals in array operations would 
 be nifty.  I don't think you need to have array literals be statically 
 sized to accomplish this.


Yes, right now you need []. I would hope some day it will work without.

Array operations on static arrays would enable the compiler to emit a 
single SSE instruction for an operation. With dynamic arrays, the 
compiler has to insert additional instructions for boundary checks, 
unpacking the array elements, etc.; and it couldn't just allocate the 
full vector in SSE registers either.


 I don't quite know why it can't just behaves as if "array" was 
 declared on module level (or as, um, the other "static").


 
 Essentially, that is what an immutable array is.  The difference is in 
 type inference.


Even today, the compiler could just do the right thing and allow 
"static" initializers even for stack allocated values. By the way, this 
is probably also the reason why nobody likes struct initializers. They 
work as static initializers only.


 If array literals become immutable, being able to initialize static 
 arrays easily from runtime values *without* heap allocation would be a 
 very nice feature to have.  Perhaps you could use variables in an array 
 literal only if they are used to initialize statically sized arrays.  
 This problem needs to be discussed if we are to get immutable array 
 literals.


Yes, but the compiler doesn't do that even today. Was it too hard to 
implement? Didn't Walter care enough? Or didn't he know of this special 
case?


 Beside all these points, if array literals are static, then obtaining an 
 immutable array literal requires a dup or a global.  I don't think 


The immutable array could be on the stack as well (it would have the 
type immutable(T[X]) for some integer X).


 that's a good strategy.  Another point is IFTI and auto type detection 
 -- what is most useful as an automatic type for array literals?  I'd say 
 immutable arrays are more useful because they can be easily converted to 
 all other types of arrays, and it is consistent with string literals.  


That's a good point.


 The only caveat as you brought up is using runtime values inside a 
 literal to initialize a static array.  I think this can be special-cased 
 by the compiler, but I'm not sure.


At least it wouldn't conflict with the semantics. But Don had this 
argument with efficiency and unobvious heal allocations. Immutable array 
literals don't solve this. Of course you could still force the arguments 
for array literals to be compile time values.
Feb 19 2010
prev sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-02-19 11:03:30 -0500, "Steven Schveighoffer" 
<schveiguy yahoo.com> said:


 //the cast is only there to shut up the stupid dmd!
 int[] arr = cast(int[])([0, 1, 2]);


 
 Yes, but a cast is a red flag.  A simple appending of brackets is not.  
 It  is well established that casts are to let you do things that in 
 most  contexts are not a good idea.  Brackets are not the same.


That's the theory. It doesn't quite work in practice because you 
sometime have to cast to convert from one type to another (float to 
int, base class to derived), or to make a literal of the type you want 
(especially with array literals). Many usages of cast are safe, many 
aren't. So seeing cast as a red flag doesn't really work, unfortunately.

Even SafeD allows cast, it just restricts it to safe usages.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Feb 19 2010
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 19 Feb 2010 12:59:36 -0500, Michel Fortin  
<michel.fortin michelf.com> wrote:


 On 2010-02-19 11:03:30 -0500, "Steven Schveighoffer"  
 <schveiguy yahoo.com> said:


 //the cast is only there to shut up the stupid dmd!
 int[] arr = cast(int[])([0, 1, 2]);


  Yes, but a cast is a red flag.  A simple appending of brackets is  
 not.  It  is well established that casts are to let you do things that  
 in most  contexts are not a good idea.  Brackets are not the same.


 That's the theory. It doesn't quite work in practice because you  
 sometime have to cast to convert from one type to another (float to int,  
 base class to derived), or to make a literal of the type you want  
 (especially with array literals). Many usages of cast are safe, many  
 aren't. So seeing cast as a red flag doesn't really work, unfortunately.

 Even SafeD allows cast, it just restricts it to safe usages.


There are some valid uses of casting, but cast is much more of a red flag  
than brackets.  Brackets are used everywhere.  Search your code for [] and  
I'm sure you'll find a lot more instances than casting.

But I think aside from this, dynamic casting should really be a separate  
function, I wouldn't mind if it's functionality moved into the library.

-Steve
Feb 19 2010
prev sibling parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Fri, 19 Feb 2010 17:51:26 +0300, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:


 The implicit static array is the problem, not the assigning of a dynamic  
 array from a static array.


Not quite true. Here is your example slightly modified:

int[] foo()
{
    int[3] staticArray;
    int[] dynamicArray = staticArray;
    return dynamicArray;
}

It doesn't involve any array literals, but the problem is still there.


 The literal doesn't look like it's a static array.


It really does to me. [a, b, c] is essentially 3 stack pushes and nothing  
else (which is the same as allocating stack space for a static array of 3  
elements).


 This is already available by using a static array explicitly.


How would you initialize such a static array with 3 given elements (a, b  
and c). I certainly wouldn't love verbosity of

int[3] array = void;
array[0] = a;
array[1] = b;
array[2] = c;


 The question is, what should be the default?  I think making it  
 immutable and non-static is the right choice because it can always  
 escape scope, and does not require an allocation.


You can't make [a, b, c] immutable unless all of the a, b and c are  
immutable (or primitive types). For example, what type would that be:

Object a = new Object();
Object b = new Object();
Object c = new Object();

auto array = [a, b, c];

It certainly can't be immutable(Object)[]. I believe it shouldn't be  
Object[] either (you can always make a dynamic array out of static but not  
vice versa). What bonuses would does Object[] provide over Object[3]?  
Object[] doesn't prevent but rather *encourages* escaping reference,  
because D returns static array by values:

// Your preferred way:
Object[] toArray(Object a, Object b, Object c)
{
     return [a, b, c]; // Bug, escaping reference to stack data
}

// My suggestion
Object[3] toArray(Object a, Object b, Object c)
{
     return [a, b, c]; // Correct
}


 I would prefer static arrays not to cast to dynamic ones implicitly,  
 but using opSlice syntax instead:

 int[] arr = [0, 1, 2]; // error
 int[] arr = [0, 1, 2][]; // fine, use on your own risk


 This is just an annoyance error.  What will happen is people will  
 constantly just use the brackets to shut up the compiler without  
 thinking about what it really means.


<sarcasm>Sure, just hide the bug without compiler worrying you about  
presence of one. </sarcasm>

You suggestion encourages unsafe by default, which is bad bad bad. It just  
shouldn't compile. People should just use .dup if they need a dynamic  
array unless they *really* know what they are doing, and use []. It should  
never happen in SafeD, and probably not even allowed.
Feb 19 2010
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 19 Feb 2010 10:32:29 -0500, Denis Koroskin <2korden gmail.com>  
wrote:


 On Fri, 19 Feb 2010 17:51:26 +0300, Steven Schveighoffer  
 <schveiguy yahoo.com> wrote:


 The implicit static array is the problem, not the assigning of a  
 dynamic array from a static array.


 Not quite true. Here is your example slightly modified:

 int[] foo()
 {
     int[3] staticArray;
     int[] dynamicArray = staticArray;
     return dynamicArray;
 }

 It doesn't involve any array literals, but the problem is still there.


Yes, but my point was that initializing a dynamic array with a literal  
should not implicitly promote unsafe code.  When someone uses a literal,  
they are expecting the type to match.  A literal should be the most useful  
type for the range of types it initializes.  I think immutable is better  
than a static because an implicit casting of a static to a dynamic array  
promotes memory corruption.  An implicit conversion from a dynamic to a  
static does not.


 The literal doesn't look like it's a static array.


 It really does to me. [a, b, c] is essentially 3 stack pushes and  
 nothing else (which is the same as allocating stack space for a static  
 array of 3 elements).


It looks like data to me, not stack data.

When you do:

int i = 1;

does the 1 look like stack data?

My point is, where it's stored is not implicit in its usage, so if it's  
stack data, it could be easily interpreted as not stack data.  Storing it  
in the immutable data segment and typing it as immutable is the safest  
option.


 This is already available by using a static array explicitly.


 How would you initialize such a static array with 3 given elements (a, b  
 and c). I certainly wouldn't love verbosity of

 int[3] array = void;
 array[0] = a;
 array[1] = b;
 array[2] = c;


What I meant was compile-time values could be stored in the array.  I  
agree storing variables in a stack array needs to be solved.  A possible  
solution is to allow variables in array literals only when being assigned  
to or used as a static array.

How do you solve this?

immutable(int)[] i = [1, 2, 3]; // add an idup?

The best thing here is to avoid a heap allocation, because immutable data  
made entirely from literals requiring a heap allocation is just as bad as  
the current situation.

With literals being immutable, going to static array is possible.  With  
literals being a static array, it is not possible.


 The question is, what should be the default?  I think making it  
 immutable and non-static is the right choice because it can always  
 escape scope, and does not require an allocation.


 You can't make [a, b, c] immutable unless all of the a, b and c are  
 immutable (or primitive types). For example, what type would that be:

 Object a = new Object();
 Object b = new Object();
 Object c = new Object();

 auto array = [a, b, c];


Yes, it's a problem that needs to be solved.  I'd say that literals that  
contain a runtime variable should be typed as static arrays, but I'm  
pretty sure Don doesn't like that.

I'm going to start a new thread about this.  I don't think this particular  
nuance is getting enough attention.


 It certainly can't be immutable(Object)[]. I believe it shouldn't be  
 Object[] either (you can always make a dynamic array out of static but  
 not vice versa). What bonuses would does Object[] provide over  
 Object[3]? Object[] doesn't prevent but rather *encourages* escaping  
 reference, because D returns static array by values:

 // Your preferred way:
 Object[] toArray(Object a, Object b, Object c)
 {
      return [a, b, c]; // Bug, escaping reference to stack data
 }


I didn't say I prefer it.  I meant it shouldn't be this easy, even in  
unsafe D.  To anticipate your next argument, I don't think requiring  
slicing brackets is difficult enough to prevent these problems also.


 // My suggestion
 Object[3] toArray(Object a, Object b, Object c)
 {
      return [a, b, c]; // Correct
 }


 I would prefer static arrays not to cast to dynamic ones implicitly,  
 but using opSlice syntax instead:

 int[] arr = [0, 1, 2]; // error
 int[] arr = [0, 1, 2][]; // fine, use on your own risk


 This is just an annoyance error.  What will happen is people will  
 constantly just use the brackets to shut up the compiler without  
 thinking about what it really means.


 <sarcasm>Sure, just hide the bug without compiler worrying you about  
 presence of one. </sarcasm>


It's not an imagined problem.  Look at Java's checked exceptions for an  
example.


 You suggestion encourages unsafe by default, which is bad bad bad. It  
 just shouldn't compile. People should just use .dup if they need a  
 dynamic array unless they *really* know what they are doing, and use [].  
 It should never happen in SafeD, and probably not even allowed.


I think you miss out on a whole class of performance gains by preventing  
slicing of static arrays, but this really is an argument for what to do in  
SafeD.

-Steve
Feb 19 2010
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
Michel Fortin Wrote:


 On 2010-02-17 04:17:09 -0500, Walter Bright <newshound1 digitalmars.com> said:
 

 Don wrote:

 This is for me the last remaining D2 issue.


 
 That would make it difficult to do things like:
 
    int*[] foo(int *p)
    {
      return [p, p + 1];
    }
 
 as all the elements of the literal would also have to be immutable. I 
 think you've made a good case, but there is also this issue.


 
 One thing I like to do with array literals is use them for appending 
 several elements at once:
 
 	array ~= [1, a+1, b+1];
 
 This way I avoid multiple array appends so it should be faster, but if 
 that temporary array literal gets allocated on the heap then it's 
 rather counterproductive for me to use array literals for this.
 
 Could cases like this, where the array never escape, be allocated on the stack?


Doesn't this do exactly that now?

array ~= 1 ~ (a+1) ~ (b+1);

-Steve
Feb 17 2010
parent Michel Fortin <michel.fortin michelf.com> writes:
On 2010-02-17 18:38:55 -0500, Steven Schveighoffer <schveiguy yahoo.com> said:


 One thing I like to do with array literals is use them for appending
 several elements at once:
 
 	array ~= [1, a+1, b+1];
 
 This way I avoid multiple array appends so it should be faster, but if
 that temporary array literal gets allocated on the heap then it's
 rather counterproductive for me to use array literals for this.
 
 Could cases like this, where the array never escape, be allocated on the stack?


 
 Doesn't this do exactly that now?
 
 array ~= 1 ~ (a+1) ~ (b+1);


It doesn't compile: Error: Can only concatenate arrays, not (int ~ int).

This works fine:

	array ~= [1] ~ (a+1) ~ (b+1);

but now we have added back an array literal. :-)

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Feb 17 2010
prev sibling parent Daniel Murphy <yebbliesnospam gmail.com> writes:
Don Wrote:


 This is for me the last remaining D2 issue.
 I've mentioned this several times before. Mutable array 'literals' are 
 wrong on so many levels. Here are some of the issues, listed in 
 decreasing importance.
 
 (1) The language has no syntax for immutable array literals.
    [1,2,3] is mutable. You can created an immutable literal using the 
 awful 'static const' syntax, but that only works for named literals.
 
 (2) Concurrency issues make (1) even more important. It ought to 
 possible to pass an array (defined at compile time) as a message.
 
 (3) Performance of the existing array 'literals' is absolutely 
 appalling. I doubled the speed of my entire app by changing ONE array 
 declaration from 'immutable [] xxx' into 'static const [] xxx'!!!
 Right now, if an array literal appears in your code, chances are it's a 
 performance bug. It's deceptive that there's a hidden heap allocation 
 happening in every array literal. It's got no business being in a 
 systems language IMHO.
 
 (4) It is unintuitive. "hello" is immutable, but ['h','e','l','l','o'] 
 is not???
 
 (5) It's trivial (a one-liner) to make a function to create a mutable 
 array literal. We don't need it in the language.
 
 Mutable arrays literals are common in toy examples, but they've got very 
 little real-world relevance. By contrast, immutable array literals are 
 used in lookup tables all the time. They are extremely common. We need 
 to be able to express them.
 
 This is the only remaining thing in D2 which I'm certain is a big 
 mistake. Of course there's ugliness in various places, but it's possible 
 to ignore them or work around them. This is a primitive, 
 bread-and-butter issue with no workaround.


I agree. This has caused huge performance problems that were fixed by changing
all my 'immutable x = []' to 'static immutable x = []'.

Can we keep the current behaviour for when some of the values are not known at
compile time? I know it's just sugar but it's very sweet.
eg.
auto y = [1, 2, x, y];
Feb 18 2010