www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Possible change to array runtime?

reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
Buried inside another large thread, Don and Dicebot complained about D2's  
prevention of array stomping as being the biggest breaking change for D2,  
not that it breaks code, but silently destroys performance.

A little background for people who aren't aware:

1. D1 took the position that if an array slice started at the beginning of  
a block, and you appended to that slice, it would extend in-block no  
matter what else was referencing that data.

In other words:

char[] arr = "abc"; // this was ok in D1, which did not have the immutable  
keyword
char[] arr2 = arr[0..1];
arr2 ~= "de";

assert(arr == "ade"); // stomped!

This was a "feature", not a bug, because the idea is, once you have built  
a large or appropriately sized buffer, you may want to forget what it had,  
and re-use it by doing:

arr.length = 0;

2. D2 added a mechanism to prevent stomping, especially in the case of  
immutable, by storing the "used" length of a block inside the block itself.

This was done by yours truly, and at the same time, I also improved  
performance of array appending by exploiting the fact that D2 has  
thread-local storage by default to avoid a global GC lock.

However, any code that is compiled from D1, the following line still works  
fine:

arr.length = 0;

But has a vastly different meaning. Instead of re-using the block, it  
*abandons* the block, and allocates a new one. This leaves the original  
block as garbage, especially if nothing is left pointing at the original  
block.

3. Don's company uses D1 as its language, I highly recommend watching  
Don's Dconf13 presentation (and look forward to his Dconf14 one!) to see  
how effective D code can create unbelievable speed, especially where array  
slices are concerned. But to the above line, in D2, they must add the  
following code to get the same behavior:

arr.assumeSafeAppend();

This is an "unsafe" operation which says, "I don't care that there is  
still data in there, that other slices may refer to, just throw it away."

And in fact, I believe the performance would improve over D1 if they did  
that. But at the moment, just "switching to D2" will cause unacceptable  
performance loss.

However, there may be a way to keep the original behavior in the right  
circumstances. It's a tad bit hacky, but makes a lot of sense. So here  
goes:

Proposal: iff you are setting the length of an array via the length  
property AND you are setting the length to 0 AND the array starts at the  
beginning of the block AND the array is mutable (not const or immutable),  
THEN assumeSafeAppend is called on that array AUTOMATICALLY.

rationale for this not affecting existing code:

If you mean to abandon an array, left to other references, are you more  
likely to use arr = null, or arr.length = 0? The former disavows all  
claims to the array, allows it to be garbage collected if no other  
references exist, and is shorter to type. arr.length = 0 retains the  
pointer to the array, which keeps it in memory, but it has no access to  
the existing data. The only valid operation on that array at that point is  
to append, which means you are going to reallocate anyway, there was no  
reason to keep a pointer at the original array.

I think in vast majority of cases, where one intends the behavior to  
reallocate, they would use arr = null. In the other cases, I would argue,  
they don't know what they are doing, they incorrectly think it's going to  
take over the array again. In that case, they likely don't care if the  
existing data gets stomped.

It does not disallow the original operation, you can do the same thing  
with arr = arr[0..0];

The reason we disallow it on const or immutable is simply to prevent  
implicit overwriting of immutable data. In cases where you have a buffer  
you are building, and especially in cases of D1 code porting to D2, const  
and immutable would not exist.

I think this would "fix" the problem Sociomantic has, and not break any  
existing code that wasn't already broken.

The single largest drawback that I can think of is that the behavior would  
be different in a very small number of cases. Are those cases valid? Is  
this enough to reject the proposal?

Destroy.

-Steve
Mar 13 2014
next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 13 March 2014 at 15:24:01 UTC, Steven Schveighoffer 
wrote:

 Destroy.

 -Steve


Please keep in mind that if the objects stored are RAII, then 
if/when we will have a finalizing GC, the stomped elements will 
have been leaked.

Clobbering elements is more than just "I won't use these elements 
anymore", it's "I won't use them, and they are safe to be 
discarded of right now".

In know that's a big "if", but it could happen. If we go the way 
of your proposal, we are definitively closing that door.

--------

Semi-on topic, it would greatly help if assumeSafeAppend was 
nothrow. As of right now, there are a lot of places in phobos 
where it could be used, but isn't, because of this. In 
particular, assumeSafeAppend can be used to both shrink an array, 
*or* grow an array up-to capacity.

Its greater use in phobos would help give it more visibility and 
exposure.
Mar 13 2014
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 13 Mar 2014 11:53:15 -0400, monarch_dodra <monarchdodra gmail.com>  
wrote:


 On Thursday, 13 March 2014 at 15:24:01 UTC, Steven Schveighoffer wrote:

 Destroy.

 -Steve


 Please keep in mind that if the objects stored are RAII, then if/when we  
 will have a finalizing GC, the stomped elements will have been leaked.

 Clobbering elements is more than just "I won't use these elements  
 anymore", it's "I won't use them, and they are safe to be discarded of  
 right now".

 In know that's a big "if", but it could happen. If we go the way of your  
 proposal, we are definitively closing that door.


I'm not understanding this. Can you explain further/give example?


 Semi-on topic, it would greatly help if assumeSafeAppend was nothrow. As  
 of right now, there are a lot of places in phobos where it could be  
 used, but isn't, because of this. In particular, assumeSafeAppend can be  
 used to both shrink an array, *or* grow an array up-to capacity.

 Its greater use in phobos would help give it more visibility and  
 exposure.


https://github.com/D-Programming-Language/druntime/pull/147

reserve and capacity were made nothrow, not sure why assumeSafeAppend  
shouldn't also be.

-Steve
Mar 13 2014
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 13 March 2014 at 16:17:17 UTC, Steven Schveighoffer 
wrote:

 On Thu, 13 Mar 2014 11:53:15 -0400, monarch_dodra 
 <monarchdodra gmail.com> wrote:

 Please keep in mind that if the objects stored are RAII, then 
 if/when we will have a finalizing GC, the stomped elements 
 will have been leaked.

 Clobbering elements is more than just "I won't use these 
 elements anymore", it's "I won't use them, and they are safe 
 to be discarded of right now".

 In know that's a big "if", but it could happen. If we go the 
 way of your proposal, we are definitively closing that door.


 I'm not understanding this. Can you explain further/give 
 example?


Well, image "File": It's a struct that owns a "file handle", and 
when the struct is destroyed, it releases/closes the file. Basic 
RAII.

Now, imagine we want to open several files. We'd use:
File[] files;

"As of today", this does not end well, as the GC does not 
finalize the array elements, and the file handles are leaked. We 
could hope, that one day, the GC *will* finalize the array 
elements.

However, with your proposal, imagine this code:

File[] files;
files ~= File("foo"); //Opens file "foo"
files.length = 0;
files ~= File("bar"); //Clobbers "foo"

With this setup, the File handling "foo" gets clobbered, ruining 
any chance of releasing it ever.

The "only way" to make it work (AFAIK), would be for "length = 0" 
to first finalize the elements in the array. However, you do 
that, you may accidentally destroy elements that are still "live" 
and referenced by another array.

The same example would work with RefCounted or whatever.



I'm not too hot about this proposal. My main gripe is that while 
"length = 0" *may* mean "*I* want to discard this data", there is 
no guarantee you don't have someone else that has a handle on 
said data, and sure as hell doesn't want it clobbered.



For what it's worth, I think the problem would go away all by 
itself if "assumeSafeAppend" had more exposition, and was 
actually used. I think a simpler solution would be to simply 
educate users of this function, and promote its use. Its simpler 
than adding a special case language change.


 https://github.com/D-Programming-Language/druntime/pull/147

 reserve and capacity were made nothrow, not sure why 
 assumeSafeAppend shouldn't also be.

 -Steve


The irony is that reserve can't actually be tagged pure nor 
nothrow: Since it can cause relocation, it can call postblit, 
which in turn may actually be impure or throw.

assumeSafeAppend, on the other hand, is *certifiably* pure and 
nothrow, since it never actually touches any data.

I had opened a pull to fix this, but it was never merged, due to 
the back and forth "fiasco" of tagging said reserve. Since I'm 
not fluent in D-runtime, I just let the issue drop.
Mar 13 2014
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 13 Mar 2014 13:44:01 -0400, monarch_dodra <monarchdodra gmail.com>  
wrote:


 On Thursday, 13 March 2014 at 16:17:17 UTC, Steven Schveighoffer wrote:

 On Thu, 13 Mar 2014 11:53:15 -0400, monarch_dodra  
 <monarchdodra gmail.com> wrote:

 Please keep in mind that if the objects stored are RAII, then if/when  
 we will have a finalizing GC, the stomped elements will have been  
 leaked.

 Clobbering elements is more than just "I won't use these elements  
 anymore", it's "I won't use them, and they are safe to be discarded of  
 right now".

 In know that's a big "if", but it could happen. If we go the way of  
 your proposal, we are definitively closing that door.


 I'm not understanding this. Can you explain further/give example?


 Well, image "File": It's a struct that owns a "file handle", and when  
 the struct is destroyed, it releases/closes the file. Basic RAII.

 Now, imagine we want to open several files. We'd use:
 File[] files;

 "As of today", this does not end well, as the GC does not finalize the  
 array elements, and the file handles are leaked. We could hope, that one  
 day, the GC *will* finalize the array elements.

 However, with your proposal, imagine this code:

 File[] files;
 files ~= File("foo"); //Opens file "foo"
 files.length = 0;
 files ~= File("bar"); //Clobbers "foo"

 With this setup, the File handling "foo" gets clobbered, ruining any  
 chance of releasing it ever.


Line 3 should be files = null. There is no point to setting length to 0,  
and mostly this is a relic from D1 code. That was my basis of why it  
shouldn't cause problems.


 The "only way" to make it work (AFAIK), would be for "length = 0" to  
 first finalize the elements in the array. However, you do that, you may  
 accidentally destroy elements that are still "live" and referenced by  
 another array.


In fact, assumeSafeAppend *should* finalize the elements in the array, if  
it had that capability. When you call assumeSafeAppend, you are telling  
the runtime that you are done with the extra elements.


 I'm not too hot about this proposal. My main gripe is that while "length  
 = 0" *may* mean "*I* want to discard this data", there is no guarantee  
 you don't have someone else that has a handle on said data, and sure as  
 hell doesn't want it clobbered.


Again, the =null is a better solution. There is no point to keeping the  
same block in reference, but without any access to elements, unless you  
want to overwrite it.

The problem is that we have this new mechanism that keeps those intact. If  
I could have imagined this outcome and was aware of this logic, I would  
have kept the length = 0 mechanics from D1 to begin with.


 For what it's worth, I think the problem would go away all by itself if  
 "assumeSafeAppend" had more exposition, and was actually used. I think a  
 simpler solution would be to simply educate users of this function, and  
 promote its use. Its simpler than adding a special case language change.


The issue I'm examining is that people are reluctant to move off of D1,  
because their D1 code behaves well when they do length = 0, and it behaves  
badly in D2, even though it compiles and works correctly. They do not have  
assumeSafeAppend in D1. I'm unsure why they have not used it, either out  
of ignorance of the function or they have decided it's not worth the  
effort, I have no idea.


 https://github.com/D-Programming-Language/druntime/pull/147

 reserve and capacity were made nothrow, not sure why assumeSafeAppend  
 shouldn't also be.


 The irony is that reserve can't actually be tagged pure nor nothrow:  
 Since it can cause relocation, it can call postblit, which in turn may  
 actually be impure or throw.

 assumeSafeAppend, on the other hand, is *certifiably* pure and nothrow,  
 since it never actually touches any data.


It does touch data, but it should be pure and nothrow.


 I had opened a pull to fix this, but it was never merged, due to the  
 back and forth "fiasco" of tagging said reserve. Since I'm not fluent in  
 D-runtime, I just let the issue drop.


Which PR?

-Steve
Mar 13 2014
next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 13 March 2014 at 18:09:55 UTC, Steven Schveighoffer 
wrote:

 I had opened a pull to fix this, but it was never merged, due 
 to the back and forth "fiasco" of tagging said reserve. Since 
 I'm not fluent in D-runtime, I just let the issue drop.


 Which PR?

 -Steve


This first one was:
https://github.com/D-Programming-Language/druntime/pull/553
It "simply" marked it as nothrow, pure, but was turned down, as 
it called "throwing" code.

The second on is:
https://github.com/D-Programming-Language/druntime/pull/632
It shuffles code around to "prove" the call is actually nothrow, 
but IMO, is not worth it.

I believe in the first pull more than the second, which I believe 
is more "proof". If you are willing to review, I can re-open the 
first.
Mar 13 2014
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 13 Mar 2014 14:54:12 -0400, monarch_dodra <monarchdodra gmail.com>  
wrote:


 On Thursday, 13 March 2014 at 18:09:55 UTC, Steven Schveighoffer wrote:

 I had opened a pull to fix this, but it was never merged, due to the  
 back and forth "fiasco" of tagging said reserve. Since I'm not fluent  
 in D-runtime, I just let the issue drop.


 Which PR?

 -Steve


 This first one was:
 https://github.com/D-Programming-Language/druntime/pull/553
 It "simply" marked it as nothrow, pure, but was turned down, as it  
 called "throwing" code.


Please reopen/update this one, I will help convince Martin.

-Steve
Mar 13 2014
prev sibling next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 13 March 2014 at 18:09:55 UTC, Steven Schveighoffer 
wrote:

 On Thu, 13 Mar 2014 13:44:01 -0400, monarch_dodra 
 <monarchdodra gmail.com> wrote:


 On Thursday, 13 March 2014 at 16:17:17 UTC, Steven 
 Schveighoffer wrote:

 On Thu, 13 Mar 2014 11:53:15 -0400, monarch_dodra 
 <monarchdodra gmail.com> wrote:

 Please keep in mind that if the objects stored are RAII, 
 then if/when we will have a finalizing GC, the stomped 
 elements will have been leaked.

 Clobbering elements is more than just "I won't use these 
 elements anymore", it's "I won't use them, and they are safe 
 to be discarded of right now".

 In know that's a big "if", but it could happen. If we go the 
 way of your proposal, we are definitively closing that door.


 I'm not understanding this. Can you explain further/give 
 example?


 Well, image "File": It's a struct that owns a "file handle", 
 and when the struct is destroyed, it releases/closes the file. 
 Basic RAII.

 Now, imagine we want to open several files. We'd use:
 File[] files;

 "As of today", this does not end well, as the GC does not 
 finalize the array elements, and the file handles are leaked. 
 We could hope, that one day, the GC *will* finalize the array 
 elements.

 However, with your proposal, imagine this code:

 File[] files;
 files ~= File("foo"); //Opens file "foo"
 files.length = 0;
 files ~= File("bar"); //Clobbers "foo"

 With this setup, the File handling "foo" gets clobbered, 
 ruining any chance of releasing it ever.


 Line 3 should be files = null. There is no point to setting 
 length to 0, and mostly this is a relic from D1 code. That was 
 my basis of why it shouldn't cause problems.


well... "should"/"could". The problem (IMO) is taking perfectly 
valid code, and making a subtle and silent change to it, changing 
its behavior and potentially breaking it. It's really the most 
pernicious kind of change.


 The "only way" to make it work (AFAIK), would be for "length = 
 0" to first finalize the elements in the array. However, you 
 do that, you may accidentally destroy elements that are still 
 "live" and referenced by another array.


 In fact, assumeSafeAppend *should* finalize the elements in the 
 array, if it had that capability. When you call 
 assumeSafeAppend, you are telling the runtime that you are done 
 with the extra elements.


Good point. Very very good point. As a matter of fact, this could 
be implemented right now, couldn't it?


 I'm not too hot about this proposal. My main gripe is that 
 while "length = 0" *may* mean "*I* want to discard this data", 
 there is no guarantee you don't have someone else that has a 
 handle on said data, and sure as hell doesn't want it 
 clobbered.


 Again, the =null is a better solution. There is no point to 
 keeping the same block in reference, but without any access to 
 elements, unless you want to overwrite it.

 The problem is that we have this new mechanism that keeps those 
 intact. If I could have imagined this outcome and was aware of 
 this logic, I would have kept the length = 0 mechanics from D1 
 to begin with.


I don't like the fact that this can only be implemented in the 
compiler. Because we *are* talking about "literal" 0, right? 
Run-time 0 wouldn't have this behavior?

By implementing this, then no user defined array wrapper would be 
able to emulate this " = 0" special behavior.

By that same token, I don't see why "0" would get such special 
treatment, when I'm certain you'd find just as many instance of 
"length = 1;", which means to say "keep the first element, and 
start clobering from there".


 For what it's worth, I think the problem would go away all by 
 itself if "assumeSafeAppend" had more exposition, and was 
 actually used. I think a simpler solution would be to simply 
 educate users of this function, and promote its use. Its 
 simpler than adding a special case language change.


 The issue I'm examining is that people are reluctant to move 
 off of D1, because their D1 code behaves well when they do 
 length = 0, and it behaves badly in D2, even though it compiles 
 and works correctly. They do not have assumeSafeAppend in D1. 
 I'm unsure why they have not used it, either out of ignorance 
 of the function or they have decided it's not worth the effort, 
 I have no idea.


Well... "s/some people/sociomantic/". How hard would it be to add 
it to D1, to help the migration process? I know it's "closed", 
but there are always exceptions.

Honestly, I'm 100% fine with braking "compile-time" changes. 
Behavioral changes on the other hand...


 https://github.com/D-Programming-Language/druntime/pull/147

 reserve and capacity were made nothrow, not sure why 
 assumeSafeAppend shouldn't also be.


 The irony is that reserve can't actually be tagged pure nor 
 nothrow: Since it can cause relocation, it can call postblit, 
 which in turn may actually be impure or throw.

 assumeSafeAppend, on the other hand, is *certifiably* pure and 
 nothrow, since it never actually touches any data.


 It does touch data, but it should be pure and nothrow.


Right. *BUT*, it is not safe. This means that any code that uses 
"length = 0;" becomes unsafe.

At least I *think* it's unsafe, there is always a think line 
between "get my hands dirty under the hood" and "memory safety".

I'm pretty sure that with creative use of assumeSafeAppend, you 
could do illegal memory access.
Mar 13 2014
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 13 Mar 2014 15:06:47 -0400, monarch_dodra <monarchdodra gmail.com>  
wrote:


 On Thursday, 13 March 2014 at 18:09:55 UTC, Steven Schveighoffer wrote:

 On Thu, 13 Mar 2014 13:44:01 -0400, monarch_dodra  
 <monarchdodra gmail.com> wrote:


 On Thursday, 13 March 2014 at 16:17:17 UTC, Steven Schveighoffer wrote:

 On Thu, 13 Mar 2014 11:53:15 -0400, monarch_dodra  
 <monarchdodra gmail.com> wrote:

 Please keep in mind that if the objects stored are RAII, then  
 if/when we will have a finalizing GC, the stomped elements will have  
 been leaked.

 Clobbering elements is more than just "I won't use these elements  
 anymore", it's "I won't use them, and they are safe to be discarded  
 of right now".

 In know that's a big "if", but it could happen. If we go the way of  
 your proposal, we are definitively closing that door.


 I'm not understanding this. Can you explain further/give example?


 Well, image "File": It's a struct that owns a "file handle", and when  
 the struct is destroyed, it releases/closes the file. Basic RAII.

 Now, imagine we want to open several files. We'd use:
 File[] files;

 "As of today", this does not end well, as the GC does not finalize the  
 array elements, and the file handles are leaked. We could hope, that  
 one day, the GC *will* finalize the array elements.

 However, with your proposal, imagine this code:

 File[] files;
 files ~= File("foo"); //Opens file "foo"
 files.length = 0;
 files ~= File("bar"); //Clobbers "foo"

 With this setup, the File handling "foo" gets clobbered, ruining any  
 chance of releasing it ever.


 Line 3 should be files = null. There is no point to setting length to  
 0, and mostly this is a relic from D1 code. That was my basis of why it  
 shouldn't cause problems.


 well... "should"/"could". The problem (IMO) is taking perfectly valid  
 code, and making a subtle and silent change to it, changing its behavior  
 and potentially breaking it. It's really the most pernicious kind of  
 change.


Yes, it's perfectly valid. We could go through a deprecation process, but  
I don't think it's worth it.

I have also proposed a mechanism to detect these calls, which could be  
turned on if you think your code might be affected by this (see my  
response to Andrei).

All in all, it's not a very good excuse for breaking anyone's code,  
especially in a way that potentially can corrupt data. It's the whole  
reason we got rid of stomping in the first place.


 The "only way" to make it work (AFAIK), would be for "length = 0" to  
 first finalize the elements in the array. However, you do that, you  
 may accidentally destroy elements that are still "live" and referenced  
 by another array.


 In fact, assumeSafeAppend *should* finalize the elements in the array,  
 if it had that capability. When you call assumeSafeAppend, you are  
 telling the runtime that you are done with the extra elements.


 Good point. Very very good point. As a matter of fact, this could be  
 implemented right now, couldn't it?


Actually, I think it can. But, no other code in the array runtime  
finalizes array elements. I don't think that it would be good to make this  
inconsistent, even if it's the right thing to do.


 I'm not too hot about this proposal. My main gripe is that while  
 "length = 0" *may* mean "*I* want to discard this data", there is no  
 guarantee you don't have someone else that has a handle on said data,  
 and sure as hell doesn't want it clobbered.


 Again, the =null is a better solution. There is no point to keeping the  
 same block in reference, but without any access to elements, unless you  
 want to overwrite it.

 The problem is that we have this new mechanism that keeps those intact.  
 If I could have imagined this outcome and was aware of this logic, I  
 would have kept the length = 0 mechanics from D1 to begin with.


 I don't like the fact that this can only be implemented in the compiler.  
 Because we *are* talking about "literal" 0, right? Run-time 0 wouldn't  
 have this behavior?


Yes, runtime 0 would have this behavior, it would be in the runtime this  
would change.


 By that same token, I don't see why "0" would get such special  
 treatment, when I'm certain you'd find just as many instance of "length  
 = 1;", which means to say "keep the first element, and start clobering  
 from there".


Because it's nonsense to keep a reference to data that you have no access  
to. A reference to a 1-element array still has elements you can access,  
you can envision a reason to keep that without wanting to append to it.

In other words, the only sane reason to shrink it to 0 but keep the  
reference not null is to append into the same block again. You should  
never see a length = 0 call without an assumeSafeAppend afterwards.


 The issue I'm examining is that people are reluctant to move off of D1,  
 because their D1 code behaves well when they do length = 0, and it  
 behaves badly in D2, even though it compiles and works correctly. They  
 do not have assumeSafeAppend in D1. I'm unsure why they have not used  
 it, either out of ignorance of the function or they have decided it's  
 not worth the effort, I have no idea.


 Well... "s/some people/sociomantic/". How hard would it be to add it to  
 D1, to help the migration process? I know it's "closed", but there are  
 always exceptions.


This was actually asked of me, by the Tango team long ago. But they just  
wanted the cache and not the stomping fix.

The stomping adds overhead, which makes things slower. But it's mitigated  
by the fact that with thread-local storage embedded into the type, we can  
avoid taking the GC lock. This results in a speedup.

I doubt it would be worthwhile.


 Honestly, I'm 100% fine with braking "compile-time" changes. Behavioral  
 changes on the other hand...


Yes, it's not a good compromise. It rests solely on the expectation that  
any code which sets length to 0 should be calling assumeSafeAppend  
afterwards, or would expect the same functionality. However, it could  
possibly be the case that someone wrote that expecting the current  
behavior.

I think realistically, as correct as the proposal is, it's not likely to  
be accepted.


 https://github.com/D-Programming-Language/druntime/pull/147

 reserve and capacity were made nothrow, not sure why assumeSafeAppend  
 shouldn't also be.


 The irony is that reserve can't actually be tagged pure nor nothrow:  
 Since it can cause relocation, it can call postblit, which in turn may  
 actually be impure or throw.

 assumeSafeAppend, on the other hand, is *certifiably* pure and  
 nothrow, since it never actually touches any data.


 It does touch data, but it should be pure and nothrow.


 Right. *BUT*, it is not safe. This means that any code that uses "length  
 = 0;" becomes unsafe.


First, I'll qualify that proposal is for this to happen only in specific  
circumstances, as outlined before.

But it's not actually un  safe, unless you are talking about immutable  
data (and this proposal specifically does not do that).


 At least I *think* it's unsafe, there is always a think line between  
 "get my hands dirty under the hood" and "memory safety".

 I'm pretty sure that with creative use of assumeSafeAppend, you could do  
 illegal memory access.


Not anything that you couldn't already do with an array element change.

-Steve
Mar 13 2014
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 13/03/14 19:09, Steven Schveighoffer wrote:

 On Thu, 13 Mar 2014 13:44:01 -0400, monarch_dodra <monarchdodra gmail.com>
wrote:

 The "only way" to make it work (AFAIK), would be for "length = 0" to first
 finalize the elements in the array. However, you do that, you may accidentally
 destroy elements that are still "live" and referenced by another array.


 In fact, assumeSafeAppend *should* finalize the elements in the array, if it
had
 that capability. When you call assumeSafeAppend, you are telling the runtime
 that you are done with the extra elements.


 I'm not too hot about this proposal. My main gripe is that while "length = 0"
 *may* mean "*I* want to discard this data", there is no guarantee you don't
 have someone else that has a handle on said data, and sure as hell doesn't
 want it clobbered.


 Again, the =null is a better solution. There is no point to keeping the same
 block in reference, but without any access to elements, unless you want to
 overwrite it.


Problem is, this still seems like safety-by-programmer-virtue.  It's far too 
easy to write ".length = 0" casually and without any attention to consequences 
one way or the other.

Isn't the whole point of the requirement to use assumeSafeAppend that it really 
forces the user to say, "Yes, I want to do away with the contents of the array 
and I take full responsibility for ensuring there's nothing else pointing to it 
that will break" ... ?

I must say that personally I'd rather have the safety-by-default and the 
obligation to write assumeSafeAppend (or use Appender!T) where performance
needs 
it, than risk code breaking because someone's function accidentally throws away 
stuff that I still had a slice of.
Mar 16 2014
prev sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Sunday, March 16, 2014 13:14:15 Joseph Rushton Wakeling wrote:

 Problem is, this still seems like safety-by-programmer-virtue. It's far too
 easy to write ".length = 0" casually and without any attention to
 consequences one way or the other.
 
 Isn't the whole point of the requirement to use assumeSafeAppend that it
 really forces the user to say, "Yes, I want to do away with the contents of
 the array and I take full responsibility for ensuring there's nothing else
 pointing to it that will break" ... ?
 
 I must say that personally I'd rather have the safety-by-default and the
 obligation to write assumeSafeAppend (or use Appender!T) where performance
 needs it, than risk code breaking because someone's function accidentally
 throws away stuff that I still had a slice of.


I tend to agree with this. Setting an array's length to 0 with the expectation 
that you will then reuse that memory is inherently unsafe. What if there are 
other arrays still referring to the same data? They'll be stomped, which could 
do some very nasty things - especially if we're talking about structs rather 
than strings.

assumeSafeAppend is explicitly unsafe and makes it clear what you're doing, 
whereas while setting an array's length to 0 may be generally nonsensical if 
you're not intending to reuse the memory again, having that essentially call 
assumeSafeAppend for you could result in very pernicious bugs when someone is 
foolish enough to set an array's length to 0 when they still have other slices 
to some or all of that array. I really think that the assumeSafeAppend needs 
to be explicit.

- Jonathan M Davis
Mar 19 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/13/14, 8:24 AM, Steven Schveighoffer wrote:

 Proposal: iff you are setting the length of an array via the length
 property AND you are setting the length to 0 AND the array starts at the
 beginning of the block AND the array is mutable (not const or
 immutable), THEN assumeSafeAppend is called on that array AUTOMATICALLY.


I think this is a sensible proposal, and well argued. However it may 
break code (even code that arguably wasn't really nicely thought 
through), and in subtle ways. "We upgraded the compiler, and our 50KLOC 
app now produces invalid results." Very hard to get working on that, in 
particular since it's unclear whether this particular matter was the 
actual culprit.

I think there's a much simpler approach that could help. Define and 
document a function clearForAppend that resets the length to 0 and puts 
the array in gear for appending. Maybe an optional parameter indicates a 
minimum desired capacity.

Then people who want to migrate from D1 to D2 or generally consider 
using this idiom can grep their source code for "length *= *0" and make 
the change on their own pace and with appropriate testing.


Andrei
Mar 13 2014
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 13 Mar 2014 12:29:59 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 3/13/14, 8:24 AM, Steven Schveighoffer wrote:

 Proposal: iff you are setting the length of an array via the length
 property AND you are setting the length to 0 AND the array starts at the
 beginning of the block AND the array is mutable (not const or
 immutable), THEN assumeSafeAppend is called on that array AUTOMATICALLY.


 I think this is a sensible proposal, and well argued. However it may  
 break code (even code that arguably wasn't really nicely thought  
 through), and in subtle ways. "We upgraded the compiler, and our 50KLOC  
 app now produces invalid results." Very hard to get working on that, in  
 particular since it's unclear whether this particular matter was the  
 actual culprit.


A good point that the error that would occur, if you depended on setting  
length = 0 to not overwrite your mutable array, would be corruption, and  
that is not a good result. Even if your code was foolish to begin with.

As a mechanism to check this, we could throw an exception when appending  
to an array that has been set to length = 0, but not had assumeSafeAppend  
called on it. This would obviously just be a bug-finding mechanism, and  
not for production.


 I think there's a much simpler approach that could help. Define and  
 document a function clearForAppend that resets the length to 0 and puts  
 the array in gear for appending. Maybe an optional parameter indicates a  
 minimum desired capacity.

 Then people who want to migrate from D1 to D2 or generally consider  
 using this idiom can grep their source code for "length *= *0" and make  
 the change on their own pace and with appropriate testing.


Arguably this is already possible within a project, such as Sociomantic's.  
They haven't done it yet. I don't know why.

-Steve
Mar 13 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/13/14, 10:18 AM, Steven Schveighoffer wrote:

 On Thu, 13 Mar 2014 12:29:59 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 I think there's a much simpler approach that could help. Define and
 document a function clearForAppend that resets the length to 0 and
 puts the array in gear for appending. Maybe an optional parameter
 indicates a minimum desired capacity.

 Then people who want to migrate from D1 to D2 or generally consider
 using this idiom can grep their source code for "length *= *0" and
 make the change on their own pace and with appropriate testing.


 Arguably this is already possible within a project, such as
 Sociomantic's. They haven't done it yet. I don't know why.


Don?

Andrei
Mar 13 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 13 March 2014 at 17:18:05 UTC, Steven Schveighoffer 
wrote:

 I think there's a much simpler approach that could help. 
 Define and document a function clearForAppend that resets the 
 length to 0 and puts the array in gear for appending. Maybe an 
 optional parameter indicates a minimum desired capacity.

 Then people who want to migrate from D1 to D2 or generally 
 consider using this idiom can grep their source code for 
 "length *= *0" and make the change on their own pace and with 
 appropriate testing.


 Arguably this is already possible within a project, such as 
 Sociomantic's. They haven't done it yet. I don't know why.

 -Steve


It has not been mentioned as example of blocker but example of 
most horrible breaking change that can happen if dmd user is not 
aware of it before upgrade.

I think only practical thing that blocks D2 transition right now 
is that no one can afford to work on it full-time and amount of 
changes needed is too big to be done as a side work - array 
stomping is not only problematic area, I personally think const 
will cause much more problems.

Really hope that with recent hires doing a focused porting effort 
will soon become a real option. D1 is so D1 :P
Mar 14 2014
next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 14 Mar 2014 10:50:43 -0400, Dicebot <public dicebot.lv> wrote:


 On Thursday, 13 March 2014 at 17:18:05 UTC, Steven Schveighoffer wrote:

 I think there's a much simpler approach that could help. Define and  
 document a function clearForAppend that resets the length to 0 and  
 puts the array in gear for appending. Maybe an optional parameter  
 indicates a minimum desired capacity.

 Then people who want to migrate from D1 to D2 or generally consider  
 using this idiom can grep their source code for "length *= *0" and  
 make the change on their own pace and with appropriate testing.


 Arguably this is already possible within a project, such as  
 Sociomantic's. They haven't done it yet. I don't know why.

 -Steve


 It has not been mentioned as example of blocker but example of most  
 horrible breaking change that can happen if dmd user is not aware of it  
 before upgrade.


I think that is very much a matter of your coding style. Many people  
wouldn't notice any changes, and might even notice an improvement.


 I think only practical thing that blocks D2 transition right now is that  
 no one can afford to work on it full-time and amount of changes needed  
 is too big to be done as a side work - array stomping is not only  
 problematic area, I personally think const will cause much more problems.

 Really hope that with recent hires doing a focused porting effort will  
 soon become a real option. D1 is so D1 :P


I think with the example Don showed, the above fix is not going to catch  
all the problems. There is no easy grep fix.

I totally get that the effort is not worth the benefit currently. Just let  
us know if we can help in any way when it becomes more of a realistic  
priority. I would love to see the most obvious commercial D success story  
on D2 :)

-Steve
Mar 14 2014
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 3/14/14, Dicebot <public dicebot.lv> wrote:

 Really hope that with recent hires doing a focused porting effort
 will soon become a real option. D1 is so D1 :P


OT and maybe NDAd: how many people have been hired? I know of at least two. :)
Mar 14 2014
prev sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, March 14, 2014 14:50:43 Dicebot wrote:

 It has not been mentioned as example of blocker but example of
 most horrible breaking change that can happen if dmd user is not
 aware of it before upgrade.
 
 I think only practical thing that blocks D2 transition right now
 is that no one can afford to work on it full-time and amount of
 changes needed is too big to be done as a side work - array
 stomping is not only problematic area, I personally think const
 will cause much more problems.
 
 Really hope that with recent hires doing a focused porting effort
 will soon become a real option. D1 is so D1 :P


Just a thought for this particular case. You could write a clearForAppend for 
D1 which simply sets the array length to 0 and slowly change your current code 
to use that instead of setting the array length to 0 directly. Then when you 
do go to port to D2, you can simply change clearForAppend's behavior so that 
it also calls assumeSafeAppend. That would allow you to do this particular 
change as a side project rather than focusing on it full time, which obviously 
isn't enough on its own, but it could help.

- Jonathan M Davis
Mar 19 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
If I may, and as author of dcollection, you'll certainly agree, I
think the way forward is improving the language when it comes to
collections.

That way, any behavior can be provided as user type. Patching the
language in such a way is simply moving the problem around.


qualifier for template type parameters. I consider this to be on


The problem is that T!ElementType is a ompletely different type
than T!const(ElementType) . Because of this, if is almost
impossible to provide anything that behave anything close to
arrays. That is th reason why collection are so scarce in D, this
is the reason why efforts to implement AA without the black magic
involved right now are failing.

This problem is a killer for the viability of D2 on the long run.
I'm not saying that lightly.
Mar 13 2014
next sibling parent reply "sclytrack" <sclytrack sexy.com> writes:
On Thursday, 13 March 2014 at 21:28:46 UTC, deadalnix wrote:

 If I may, and as author of dcollection, you'll certainly agree, 
 I
 think the way forward is improving the language when it comes to
 collections.

 That way, any behavior can be provided as user type. Patching 
 the
 language in such a way is simply moving the problem around.


 qualifier for template type parameters. I consider this to be on


 The problem is that T!ElementType is a ompletely different type
 than T!const(ElementType) . Because of this, if is almost
 impossible to provide anything that behave anything close to
 arrays. That is th reason why collection are so scarce in D, 
 this
 is the reason why efforts to implement AA without the black 
 magic
 involved right now are failing.

 This problem is a killer for the viability of D2 on the long 
 run.
 I'm not saying that lightly.



	Language changes? Like this?

	-------------

	Delayed D Language			D Normal

	immutable int * a;			int * a;
	immutable qual(int) * a;		immutable(int) * a;
	immutable int qual(*) a;		immutable int * a;

	-------------

	struct A
	{
		int a;
		qual(int) b;
	}
			

	A vari;
	const A cvari;
	vari.a = 1;
	cvari = vari;
	cvari.a = 2;
	cvari.b = 3;	//error

	-------------

	qual
	class Test
	{
	}


	Delayed D				D struct language

	immutable Test t;			immutable(Test) * t;
	immutable qual(Test) t; 		immutable Test * t;

	-------------


	struct Array(T)
	{
		qual(T) * data;
		size_t length;
	}


	Array!(int) a;	
	const Array!(int) b;

	void routine(const Array!(int) b)
	{

	}

	b = a;
	routine(a);
Mar 13 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 13 March 2014 at 22:21:56 UTC, sclytrack wrote:

 	Language changes? Like this?

 	-------------

 	Delayed D Language			D Normal

 	immutable int * a;			int * a;
 	immutable qual(int) * a;		immutable(int) * a;
 	immutable int qual(*) a;		immutable int * a;

 	-------------

 	struct A
 	{
 		int a;
 		qual(int) b;
 	}
 			

 	A vari;
 	const A cvari;
 	vari.a = 1;
 	cvari = vari;
 	cvari.a = 2;
 	cvari.b = 3;	//error

 	-------------

 	qual
 	class Test
 	{
 	}


 	Delayed D				D struct language

 	immutable Test t;			immutable(Test) * t;
 	immutable qual(Test) t; 		immutable Test * t;

 	-------------


 	struct Array(T)
 	{
 		qual(T) * data;
 		size_t length;
 	}


 	Array!(int) a;	
 	const Array!(int) b;

 	void routine(const Array!(int) b)
 	{

 	}

 	b = a;
 	routine(a);


I'm not sure how to understand your change proposal, you should
explicit it more. But it seems that you get what the problem is
(or at least, you are poking the right language constructs in
your proposal).
Mar 13 2014
parent "sclytrack" <sclytrack fake.com> writes:
On Thursday, 13 March 2014 at 22:21:56 UTC, sclytrack wrote:


 	Delayed D Language			D Normal

 	immutable int * a;			int * a;
 	immutable qual(int) * a;		immutable(int) * a;
 	immutable int qual(*) a;		immutable int * a;

 	-------------

 	struct A
 	{
 		int a;
 		qual(int) b;
 	}
 			

 	A vari;
 	const A cvari;
 	vari.a = 1;
 	cvari = vari;
 	cvari.a = 2;
 	cvari.b = 3;	//error

 	-------------

 	qual
 	class Test
 	{
 	}


 	Delayed D				D struct language

 	immutable Test t;			immutable(Test) * t;
 	immutable qual(Test) t; 		immutable Test * t;

 	-------------


 	struct Array(T)
 	{
 		qual(T) * data;
 		size_t length;
 	}


 	Array!(int) a;	
 	const Array!(int) b;

 	void routine(const Array!(int) b)
 	{

 	}

 	b = a;
 	routine(a);


I'm not sure how to understand your change proposal, you should
explicit it more. But it seems that you get what the problem is
(or at least, you are poking the right language constructs in
your proposal).


qual would be the entry point of the qualifier so

immutable int * * * * qual(*) * * * data;

So everything to the left would be immutable and
everything to the right would be mutable.

	struct A
  	{
  		char [] a;
  		qual(char) [] b;
  	}

When defining const A data; Again the const
is delayed until the entry qual(int). So ...

A data;

	struct A
	{
		char [] a;
		char [] b;
	}

const A cdata;

	struct A
	{
		char [] a;
		const(char) [] b;
	}

immutable A idata;

	struct A
	{
		char [] a;
		immutable(char) [] b;
	}

All this A has a single memory layout and
a single qualifier that modifies them.

cdata = data;
cdata = idata;

---

	struct B
	{
		A data;
		qual(A) qdata;
	}

B bdata;

	struct B
	{
		A data;
		A qdata;
	}


const B cbdata;

	struct B
	{
		A data;
		const(A) qdata;     //struct A { char [] a; const(char) [] b; 
}		
	}

immutable B ibdata;

	struct B
	{
		A data;
		immutable(A) qdata;	//struct A { char [] a; immutable(char) [] 
b; }
	}


cbdata = bdata
cbdata = ibdata

For functions it the qualifier would also be delayed which
means having the word const in front of it doesn't necessarily
mean it is const.

void routine(const A * data)
{
	data.a = "modifiable";
}

If we are forcing a delayed qualifier on A we are forcing
the same one as the one on the pointer.

const A qual(*) data;

It is the same const applied to the pointer and the
struct A.

---

Okay now to more conventional D.

These two are more or less the same type.

Container!(PayloadType)
Container!(PayloadType, const)

The following two are different types.

Container!(PayloadType)
Container!(const(PayloadType))

---

Container!(PayloadType, const)

The qualifier can be applied to your payload
field and even none-payload fields.

The thing is only PayLoadType is part of the
type definition and the qualifier at the
end is not.

The qualifier at the end
must not participate in the type definition
in any way. Not in static if and so forth.
By some magic needs to be prevented.

Having to deal with a single qualifier
is much easier than having to deal with
the qualifier of every payload.



Again a bit of qual code.

struct Container!(PayloadType)
{
	qual(char) [] a;
	PayloadType payload;
}

PayloadType participates in the type and
sets the memory layout of the type.

Here the qualifier isn't even on the payload.

---

Container!(const(int), int, char [], const)

The first 3 define the type.

	(const(int), int, char [])

The last bit. The qualifier (const) of the
container does not participate
in the definition of the type.

---
For overloading there are too many options.

void routine( immutable Container!(int, const) data);		//a
void routine( immutable Container!(int, immutable) data);	//b

Here the first one (a) is not needed. In delayed qualifier D
this would be.

void routine(imutable qual(Container!(int)) data)		//b

Setting the qual in right in front of the container means that
immutable applies to all the fields of the Container!(int) and
not only those marked with qual.

---

In Normal D style

Existing style. Already possible.

void routine( Container!(int,  ) data);				//a
void routine( const Container!(int, const) data);		//b
void routine( immutable Container!(int, immutable) data);	//c

The new ones.

void routine( Container!(int, const ) data);	
void routine( Container!(int, immutable) data);

The nonsense ones.

void routine( const Container!(int, ) data);		
void routine( immutable Container!(int, const) data);

---


Anyways. Thank you for your time reading this mess.
Mar 17 2014
prev sibling parent "sclytrack" <sclytrack collector.com> writes:
On Thursday, 13 March 2014 at 21:28:46 UTC, deadalnix wrote:

 If I may, and as author of dcollection, you'll certainly agree, 
 I
 think the way forward is improving the language when it comes to
 collections.

 That way, any behavior can be provided as user type. Patching 
 the
 language in such a way is simply moving the problem around.


------------------------------------------

If it is not in the language then maybe a design pattern.

------------------------------------------



 qualifier for template type parameters. I consider this to be on



------------------------------------------

Number 1 problem. Noted.

------------------------------------------



 The problem is that T!ElementType is a ompletely different type
 than T!const(ElementType) . Because of this, if is almost


------------------------------------------

I grasp that you need some sort of an implicit cast from
T!ElementType to T!const(ElementType).

void routine(T!const(ElementType) p) {}
T!ElementType a;

routine(a);

------------------------------------------

Collection!(const(Element1), immutable(Element2), Element3);

There are a lot of different implicit conversions that
stacking up. Now separating the qualifier like this.

T!(ElementType, immutable)

The implicit conversions would then be based on that sole
qualifier at the end. Would this not be sufficient for your
collections? (Not rethorical).

Collection!(const(Element1), immutable(Element2), Element3, 
const);


------------------------------------------


void routine(immutable Collection!(ElementType, const));     //Meh
void routine(immutable Collection!(ElementType, immutable));

I also feel like the above //Meh should not be there.

------------------------------------------



 impossible to provide anything that behave anything close to
 arrays. That is th reason why collection are so scarce in D, 
 this
 is the reason why efforts to implement AA without the black 
 magic


------------------------------------------

I dunno. Not a magician. Have to think.

------------------------------------------



 involved right now are failing.



 This problem is a killer for the viability of D2 on the long 
 run.
 I'm not saying that lightly.


------------------------------------------


Everybody wants a solution to this.

That is why everybody is jumping on
this like crazy instead of responding to the
finalize by default thread. (irony)


Some more "Collection and Const Noise".  Collection!const(Noise)
Mar 15 2014
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 13 Mar 2014 11:24:01 -0400, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:



 arr.length = 0;


...

 3. Don's company uses D1 as its language, I highly recommend watching  
 Don's Dconf13 presentation (and look forward to his Dconf14 one!) to see  
 how effective D code can create unbelievable speed, especially where  
 array slices are concerned. But to the above line, in D2, they must add  
 the following code to get the same behavior:

 arr.assumeSafeAppend();


Just a quick note, buried in same thread that Don mentioned, he outlined a  
more specific case, and this does not involve setting length to 0, but to  
any arbitrary value.

This means my approach does not help them, and although it makes sense,  
the idea that it would help Sociomantic move to D2 is not correct.

-Steve
Mar 14 2014
parent reply "Don" <x nospam.com> writes:
On Friday, 14 March 2014 at 14:48:13 UTC, Steven Schveighoffer 
wrote:

 On Thu, 13 Mar 2014 11:24:01 -0400, Steven Schveighoffer 
 <schveiguy yahoo.com> wrote:



 arr.length = 0;


 ...

 3. Don's company uses D1 as its language, I highly recommend 
 watching Don's Dconf13 presentation (and look forward to his 
 Dconf14 one!) to see how effective D code can create 
 unbelievable speed, especially where array slices are 
 concerned. But to the above line, in D2, they must add the 
 following code to get the same behavior:

 arr.assumeSafeAppend();


 Just a quick note, buried in same thread that Don mentioned, he 
 outlined a more specific case, and this does not involve 
 setting length to 0, but to any arbitrary value.

 This means my approach does not help them, and although it 
 makes sense, the idea that it would help Sociomantic move to D2 
 is not correct.

 -Steve


Actually it would help a great deal. In most cases, we do set the 
length to 0. That example code is unusual.

FYI: In D1, this was the most important idiom in the language.
In the first D conference in 2007, a feature T[new] was 
described, specifically to support this idiom in a safe manner. 
Implementation was begun in the compiler. Unfortunately, it 
didn't happen in the end. I'm not sure if it would actually have 
worked or not.


BTW you said somewhere that concatenation always allocates. What 
I actually meant was ~=, not ~. In our code it is always preceded 
by .length = 0 though.
It's important that ~= should not allocate, when the existing 
capacity is large enough.
Mar 14 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/14/14, 8:58 AM, Don wrote:

 Actually it would help a great deal. In most cases, we do set the length
 to 0. That example code is unusual.

 FYI: In D1, this was the most important idiom in the language.
 In the first D conference in 2007, a feature T[new] was described,
 specifically to support this idiom in a safe manner. Implementation was
 begun in the compiler. Unfortunately, it didn't happen in the end. I'm
 not sure if it would actually have worked or not.


Some perspective would be interesting.

1. Do you think eliminating stomping was a strategic mistake? (Clearly 
it had tactical issues with regard to portability.)

2. If you could do it over again, would you go with T[new]? Something else?


Andrei
Mar 14 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 14 March 2014 at 16:38:23 UTC, Andrei Alexandrescu
wrote:

 Some perspective would be interesting.

 1. Do you think eliminating stomping was a strategic mistake? 
 (Clearly it had tactical issues with regard to portability.)


We should stomp isolated arrays.
We must not stomp immutable arrays. That mean we must not stomp
const ones as well.


 2. If you could do it over again, would you go with T[new]? 
 Something else?


No much choixe here, because of the problem mentioned earlier in
this thread. i'm baffled that this doesn't grab more attention.
Mar 14 2014
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 14 Mar 2014 11:58:06 -0400, Don <x nospam.com> wrote:


 On Friday, 14 March 2014 at 14:48:13 UTC, Steven Schveighoffer wrote:

 On Thu, 13 Mar 2014 11:24:01 -0400, Steven Schveighoffer  
 <schveiguy yahoo.com> wrote:



 arr.length = 0;


 ...

 3. Don's company uses D1 as its language, I highly recommend watching  
 Don's Dconf13 presentation (and look forward to his Dconf14 one!) to  
 see how effective D code can create unbelievable speed, especially  
 where array slices are concerned. But to the above line, in D2, they  
 must add the following code to get the same behavior:

 arr.assumeSafeAppend();


 Just a quick note, buried in same thread that Don mentioned, he  
 outlined a more specific case, and this does not involve setting length  
 to 0, but to any arbitrary value.

 This means my approach does not help them, and although it makes sense,  
 the idea that it would help Sociomantic move to D2 is not correct.

 -Steve


 Actually it would help a great deal. In most cases, we do set the length  
 to 0. That example code is unusual.


If that example is not usual, it means that case is easy to fix (as I  
stated in the other post). Can you estimate how many ways your code  
contracts the length of an array? I assume all of them must be fully  
referencing the block, since that was the requirement in D1 (the slice had  
to point at the beginning of the block for append to work).

I imagine that adding this 'feature' of setting length = 0 would help, but  
maybe just adding a new, but similar symbol for length that means "Do what  
D1 length would have done" would be less controversial for adding to  
druntime. Example strawman:

arr.slength = 0; // effectively the same as arr.length = 0;  
arr.assumeSafeAppend();

It would do the same thing, but the idea is it would work for extension  
too -- if arr points at the beginning of the block, and slength *grows*  
into the block, it would work the same as D1 as well -- adjusting the  
"used" length and not reallocating.

Essentially, you would have to s/.length/.slength, and everything would  
work. Of course, length is not a keyword, so there may be other cases  
where length is used (read property for instance) where slength would not  
necessarily have to be used.

However, one thing we cannot fix is:

arr = arr[0..$-1];
arr ~= x;

This would reallocate due to stomp prevention, and I can't fix that. Do  
you have any cases like this, or do you always use .length?


 FYI: In D1, this was the most important idiom in the language.
 In the first D conference in 2007, a feature T[new] was described,  
 specifically to support this idiom in a safe manner. Implementation was  
 begun in the compiler. Unfortunately, it didn't happen in the end. I'm  
 not sure if it would actually have worked or not.


I think the benefits would have been very minimal. T[new] would be almost  
exactly like T[]. And you still would have to have updated all your code  
to use T[new]. When to use T[new] or T[] would have driven most people mad.

I was around back then, and I also remember Andrei, when writing TDPL,  
stating that the T[new] and T[] differences were so subtle and confusing  
that he was glad he didn't have to specify that.


 BTW you said somewhere that concatenation always allocates. What I  
 actually meant was ~=, not ~.


OK


 In our code it is always preceded by .length = 0 though.
 It's important that ~= should not allocate, when the existing capacity  
 is large enough.


That is the case for D2, as long as the slice is qualified as ending at  
the end of the valid data (setting length to 0 doesn't do this,  
obviously). This is a slight departure from D1, which required the slice  
to point at the *beginning* of the data.

-Steve
Mar 14 2014
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 03/14/2014 06:38 PM, Steven Schveighoffer wrote:

 I imagine that adding this 'feature' of setting length = 0 would help,
 but maybe just adding a new, but similar symbol for length that means
 "Do what D1 length would have done" would be less controversial for
 adding to druntime.  Example strawman:

 arr.slength = 0; // effectively the same as arr.length = 0;
 arr.assumeSafeAppend();

 It would do the same thing, but the idea is it would work for extension
 too -- if arr points at the beginning of the block, and slength *grows*
 into the block, it would work the same as D1 as well -- adjusting the
 "used" length and not reallocating.

 Essentially, you would have to s/.length/.slength, and everything would
 work. Of course, length is not a keyword, so there may be other cases
 where length is used (read property for instance) where slength would
 not necessarily have to be used.

 However, one thing we cannot fix is:

 arr = arr[0..$-1];
 arr ~= x;

 This would reallocate due to stomp prevention, and I can't fix that.


auto sslice(T)(T[] arr, size_t s, size_t e){
     auto r=arr[s..e];
     r.assumeSafeAppend();
     return r;
}

arr = arr.sslice(0, $-1);
arr ~= x;


 Do you have any cases like this, or do you always use .length?


I think just replacing all usages of features that have changed 
behaviour with library implementations that simulate the old behaviour 
would be quite easy.
Mar 16 2014