www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP 1025--Dynamic Arrays Only Shrink, Never Grow--Community Review

reply Mike Parker <aldacron gmail.com> writes:
This is the feedback thread for the first round of Community 
Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

All review-related feedback on and discussion of the DIP should 
occur in this thread. The review period will end at 11:59 PM ET 
on November 25, or when I make a post declaring it complete.

At the end of Round 1, if further review is deemed necessary, the 
DIP will be scheduled for another round of Community Review. 
Otherwise, it will be queued for the Final Review and Formal 
Assessment.

Anyone intending to post feedback in this thread is expected to 
be familiar with the reviewer guidelines:

https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

*Please stay on topic!*

Thanks in advance to all who participate.
Nov 11 2019
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 3:27:26 AM MST Mike Parker via Digitalmars-d 
wrote:

 This is the feedback thread for the first round of Community
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e
 4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should
 occur in this thread. The review period will end at 11:59 PM ET
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, the
 DIP will be scheduled for another round of Community Review.
 Otherwise, it will be queued for the Final Review and Formal
 Assessment.

 Anyone intending to post feedback in this thread is expected to
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.


Ouch. This would be a _huge_ breaking change. The ability to grow dynamic
arrays is used all over the place. It's also one of the great benefits of
D's dynamic array IMHO. The fact that you append to a dynamic array without
caring where it came from can be incredibly useful, and if you really care
about checking whether it has the capacity to grow or whether it's managed
by the GC, we have functions to check for that.

How would this interact with stuff like assumeSafeAppend? There are code
bases that take advantage of the combination of assumeSafeAppend so that
they don't have to worry about managing allocations (since ~= takes care of
it all), but they can still reuse the dynamic array and thus avoid
reallocations if the GC-managed memory block that the dynamic array is a
slice of is large enough to hold whatever data is appended. It looks to me
like change would break the code in all such code bases.

As for Appender, it's designed around the idea that you're going to build an
array and then use it rather than you're going to grow it or shrink it as
needed, so it doesn't work anywhere near as well when dealing with an array
shrinking, let alone growing and shrinking. It also does not work for the
use case where you're passing dynamic arrays around and then appending to
them, since then you don't have access to the Appender, just the dynamic
array. Also, the ability to append to a dynamic array without worrying about
whether it allocates or not such that the array can grow in place when it
can (thus avoiding unnecessary allocations) but still have reallocations
occur when it needs to is something that isn't going to work with Appender.
In my experience, code operating on strings in particular takes advantage of
being able to slice and append to dynamic arrays with impunity - which works
particularly well with immutable, since then you don't have to worry about
any of the characters being mutated. Changing such code to use Appender or ~
would be far more complicated and is likely to cause a lot more unnecessary
allocations to occur.

Also, I fail to see how getting rid of the ability to append and reallocate
dynamic arrays really helps with managing memory when the memory that the
dynamic array is a slice of is not allocated by the GC. Dynamic arrays
aren't reference-counted. You can have slices throughout the code which all
refer to the same memory even if they never do an append operation. All of
the tools that you'd need to use in terms of scope or pure or coding
conventions or whatever to know whether a slice of that memory exists
anywhere (and thus whether it's safe to free it) would be the same whether
it can be appended to or not. And if you want to disable the ability to
append to a dynamic array, you can always just mark the code as  nogc. Why
do programmers who _do_ want to use the GC in their code need to be punished
by making dynamic array worse, when anyone who doesn't want to use the GC
can already explicitly add restrictions forbidding it?

Honestly, this seems like a prime case for how trying to make D work better
with code that doesn't want to use the GC is making D worse for code that
does want to use the GC. I really, really, really hope that this change does
not make it into the language. From where I sit, it would make the language
hands down worse. In fact, as far as usability goes, it would make D's
strings _worse_ than C++'s strings. Sure, we'd still be able to get
substrings more efficiently than C++ can, but not being able to append to a
string makes it a pretty terrible string. At least C++'s strings can do that
efficiently.

IMHO, where D's dynamic arrays and strings sit right now as far as memory
allocation goes is the sweet spot. They're efficient at both growing and
shrinking. If I wanted a language that was catering to the non-GC case over
the GC case, I'd use C++. Improvements to D that make it work better with
 nogc code without compromising code that uses the GC seems like a good idea
to me, but this very much compromises the language for code that uses the
GC. _Please_ don't do this.

- Jonathan M Davis
Nov 11 2019
next sibling parent bachmeier <no spam.net> writes:
On Monday, 11 November 2019 at 11:17:26 UTC, Jonathan M Davis 
wrote:


 Honestly, this seems like a prime case for how trying to make D 
 work better with code that doesn't want to use the GC is making 
 D worse for code that does want to use the GC. I really, 
 really, really hope that this change does not make it into the 
 language. From where I sit, it would make the language hands 
 down worse. In fact, as far as usability goes, it would make 
 D's strings _worse_ than C++'s strings. Sure, we'd still be 
 able to get substrings more efficiently than C++ can, but not 
 being able to append to a string makes it a pretty terrible 
 string. At least C++'s strings can do that efficiently.

 IMHO, where D's dynamic arrays and strings sit right now as far 
 as memory allocation goes is the sweet spot. They're efficient 
 at both growing and shrinking. If I wanted a language that was 
 catering to the non-GC case over the GC case, I'd use C++. 
 Improvements to D that make it work better with  nogc code 
 without compromising code that uses the GC seems like a good 
 idea to me, but this very much compromises the language for 
 code that uses the GC. _Please_ don't do this.


I'll move on from D if this change is incorporated. It's a 
terrible idea that would break code that shouldn't be broken. It 
would also be a sign that D is effectively giving up on GC in an 
attempt to compete with Rust, and as such is no longer suited for 
general purpose programming.
Nov 11 2019
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 11:17:26 UTC, Jonathan M Davis 
wrote:

 D's strings _worse_ than C++'s strings. Sure, we'd still be 
 able to get substrings more efficiently than C++ can, but not 
 being able to append to a string makes it a pretty terrible 
 string. At least C++'s strings can do that efficiently.


C++17 differentiate between the owner of the string:

https://en.cppreference.com/w/cpp/string/basic_string

And the "borrowed" slice of a string:

https://en.cppreference.com/w/cpp/string/basic_string_view

In C++20 this is extended to contiguous buffers with std::span.
Nov 11 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 3:17 AM, Jonathan M Davis wrote:

 Ouch. This would be a _huge_ breaking change. The ability to grow dynamic
 arrays is used all over the place. It's also one of the great benefits of
 D's dynamic array IMHO. The fact that you append to a dynamic array without
 caring where it came from can be incredibly useful, and if you really care
 about checking whether it has the capacity to grow or whether it's managed
 by the GC, we have functions to check for that.


While:

    a ~= b;

would no longer be allowed,

    a = a ~ b;

would be. Yes, it is a breaking change, but the fix for when that behavior is 
desired is pretty simple.
Nov 11 2019
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/11/19 4:32 PM, Walter Bright wrote:

 On 11/11/2019 3:17 AM, Jonathan M Davis wrote:

 Ouch. This would be a _huge_ breaking change. The ability to grow dynamic
 arrays is used all over the place. It's also one of the great benefits of
 D's dynamic array IMHO. The fact that you append to a dynamic array 
 without
 caring where it came from can be incredibly useful, and if you really 
 care
 about checking whether it has the capacity to grow or whether it's 
 managed
 by the GC, we have functions to check for that.


 
 While:
 
     a ~= b;
 
 would no longer be allowed,
 
     a = a ~ b;
 
 would be. Yes, it is a breaking change, but the fix for when that 
 behavior is desired is pretty simple.


This is not the same. Appending has amortized cost, while concatenation 
is not. Expect to have D applications which use this approach for 
updating explode in runtime.

-Steve
Nov 11 2019
prev sibling next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":


The rationale is severely lacking:

Any issue related to the GC is completely orthogonal to any 
issues with memory safety.
The first example is buggy, but is  system because it uses free, 
and thus the onus is on the user.
The second example is well defined because ~= copies if the 
capacity is exceeded[1].


The prior work section lists no references and does not present 
any logical connection to the ideas presented in the rationale.


The breaking changes are massive and not well argued for.
The suggested workaround of

1) a = a ~ b;

in place of

2) a ~= b;

is functional identical, thus any issues present in 2 are also 
present in 1. The suggestion that doing so in a loop would create 
more garbage is false as it is functionally identical.



The author should revisit the core assumptions made and rewrite 
the DIP accordingly with references for prior art. If the issue 
in example one is considered bad enough to warrant doing 
something about then that issue should be tacked specifically, 
starting perhaps with a druntime / GC option to later 
statistics/log when an unmanaged slice is appended to.

[1]:

void main()
{
     import std.stdio;
     enum { dead, alive }
     foreach (i;0 .. 10000)
     {
         int[] cat = new int[i+1];
         cat[i] = alive;
     	writeln(cat[i]);
         int[] b = cat;
         b ~= 1;
         b[i] = dead;
     	writeln(cat[i]);
         if (i % 1000)
         {
             import core.memory;
             GC.collect();
         }
     }
}
Nov 11 2019
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Monday, 11 November 2019 at 11:26:49 UTC, Nicholas Wilson 
wrote:

 is functional identical, thus any issues present in 2 are also 
 present in 1. The suggestion that doing so in a loop would 
 create more garbage is false as it is functionally identical.


There is a difference: a = a ~ b is guaranteed to make a copy 
while a ~= b might just use spare capacity of the memory that 
slice `a` refers to.

https://dlang.org/spec/arrays.html#array-concatenation

This is what the dead/alive example is supposed to express.
It is still true however that the rationale is not very 
convincing.
The first example indeed doesn't show any benefits of the DIP:

```
int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
slice = slice ~ 1; // now guaranteed to make a copy
free(slice.ptr); // Still oops
```

The following claim in the DIP is also unsubstantiated:


 This change is a necessary part of D evolving towards being 
 memory safe without using
 a GC.


I would like to see an example of memory corruption in ` safe` 
code that can happen because of slice appending.
Nov 11 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 5:13 AM, Dennis wrote:

 On Monday, 11 November 2019 at 11:26:49 UTC, Nicholas Wilson wrote:

 is functional identical, thus any issues present in 2 are also present in 1. 
 The suggestion that doing so in a loop would create more garbage is false as 
 it is functionally identical.


 
 There is a difference: a = a ~ b is guaranteed to make a copy while a ~= b
might 
 just use spare capacity of the memory that slice `a` refers to.
 
 https://dlang.org/spec/arrays.html#array-concatenation


That's right. I had overlooked that.



 This is what the dead/alive example is supposed to express.
 It is still true however that the rationale is not very convincing.
 The first example indeed doesn't show any benefits of the DIP:
 
 ```
 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 slice = slice ~ 1; // now guaranteed to make a copy
 free(slice.ptr); // Still oops
 ```


Imagine these 3 lines are spread out over a large code base.



 The following claim in the DIP is also unsubstantiated:
 

 This change is a necessary part of D evolving towards being memory safe 
 without using
 a GC.




Memory safety cannot be achieved without control over who is the owner of
memory.



 I would like to see an example of memory corruption in ` safe` code that can 
 happen because of slice appending.


    trusted void myfree(void* p) { free(p); }
   free(slice.ptr);
Nov 11 2019
next sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright wrote:

 On 11/11/2019 5:13 AM, Dennis wrote:

 [...]


 That's right. I had overlooked that.



 [...]


 Imagine these 3 lines are spread out over a large code base.



 [...]




 Memory safety cannot be achieved without control over who is 
 the owner of memory.



 [...]


    trusted void myfree(void* p) { free(p); }
   free(slice.ptr);


That's not  safe code, it's  trusted, it's not mechanically 
checkable, and all the horrors summoned in a programmers mind can 
happens here. My impression is that this is not the point at all.
Nov 11 2019
prev sibling next sibling parent Exil <Exil gmall.com> writes:
On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright wrote:

 This is what the dead/alive example is supposed to express.
 It is still true however that the rationale is not very 
 convincing.
 The first example indeed doesn't show any benefits of the DIP:
 
 ```
 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 slice = slice ~ 1; // now guaranteed to make a copy
 free(slice.ptr); // Still oops
 ```


 Imagine these 3 lines are spread out over a large code base.


You realize he changed the code right? That is still valid code 
that exhibits the same problem even after this DIP gets 
integrated into DMD.



 The following claim in the DIP is also unsubstantiated:
 

 This change is a necessary part of D evolving towards being 
 memory safe without using
 a GC.




 Memory safety cannot be achieved without control over who is 
 the owner of memory.


So is the GC being removed from D? Is this the ultimate goal? 
Cause the convenience of the language that is provided by using a 
GC is being removed along with these DIPs. At some point there's 
not going to be a reason to the GC because all of its' 
conveniences will have been removed at this rate.
Nov 11 2019
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 11.11.19 22:41, Walter Bright wrote:

 
 Memory safety cannot be achieved without control over who is the owner 
 of memory.


The GC is a perfectly fine owner of memory. Please take a step back and 
stop gutting the language.
Nov 11 2019
prev sibling next sibling parent Dennis <dkorpel gmail.com> writes:
On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright wrote:

 That's right. I had overlooked that.


It looked like you were aware since you mentioned "although that 
will generate more GC garbage if used in a loop" when referring 
to the workaround of `a = a ~ b`.


 Imagine these 3 lines are spread out over a large code base.


With or without this DIP, that would be problematic.


 On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright 
 wrote:
 Memory safety cannot be achieved without control over who is 
 the owner of memory.


 I would like to see an example of memory corruption in ` safe` 
 code that can happen because of slice appending.


    trusted void myfree(void* p) { free(p); }
   free(slice.ptr);


free cannot be  trusted on a raw pointer type.
In that definition, you can (from  safe code) give `myFree` a 
pointer to a local variable, global variable or GC pointer, all 
of which may corrupt memory. (And I'm ignoring the lack of  live 
here. Without  live you can obviously add double free / 
use-after-free to that list.)

The only way to make `free` safe is to encapsulate it in a struct 
or class that ensures the pointer that is being freed was 
allocated with malloc. If you have a custom type, then 
concatenation like ~= will not work unless you implement it, so 
disabling it on the built-in slice type does not do much good.

In fact, you can consider ~= only syntactic sugar for a call to 
_d_arrayappendcTX.
You can disable the ~= construct, but everywhere you would use 
that operator you can still do something functionally equivalent 
and change what kind of data (i.e. stack memory, heap memory) the 
slice refers to.

So the current proposal helps discouraging certain code, but it 
does not prevent anything.
Nov 11 2019
prev sibling next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 2:41:14 PM MST Walter Bright via Digitalmars-d 
wrote:

 Memory safety cannot be achieved without control over who is the owner of
 memory.


And how does this DIP help with that? Dynamic arrays do not know or care
where their memory came from, and they do not track lifetimes in any way,
shape, or form. Doing anything that might append to them then guarantees
that they're allocated by the GC, because that's the only way that they can
grow. Without doing one of those operations, you would have to actually look
at the ptr value of the dynamic array and ask the GC whether it's owned by
the GC. Getting rid of the append operations does not change that. It's
still impossible to know who owns the memory by simply looking at the type,
and code that passes slices of non-GC allocated memory around has to be
aware of the lifetime of those dynamic arrays and what's done with them to
know whether any dynamic arrays referring to that memory still exist. The
type system doesn't help with that beyond stuff like scope and pure limiting
what can be done with the dynamic array that gets passed in. And getting rid
of the append operations doesn't change that.

Sure, right now, if you pass a dynamic array which is a slice of malloc-ed
memory to a function, that function could append to that dynamic array and
result in a reallocation. Why is that a problem? It's not like that's going
to cause a memory leak unless it was your only pointer to that memory, and
if that's the case, having the dynamic array shrink on you would be just as
big a problem, if not bigger. Either way, you have to know what the code is
doing with the dynamic array if you're going to try to determine when it's
safe to free that memory, because dynamic arrays can normally be passed
around and copied with impunity, and they're not reference-counted. Their
very nature pretty requires that if you're going to have a dynamic array
which slices non-GC-allocated memory, you have to carefully control what you
do with it, because nothing about the dynamic array manages the lifetime of
the memory it points to. That's the case whether appending is allowed or
not.

If for some reason, it is actually a problem for a piece of code if a
dynamic array ever gets appended to, then requiring that the code be  nogc
fixes that. It's then not possible to pass the dynamic array to any
functions which might append to it.

Really, code that's using dynamic arrays which are slices of anything other
than GC-allocated memory has to be very limited in how it passes them around
regardless of whether appending is possible, because dynamic arrays don't
manage their own memory at all. Code that wants to do much passing around of
malloc-ed memory to anything other than a pure function where it's clear
that the slices can't be squirreled away in any of the arguments or return
value is going to tend to need to wrap that memory in something other than a
dynamic array in order to manage its lifetime - either that, or it needs to
be written in a way that the programmer can clearly know that anything that
even might contain a reference to that memory is long gone before they're
looking to free that memory (e.g. because it's allocated at the beginning of
the program and freed at the end).

It looks to me like you're crippling the normal use case in attempt to deal
with a less common one. And it doesn't look to me like it's really fixing
anything for that less common use case.

Honestly, if it really is the case that we have to lose the ability to
append to dynamic arrays in order to get  safety for malloc-ed memory, then
I'd rather give up on  safety for malloc-ed memory and keep the ability to
append to dynamic arrays. But from what I can tell, the only argument that
the DIP has for why appending to dynamic arrays even could be a problem is
because it might be problematic if a dynamic array which slices
non-GC-allocated memory ends up being copied into newly allocated GC memory,
because it's appended to. I dispute that that's a real problem, and even if
it is one,  nogc prevents it without any language changes.

It should be clear to anyone seriously using dynamic arrays that

int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
slice ~= 1;
free(slice.ptr); // Oops!

is a problem. It's a given that appending can cause a reallocation. But even
if appending weren't allowed, you could still have something like

int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
slice = slice[1 .. $];
free(slice.ptr); // Oops!

and have problems, because the dynamic array isn't slicing the beginning of
that block of memory anymore. So, allowing a dynamic array to shrink (or
even be assigned to at all) causes similar problems as appending in this
case. Either way, relying on the dynamic array itself to know which pointer
to use to free that memory is just plain bad practice unless the code is
definitely never going to do anything that would mutate the dynamic array
itself (be it slicing, appending, or assignment from another array). In
general, if code is going to slice malloc-ed memory to get a dynamic array,
it should be keeping around a separate pointer which refers to the block
that was allocated so that it can free it when no more dynamic arrays exist
which point to that memory rather than relying on the dynamic array for that
information. Either way, the general inability to track the lifetime of
dynamic arrays outside of things like scope and pure means that you can't
rely on the type system to track the lifetime of malloc-ed memory outside of
functions that simply take an argument and return a result (with scope and
pure used to ensure that no slices of that memory are holed away somewhere).

As for code such as

enum { dead, alive }
int[] cat = new int[6];
cat[5] = alive;
int[] b = cat;
b ~= 1;      // may or may not move b to new location
b[5] = dead; // indeterminate whether cat[5] is dead or alive

So what? It's a given that appending to a dynamic array could cause a
reallocation. If that's a problem, the code can check the array's capacity
first to see if appending would cause a reallocation. This behavior of
arrays is nothing new and isn't specific to dynamic arrays which are slices
of memory which wasn't allocated by the GC. If the code actually cares about
whether a reallocation takes place, it can easily check whether it will or
has taken place, and it can easily prevent it from taking place by requiring
that the code be  nogc. The DIP does nothing to show why this behavior is
actually a problem for  safe code.

- Jonathan M Davis
Nov 11 2019
parent Andrea Fontana <nospam example.com> writes:
On Monday, 11 November 2019 at 22:40:44 UTC, Jonathan M Davis 
wrote:

 On Monday, November 11, 2019 2:41:14 PM MST Walter Bright via 
 Digitalmars-d wrote:
 It should be clear to anyone seriously using dynamic arrays that

 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 slice ~= 1;
 free(slice.ptr); // Oops!

 is a problem. It's a given that appending can cause a 
 reallocation. But even if appending weren't allowed, you could 
 still have something like

 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 slice = slice[1 .. $];
 free(slice.ptr); // Oops!

 and have problems, because the dynamic array isn't slicing the 
 beginning of that block of memory anymore.


+1
It was my first thought
Nov 12 2019
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright wrote:

 On 11/11/2019 5:13 AM, Dennis wrote:

 ```
 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 slice = slice ~ 1; // now guaranteed to make a copy
 free(slice.ptr); // Still oops
 ```


 Imagine these 3 lines are spread out over a large code base.


People doing manual memory management without RAII deserve the 
codebase complexity  they get.


 The following claim in the DIP is also unsubstantiated:
 

 This change is a necessary part of D evolving towards being 
 memory safe without using
 a GC.




 Memory safety cannot be achieved without control over who is 
 the owner of memory.


 I would like to see an example of memory corruption in ` safe` 
 code that can happen because of slice appending.


    trusted void myfree(void* p) { free(p); }
   free(slice.ptr);


As noted elsewhere that is a broken  trusted, and therefore a 
user fault, not a language fault. This is no different to 
breaking the type system in any other way.
Nov 11 2019
prev sibling parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright wrote:

 On 11/11/2019 5:13 AM, Dennis wrote:

 On Monday, 11 November 2019 at 11:26:49 UTC, Nicholas Wilson 
 wrote:

 is functional identical, thus any issues present in 2 are 
 also present in 1. The suggestion that doing so in a loop 
 would create more garbage is false as it is functionally 
 identical.


 
 There is a difference: a = a ~ b is guaranteed to make a copy 
 while a ~= b might just use spare capacity of the memory that 
 slice `a` refers to.
 
 https://dlang.org/spec/arrays.html#array-concatenation


 That's right. I had overlooked that.



 This is what the dead/alive example is supposed to express.
 It is still true however that the rationale is not very 
 convincing.
 The first example indeed doesn't show any benefits of the DIP:
 
 ```
 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 slice = slice ~ 1; // now guaranteed to make a copy
 free(slice.ptr); // Still oops
 ```


 Imagine these 3 lines are spread out over a large code base.



 The following claim in the DIP is also unsubstantiated:
 

 This change is a necessary part of D evolving towards being 
 memory safe without using
 a GC.




 Memory safety cannot be achieved without control over who is 
 the owner of memory.


I thought the GC knows if the memory is in its control or not? 
Why not prohibit the concatenation after the GC checked if it is 
memory it has no view over. That slice was allocated outside or 
points on the stack so stop with an error. It's not compile time, 
but better than nothing, but doesn't break all the apps that want 
to live a simple life of GC memory handling. If I had wanted to 
make my programmers life difficult I would have chosen Rust or 
C++.


 I would like to see an example of memory corruption in ` safe` 
 code that can happen because of slice appending.


    trusted void myfree(void* p) { free(p); }
   free(slice.ptr);


cheating doesn't count.  trusted in the same vein ad casting, is 
explicit security overriding, you get what you called for.
Nov 11 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 3:26 AM, Nicholas Wilson wrote:

 The breaking changes are massive and not well argued for.
 The suggested workaround of
 
 1) a = a ~ b;
 
 in place of
 
 2) a ~= b;
 
 is functional identical, thus any issues present in 2 are also present in 1.


The point is making it obvious that the GC is being used, i.e. it is
intentional 
behavior.
Nov 11 2019
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 2:34:39 PM MST Walter Bright via Digitalmars-d 
wrote:

 On 11/11/2019 3:26 AM, Nicholas Wilson wrote:

 The breaking changes are massive and not well argued for.
 The suggested workaround of

 1) a = a ~ b;

 in place of

 2) a ~= b;

 is functional identical, thus any issues present in 2 are also present
 in 1.


 The point is making it obvious that the GC is being used, i.e. it is
 intentional behavior.


Both ~ and ~= use the GC, and it should be obvious that they do, since they
can't work if they don't. Using ~ isn't any clearer than ~=, and ~ has worse
performance when a dynamic array is able to grow in-place.

- Jonathan M Davis
Nov 11 2019
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/11/19 4:54 PM, Jonathan M Davis wrote:

 On Monday, November 11, 2019 2:34:39 PM MST Walter Bright via Digitalmars-d
 wrote:

 On 11/11/2019 3:26 AM, Nicholas Wilson wrote:

 The breaking changes are massive and not well argued for.
 The suggested workaround of

 1) a = a ~ b;

 in place of

 2) a ~= b;

 is functional identical, thus any issues present in 2 are also present
 in 1.


 The point is making it obvious that the GC is being used, i.e. it is
 intentional behavior.


 
 Both ~ and ~= use the GC, and it should be obvious that they do, since they
 can't work if they don't. Using ~ isn't any clearer than ~=, and ~ has worse
 performance when a dynamic array is able to grow in-place.


Yes. And the more I see this example, the worse job it does as an 
example of why this DIP is needed.

In order for the DIP to guarantee the pointer of a slice is still the 
pointer to memory to be freed, you have to *disable assignment*. Do we 
want to do that too?

It also must disable popFront, or really any operation except for opIndex.

Here's another example of an error like the item in the DIP, which is 
totally possible even with the DIP:

 nogc: // just to emphasize
int[] slice = (cast(int*)malloc(10 * int.sizeof))[0 .. 10];
version(bad)
{
   int[1] arr;
   slice = arr[];
}
else version(alsoBad)
{
   slice = slice [1 .. $]; // slice doesn't grow!
}
free(slice.ptr); // Oops;

-Steve
Nov 11 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 22:04:17 UTC, Steven Schveighoffer 
wrote:

 In order for the DIP to guarantee the pointer of a slice is 
 still the pointer to memory to be freed, you have to *disable 
 assignment*. Do we want to do that too?

 It also must disable popFront, or really any operation except 
 for opIndex.


Actually, I think the example I gave was bad, since the DIP 
states that a slice is for borrowing, and then it should not be 
used for ownership. I retract it.

If a slice is a strictly borrowed view then you can have popFront 
etc, but not extending it.

This would basically be the same as C++ span with dynamic extents.


Then you would need something else for ownership of non-GC memory 
that can provide borrowed slices.

And yet again something else for write-buffer-views (e.g. a view 
on a field in a struct that should be written to, reallocation is 
bad in that situation).

So you need basically 3 different types. I think…
Nov 11 2019
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 21:54:23 UTC, Jonathan M Davis 
wrote:

 Both ~ and ~= use the GC, and it should be obvious that they 
 do, since they can't work if they don't. Using ~ isn't any 
 clearer than ~=, and ~ has worse performance when a dynamic 
 array is able to grow in-place.


a~b is easier to deal with in code analysis than a~=b because the 
latter has unknown aliasing properties.

The compiler knows more when you can either say "will always 
reallocate" or "will never reallocate".

An optimizer could always do it in place if it is known to be 
safe. Especially in a loop or in a series of concatenations.
Nov 11 2019
prev sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 11 November 2019 at 21:34:39 UTC, Walter Bright wrote:

 On 11/11/2019 3:26 AM, Nicholas Wilson wrote:

 The breaking changes are massive and not well argued for.
 The suggested workaround of
 
 1) a = a ~ b;
 
 in place of
 
 2) a ~= b;
 
 is functional identical, thus any issues present in 2 are also 
 present in 1.


 The point is making it obvious that the GC is being used, i.e. 
 it is intentional behavior.


and ~= doesn't? As noted elsewhere there is no safety issues with 
it because the GC will copy the slice or fill up available 
capacity. This entire line of reasoning is a non-sequiter.
Nov 11 2019
prev sibling next sibling parent FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.


In my homebrew D-lite language, I used T[~] as the built-in type 
for "an array that carries capacity information, owns memory 
(T[auto~] but nevermind that) and can be appended to." This 
seemed to work well. I definitely think that if we deprecate ~, 
such a type should be built-in, because this is vital 
functionality. appender is just too weird a type (it's a weird 
class-struct hybrid??) to carry this functionality.

Simple things must be easy. We don't have to import std.array for 
T[], we shouldn't have to import it for T[~].
Nov 11 2019
prev sibling next sibling parent reply Dukc <ajieskola gmail.com> writes:
I'll repeat what I said in the draft review, because I don't know 
whether my feedback wasn't noticed. In addition, I'll be more 
explicit.

In no practical way we can implement this. The breakage would 
most likely dwarf breaking autodecoding.

However, the above only applies to breaking appending to slices 
in general. I agree that if the slice is not either:

1. GC-allocated
2. referring to a static array.
3. referring to an initialization value of an array of `static` 
storage class

...then the chances of somebody appending to it are low enough 
that breaking might be viable. Even then, somebody might append 
to a mallocated slice by GC if there is another reference to the 
original slice lying around, but for me that sounds rare enough 
that finding memory leaks could justify the breakage.

I see two alternatives. We can simply discourage growing slices 
with the garbage collector. A transition switch would be 
introduced, but no removing `~` or `slice.length += 2` totally, 
or at least not for many years. We also could introduce a new 
appending operator, say >< (along with ><=), which behaves just 
like ~ but refuses to work with slices, and encourage to use that 
istead of the present append operator.

The other alternative is that slice growing by GC would check 
where the array is referring to. First, if it refers to null, the 
static memory area or the stack, check passed. Second, if it 
refers to anything the GC controls, check passed. Otherwise, it's 
assumed it's pointing to mallocated memory, or just corrupt - 
program terminated with an unrecoverable error. For  system code, 
the compiler could forgo these checks with the same rules as 
array bounds checks can be forgoed. Growing a slice of mallocated 
memory where the check is omitted would be undefined behaviour.
Nov 11 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 4:33:00 AM MST Dukc via Digitalmars-d wrote:

 I'll repeat what I said in the draft review, because I don't know
 whether my feedback wasn't noticed. In addition, I'll be more
 explicit.

 In no practical way we can implement this. The breakage would
 most likely dwarf breaking autodecoding.

 However, the above only applies to breaking appending to slices
 in general. I agree that if the slice is not either:

 1. GC-allocated
 2. referring to a static array.
 3. referring to an initialization value of an array of `static`
 storage class

 ...then the chances of somebody appending to it are low enough
 that breaking might be viable. Even then, somebody might append
 to a mallocated slice by GC if there is another reference to the
 original slice lying around, but for me that sounds rare enough
 that finding memory leaks could justify the breakage.

 I see two alternatives. We can simply discourage growing slices
 with the garbage collector. A transition switch would be
 introduced, but no removing `~` or `slice.length += 2` totally,
 or at least not for many years. We also could introduce a new
 appending operator, say >< (along with ><=), which behaves just
 like ~ but refuses to work with slices, and encourage to use that
 istead of the present append operator.

 The other alternative is that slice growing by GC would check
 where the array is referring to. First, if it refers to null, the
 static memory area or the stack, check passed. Second, if it
 refers to anything the GC controls, check passed. Otherwise, it's
 assumed it's pointing to mallocated memory, or just corrupt -
 program terminated with an unrecoverable error. For  system code,
 the compiler could forgo these checks with the same rules as
 array bounds checks can be forgoed. Growing a slice of mallocated
 memory where the check is omitted would be undefined behaviour.


IMHO, the ability to append to a dynamic array without caring whether it's
backed by GC-allocated memory, malloc-ed memory, stack allocated memory, or
whatever is a huge benefit. Right now, you can pass a dynamic array that
isn't backed by the GC to a function, and it all just works even if the
function needs to do a reallocation to make the array large enough. Doing so
is not necessarily a bug and allows you to use non-GC allocated memory where
appropriate while still being able to grow the array, and it does so without
the code having to care about what kind of memory is currently backing the
dynamic array. And if the code _does_ care for some reason, the array's
capacity property will tell you whether the array can be grown in-place, and
multiple functions in core.memory can be used to determine whether the
dynamic array is a slice of GC-allocated memory.

- Jonathan M Davis
Nov 11 2019
parent reply Dukc <ajieskola gmail.com> writes:
On Monday, 11 November 2019 at 11:51:09 UTC, Jonathan M Davis 
wrote:

 IMHO, the ability to append to a dynamic array without caring 
 whether it's backed by GC-allocated memory, malloc-ed memory, 
 stack allocated memory, or whatever is a huge benefit.


Yeah... And I definitely think all of these should continue to 
exist with the possible exception of malloced memory.


 Right now, you can pass a dynamic array that isn't backed by 
 the GC to a function, and it all just works even if the 
 function needs to do a reallocation to make the array large 
 enough. Doing so is not necessarily a bug and allows you to use 
 non-GC allocated memory where appropriate while still being 
 able to grow the array, and it does so without the code having 
 to care about what kind of memory is currently backing the 
 dynamic array.


But chances are that if you grow a malloced slice bu GC, you 
cause a memory leak, and that's what Walter is trying to solve. 
On the other hand, there should be a way for the caller to 
explicitly allow a function to grow a GC-allocated slice, without 
touching the function, and my idea does not solve that. I dunno...
Nov 11 2019
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 5:15:23 AM MST Dukc via Digitalmars-d wrote:

 On Monday, 11 November 2019 at 11:51:09 UTC, Jonathan M Davis

 wrote:

 IMHO, the ability to append to a dynamic array without caring
 whether it's backed by GC-allocated memory, malloc-ed memory,
 stack allocated memory, or whatever is a huge benefit.


 Yeah... And I definitely think all of these should continue to
 exist with the possible exception of malloced memory.


 Right now, you can pass a dynamic array that isn't backed by
 the GC to a function, and it all just works even if the
 function needs to do a reallocation to make the array large
 enough. Doing so is not necessarily a bug and allows you to use
 non-GC allocated memory where appropriate while still being
 able to grow the array, and it does so without the code having
 to care about what kind of memory is currently backing the
 dynamic array.


 But chances are that if you grow a malloced slice bu GC, you
 cause a memory leak, and that's what Walter is trying to solve.
 On the other hand, there should be a way for the caller to
 explicitly allow a function to grow a GC-allocated slice, without
 touching the function, and my idea does not solve that. I dunno...


Whether you risk causing a memory leak or not doesn't really have anything
to do with whether reallocations can occur. The code needs to be aware of
when the malloc-ed memory is definitely no longer used and then manually
free it. It's certainly true that the lack of reference counting makes it
harder to write code that uses slices of malloc-ed memory, but having slices
of malloc-ed memory poses exactly the same risks of keeping those slices
around too long whether ~= works on them or not. You either write the code
in a way that you know when the slices are no longer used so that you know
when you can free the memory, or you don't slice non-GC-allocated memory.

- Jonathan M Davis
Nov 11 2019
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":


This will make for stronger typing and is therefore a-good-thing.
Nov 11 2019
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":


«The author is unaware of any slices in other languages that 
don't also carry a .capacity property to keep track of how much 
the slice can grow before it must be reallocated.»

C++20 span:

https://en.cppreference.com/w/cpp/container/span

Maybe you can achieve binary compatibility, somehow.
Nov 11 2019
prev sibling next sibling parent ixid <adamsibson gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":


These changes are becoming too big for a production language. 
They appear to be driven by the reasonable desire to fully 
separate D from the GC and add a stronger model for memory safety 
but with this extent of change would be more appropriate for a 
test version of D to develop the whole rather than piecemeal and 
potentially release as a D3.0 if the changes become sufficiently 
major when taken together.
Nov 11 2019
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 [snip]



In that first example, the following
int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
doesn't actually compile.

You would compile it as
auto slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];

So actually
auto slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
slice ~= 1;
generates an error because you can't append an int to an int*.
Nov 11 2019
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 11 November 2019 at 13:25:35 UTC, jmh530 wrote:

 In that first example, the following
 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 doesn't actually compile.

 You would compile it as
 auto slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];


It should probably just be

int[] slice = (cast(int*)malloc(10 * int.sizeof))[0 .. 10];


Since the type certainly should be int[], and it will compile.
Nov 11 2019
parent jmh530 <john.michael.hall gmail.com> writes:
On Monday, 11 November 2019 at 13:26:56 UTC, Adam D. Ruppe wrote:

 [snip]

 It should probably just be

 int[] slice = (cast(int*)malloc(10 * int.sizeof))[0 .. 10];


 Since the type certainly should be int[], and it will compile.


Ah, thanks.

So I could probably re-write this as

```
     int* data = cast(int*)malloc(10 * int.sizeof);
     int[] slice = data[0 .. 10];
```

So maybe the issue is with being able to slice arbitrary 
pointers? I don't know...

It all seems tied in to the ability to conflate a malloc'ed slice 
with a GC'ed slice. If anything, I don't think it should be easy 
to do this.
Nov 11 2019
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Prior art fails to mention  nogc, which already prohibits these 
operations in D today. So does -betterC. These work without 
breaking nearly every D module I have ever seen for marginal 
benefit at best.
Nov 11 2019
next sibling parent reply bachmeier <no spam.net> writes:
On Monday, 11 November 2019 at 13:33:35 UTC, Adam D. Ruppe wrote:

 Prior art fails to mention  nogc, which already prohibits these 
 operations in D today. So does -betterC. These work without 
 breaking nearly every D module I have ever seen for marginal 
 benefit at best.


The "marginal benefit at best" depends. If the goal is to move 
away from the GC and make using D without the GC the priority, as 
seems to be the case based on this and other recent work, the 
benefit is not marginal. Some communication about the overall 
goal here would be nice. Is there even a plan?
Nov 11 2019
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 11 November 2019 at 14:02:36 UTC, bachmeier wrote:

 If the goal is to move away from the GC and make using D 
 without the GC the priority,


This already works - and you can statically enforce it on a 
per-function level by  nogc. Or on a per-type basis with a 
trivial wrapper struct (except null doesn't implicitly construct 
- we should fix THAT). Or on a per-project basis with -betterC.

All this proposal does is break code that hasn't opted into those 
existing options.


 Some communication about the overall goal here would be nice. 
 Is there even a plan?


I agree with this regardless though.
Nov 11 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 6:22 AM, Adam D. Ruppe wrote:

 This already works - and you can statically enforce it on a per-function level 
 by  nogc. Or on a per-type basis with a trivial wrapper struct (except null 
 doesn't implicitly construct - we should fix THAT). Or on a per-project basis 
 with -betterC.
 
 All this proposal does is break code that hasn't opted into those existing
options.


Global solutions do exist, but they don't allow mixed mode code.

The idea is to make code mechanically auditable. It isn't auditable if creating 
a wrapper is expected.



 Some communication about the overall goal here would be nice. Is there even a 
 plan?


 I agree with this regardless though.


Be able to mechanically verify that code does not have memory safety bugs. We 
may not be able to get there 100%, but the closer the better.
Nov 11 2019
prev sibling parent =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:
On 2019-11-11 13:33:35 +0000, Adam D. Ruppe said:


 Prior art fails to mention  nogc, which already prohibits these 
 operations in D today. So does -betterC. These work without breaking 
 nearly every D module I have ever seen for marginal benefit at best.


Yes, that's a very good point and gives one everything to manually 
manage memory and be sure the GC doesn't jump in.

If a mixed memory management model is used, the programmer needs to 
understand the side-effects and what happens if the GC jumps in and 
take full responsibility.

So, I don't like this change at all. KISS.

-- 
Robert M. Münch
http://www.saphirion.com
smarter | better | faster
Nov 11 2019
prev sibling next sibling parent reply IGotD- <nise nise.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.


"Enlarging a slice, using the append operator or by setting the 
.length property, makes use of the garbage collector. This 
fosters the incorrect notion that D slices require the GC."

The title mentions arrays but the rest of the text is about 
slices.

I need one clarification, this DIP regards slices not arrays? So 
an array is being able to be resized just as before. It's just 
that this will no longer apply for slices?

If this only regards slices, then I think this is a sound change, 
even if it is breaking. A slice should not really know the 
underlying algorithm how the data is stored. It could be an 
array, it could be part of a buffer or a view into any other 
structure. Limiting the slice so that it cannot grow beyond its 
original boundaries makes sense.

If this include arrays, that they aren't able to grow then I'm 
against this change but again it doesn't make any sense. Then it 
is not a dynamic array.
Nov 11 2019
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Monday, 11 November 2019 at 14:48:54 UTC, IGotD- wrote:

 I need one clarification, this DIP regards slices not arrays? 
 So an array is being able to be resized just as before. It's 
 just that this will no longer apply for slices?


Maybe this article will help:
https://dlang.org/articles/d-array-article.html#responsibility


 So where is the true dynamic array type in D? It's hidden by 
 the runtime, and in fact, has no formal type. Slices are good 
 enough, and as it turns out, the runtime is smart enough about 
 what you want to do with the data, that you almost never notice 
 dynamic arrays are missing as a full-fledged type. In fact, 
 most D coders consider the D slice to be the dynamic array type 
 -- it's even listed as a dynamic array type in the spec! The 
 lack of ownership is very subtle and easy to miss.
Nov 11 2019
parent reply IGotD- <nise nise.com> writes:
On Monday, 11 November 2019 at 14:56:22 UTC, Dennis wrote:

 On Monday, 11 November 2019 at 14:48:54 UTC, IGotD- wrote:

 I need one clarification, this DIP regards slices not arrays? 
 So an array is being able to be resized just as before. It's 
 just that this will no longer apply for slices?


 Maybe this article will help:
 https://dlang.org/articles/d-array-article.html#responsibility


 So where is the true dynamic array type in D? It's hidden by 
 the runtime, and in fact, has no formal type. Slices are good 
 enough, and as it turns out, the runtime is smart enough about 
 what you want to do with the data, that you almost never 
 notice dynamic arrays are missing as a full-fledged type. In 
 fact, most D coders consider the D slice to be the dynamic 
 array type -- it's even listed as a dynamic array type in the 
 spec! The lack of ownership is very subtle and easy to miss.




Does this mean if this DIP goes though, a "real" distinctive 
dynamic array implementation must be done in order facilitate the 
functionality that has previously been done by D-slices?
Nov 11 2019
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 11 November 2019 at 15:00:50 UTC, IGotD- wrote:

 Does this mean if this DIP goes though, a "real" distinctive 
 dynamic array implementation must be done in order facilitate 
 the functionality that has previously been done by D-slices?


Reminds me of the old T[new] thing from like.... dconf 2007 or 
something. What's old is new again.
Nov 11 2019
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 7:48:54 AM MST IGotD- via Digitalmars-d wrote:

 On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee5
 6e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should
 occur in this thread. The review period will end at 11:59 PM ET
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary,
 the DIP will be scheduled for another round of Community
 Review. Otherwise, it will be queued for the Final Review and
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.


 "Enlarging a slice, using the append operator or by setting the
 .length property, makes use of the garbage collector. This
 fosters the incorrect notion that D slices require the GC."

 The title mentions arrays but the rest of the text is about
 slices.

 I need one clarification, this DIP regards slices not arrays? So
 an array is being able to be resized just as before. It's just
 that this will no longer apply for slices?

 If this only regards slices, then I think this is a sound change,
 even if it is breaking. A slice should not really know the
 underlying algorithm how the data is stored. It could be an
 array, it could be part of a buffer or a view into any other
 structure. Limiting the slice so that it cannot grow beyond its
 original boundaries makes sense.

 If this include arrays, that they aren't able to grow then I'm
 against this change but again it doesn't make any sense. Then it
 is not a dynamic array.


Dynamic arrays _are_ slices. T[] is a dynamic array which is a slice of a
block of memory which may or may not be GC-allocated. Some folks do try to
talk like dynamic arrays are the GC-allocated memory block that a slice
refers to, but that's not the definition used by the language. As far as the
language is concerned, T[] is a dynamic array no matter what memory it
refers to, and the language has no specific term for a memory block
allocated by the GC which is sliced by a dynamic array.

This DIP makes it so that it would no longer be possible to use any
operations on a dynamic array which would increase its size, whereas right
now those operations always work in code that isn't  nogc no matter what
memory the dynamic array is a slice of. It's just that in the case where the
dynamic array has no capacity to grow, you always get a reallocation.
Whether the dynamic array referred to GC-allocated memory or not is
irrelevant except that if the memory is not GC-allocated, there definitely
isn't room to grow the dynamic array in-place, whereas if it was
GC-allocated then there might be room to grow in-place. In either case, if
there isn't room to grow in-place, then the GC allocates a new memory block,
copies the elements into the new memory block, and alters the dynamic array
to point to the new memory block.

- Jonathan M Davis
Nov 11 2019
prev sibling next sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
The Rationale of this DIP is lacking IMO.
It highlights a problem, that the DIP will fix, but doesn't 
explain the greater
context of the problem.
The last few DIPs started from a fairly low level IMO.
To me it is not clear where this journey should go.
What is the long term payoff for this pain?
Are we aiming for rust like memory management? C++ like? Python 
like?
Something in between?

I think this needs to addressed before this DIP can be discussed.

I know this pushes the bounds of the review guidelines, but I 
think this
DIP starts a journey in the wrong direction.
IMO D's problem is not the GC nor ~=.
I think it is the "wrong to assume" that manual memory management 
(mmm) should
play nice with the GC.
The GC should be the  safe default.
If you use mmm you made your bed, don't force me to reshuffle my 
GC bed.
I think mmm and safety are very hard to consolidate.
We should spend our resources on making range based, functional, 
programming
in combination with the GC  safe and easy.

Another valid way to solve the problem given in the rationale of 
the DIP would
be to disallow the use of malloc or the cast.
Nov 11 2019
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Monday, 11 November 2019 at 15:11:32 UTC, Robert Schadek wrote:

 

 Another valid way to solve the problem given in the rationale 
 of the DIP would
 be to disallow the use of malloc or the cast.


malloc is also only allowed in  system code.
Nov 11 2019
next sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Monday, 11 November 2019 at 15:24:48 UTC, jmh530 wrote:

 malloc is also only allowed in  system code.


Now I'm even less sure what this DIP tries to achieve in the long 
run.
Nov 11 2019
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 11 November 2019 at 15:29:21 UTC, Robert Schadek wrote:

 Now I'm even less sure what this DIP tries to achieve in the 
 long run.


The question isn't the allocation itself, but rather the slice.

void main() {
    int[5] buffer; // or malloc or whatever
    foo(buffer[]); // this is OK
}

 safe void foo(scope int[] a) {
    a ~= 0; // and this now reallocates on the GC
}



but i'd argue this is the *right* thing to do. and if you don't 
want the GC function, do  nogc on it, now it is a compile error.

If we were saying that T[] shouldn't have a ~= operator at all... 
well i can kinda see the point, you could use a container object 
instead.

But a lot of times the current behavior is perfectly sane. Like 
this example has no memory leak, no out of bounds access, and is 
convenient to use.

If anything maybe we should make  nogc more granular so like have 
independent  nosliceappend or whatever. But like meh.
Nov 11 2019
parent Robert Schadek <rschadek symmetryinvestments.com> writes:
On Monday, 11 November 2019 at 15:56:02 UTC, Adam D. Ruppe wrote:
I was not necessarily trying to make that point.

I was merely trying to say, that IMO, mmm should not be considered
a necessarily desirable concept, especially when  safety is the 
goal.

In other words, approxEqual(!mmm, "Less problems making D  safe")
Nov 11 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 7:24 AM, jmh530 wrote:

 malloc is also only allowed in  system code.


Although malloc() is marked as  system, it actually is  safe. It's free() that 
needs to be  system.
Nov 11 2019
next sibling parent reply Jab <jab_293 gmall.com> writes:
On Tuesday, 12 November 2019 at 03:08:28 UTC, Walter Bright wrote:

 On 11/11/2019 7:24 AM, jmh530 wrote:

 malloc is also only allowed in  system code.


 Although malloc() is marked as  system, it actually is  safe. 
 It's free() that needs to be  system.


You can only really consider malloc  safe cause "void*" is 
completely useless in  safe, and casting it to a different type 
is what is unsafe. By those constraints free() could also be 
considered "safe". If you do cast the pointer to another type, 
then both are not safe.
Nov 11 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 7:35 PM, Jab wrote:

 On Tuesday, 12 November 2019 at 03:08:28 UTC, Walter Bright wrote:

 On 11/11/2019 7:24 AM, jmh530 wrote:

 malloc is also only allowed in  system code.


 Although malloc() is marked as  system, it actually is  safe. It's free() that 
 needs to be  system.


 
 You can only really consider malloc  safe cause "void*" is completely useless
in 
  safe, and casting it to a different type is what is unsafe. By those 
 constraints free() could also be considered "safe". If you do cast the pointer 
 to another type, then both are not safe.


The cast of the pointer returned from malloc() is indeed unsafe, but the 
malloc() call is safe.
Nov 11 2019
prev sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Tuesday, 12 November 2019 at 03:08:28 UTC, Walter Bright wrote:

 Although malloc() is marked as  system, it actually is  safe. 
 It's free() that needs to be  system.


I think that is a mute point.
malloc gives you a void*, you can't really do anything with that 
void*
in  safe code.
You can't even cast it to int*, and it shouldn't.

I know I sound like a broken record, but IMO mmm and  safe is the 
wrong direction.
Nov 12 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, November 12, 2019 1:43:27 AM MST Robert Schadek via Digitalmars-
d wrote:

 On Tuesday, 12 November 2019 at 03:08:28 UTC, Walter Bright wrote:

 Although malloc() is marked as  system, it actually is  safe.
 It's free() that needs to be  system.


 I think that is a mute point.
 malloc gives you a void*, you can't really do anything with that
 void*
 in  safe code.
 You can't even cast it to int*, and it shouldn't.

 I know I sound like a broken record, but IMO mmm and  safe is the
 wrong direction.


Honestly, I don't think that it can really be done without seriously
hamstringing ourselves.

Dynamic arrays and manually managed memory are a _terrible_ combination,
because dynamic arrays do nothing to manage lifetimes. They're really not
any different from passing naked pointers around as far as lifetime tracking
goes. This means that the only way that you can automatically manage that is
to restrict where they can be stored. scope does that, but the result is
that you can pretty much just pass the data into a function that operates on
it and doesn't store it. It doesn't even allow you to do something as simple
as have the function return a range wrapper like a lot of Phobos does. And
real world code typically needs to store stuff, not just pass it around.
Maybe what Walter is looking to do with  live will let it go a bit further,
but I don't see how it could possibly allow anywhere near the freedom you
get when passing around dynamic arrays that are slices of GC-allocated
memory. You end up throwing all kinds of complications into the mix just to
try to get a bit more  safety for non-GC-allocated memory. Maybe some of
those complications are worth it, but at some point, it's just too much
extra machinery for too little benefit, and I'd argue that if it harms
common use patterns of GC-allocated objects (which this DIP definitely
does), then it's going to far.

Personally, I don't think that it makes much sense to try to improve passing
around naked slices of malloc-ed memory. The lack of lifetime management
makes it so that you can't possibly guarantee that it's  safe to free the
memory without severely restricting what's allowed, and in general, I'd
strongly argue that in most cases, it's bad practice to pass around naked
slices of malloc-ed memory unless what the code is deing is restricted
enough that the programmer knows what's being done with that memory so that
they can know when it's safe to free it. Some use of features like scope may
help make it more manageable, but I think that it's folly to expect code to
do something like

auto arr = (cast(int*)malloc(length * int.sizeof))[0 .. length];

pass that memory around, and then later do

free(arr.ptr);

and have that be  safe. Outside of simple examples, to pass around malloc-ed
memory  safely, it needs to be wrapped in a type that manages its lifetime
(e.g. a smart pointer). Language improvements (like DIP 1000) may make it
easier to manage when pointers or references to that memory temporarily
escape the smart pointer in order to operate on it, but to pass around and
store that memory like you'd frequently do with GC-allocated memory, it
needs to be wrapped in something that manages its lifetime, just like you'd
do in languages like C++.

Regardless, even if all agreed that we wanted to improve the  safety of
malloc-ing a naked dynamic array, passing it around, and then freeing it, I
don't see how this DIP really helps with that. As has been pointed out
already in this thread, simply mutating the dynamic array itself screws with
your ability to get the pointer from it to free (showing that you really
should be storing a pointer for that separately regardless of what's going
on with appending), and appending doesn't cause any  safety problems. It
just makes it harder to track whether the dynamic array still refers to any
portion of that malloc-ed memory, but even then, it doesn't make it
impossible to do so, and if you want to forbid appending, you can simply
require that the code be  nogc. So, the DIP fails to even do what it's
trying to do, and in the process, it hurts very common use cases.

- Jonathan M Davis
Nov 12 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 12 November 2019 at 11:12:30 UTC, Jonathan M Davis 
wrote:

 Dynamic arrays and manually managed memory are a _terrible_ 
 combination, because dynamic arrays do nothing to manage 
 lifetimes. They're really not any different from passing naked 
 pointers around as far as lifetime tracking goes. This means 
 that the only way that you can automatically manage that is to 
 restrict where they can be stored. scope does that, but the 
 result is that you can pretty much just pass the data into a 
 function that operates on it and doesn't store it.


One option, albeit unorthodox, is to require heap-stored slices 
to be "concrete" and therefore "encode" the properties of what it 
is referencing.

Whereas you could allow slices used in the call-graph to be 
"abstract" (templated, but without the syntax) and then infer 
their concrete properties using static analysis.

So, the counter-argument would be "combinatorial explosion", but 
if the compiler deduce that the actual function that is being 
compiled generate the same code for a set of concrete slice types 
then you can emit a single copy of the compiled code.

Anyway, that ought to be sufficient to distinguish GC, reference 
counted and RAII objects. And it would also allow D slices to 
differ from C++ slices, but also allow you to write one function 
to cover both scenarios.
Nov 12 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
The idea is to be able to guarantee memory safety. I've spoken about this in my 
presentations and DConf, and that's clearly what DIP25 and DIP1000 were
targeted 
at, and my proposals for an Ownership/Borrowing system for D.

This is where the industry is moving. Even C++ is heading that direction:

   http://open-std.org/JTC1/SC22/WG21/docs/papers/2018/p1179r0.pdf

This proposal is written by Herb Sutter, and is no joke.

We either get on the bus or get run over by the bus.
Nov 11 2019
next sibling parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright wrote:


 We either get on the bus or get run over by the bus.


Frankly speaking, I've read such argumentations a lot of times 
along many years in the D history: the concurrency chapter of the 
D Programming Language is here to testify that this type of 
argumentation should be avoided, in a technical discussion.

Let's move forward from that, and let's try to find a sound 
solution to the memory safety problem, but without killing the 
language in the attempt.

There's no hurry: D is _already_ a safe language, thanks to your 
intuition of 20 years ago to bake the GC.
Nov 11 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 2:33 PM, Paolo Invernizzi wrote:

 On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright wrote:
 

 We either get on the bus or get run over by the bus.


 
 Frankly speaking, I've read such argumentations a lot of times along many
years 
 in the D history: the concurrency chapter of the D Programming Language is
here 
 to testify that this type of argumentation should be avoided, in a technical 
 discussion.
 
 Let's move forward from that, and let's try to find a sound solution to the 
 memory safety problem, but without killing the language in the attempt.
 
 There's no hurry: D is _already_ a safe language, thanks to your intuition of
20 
 years ago to bake the GC.


D is quite a bit safer currently than C or C++, but we've been a bit complacent 
about that advantage and are at risk of falling behind.

I am replying to the question about what is the direction D is going. This is 
the direction and the motivation for it. Whether DIP 1025 is the correct move
in 
that direction or not is certainly worthy of technical discussion.

BTW, after not having memory corruption bugs for years, I've lost days of work 
in the last couple months looking for the cause of two memory corruption 
problems in DMD. I had grown complacent.
Nov 11 2019
parent reply Jab <jab_293 gmall.com> writes:
On Tuesday, 12 November 2019 at 00:51:28 UTC, Walter Bright wrote:

 On 11/11/2019 2:33 PM, Paolo Invernizzi wrote:

 On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright 
 wrote:
 

 We either get on the bus or get run over by the bus.


 
 Frankly speaking, I've read such argumentations a lot of times 
 along many years in the D history: the concurrency chapter of 
 the D Programming Language is here to testify that this type 
 of argumentation should be avoided, in a technical discussion.
 
 Let's move forward from that, and let's try to find a sound 
 solution to the memory safety problem, but without killing the 
 language in the attempt.
 
 There's no hurry: D is _already_ a safe language, thanks to 
 your intuition of 20 years ago to bake the GC.


 D is quite a bit safer currently than C or C++, but we've been 
 a bit complacent about that advantage and are at risk of 
 falling behind.

 I am replying to the question about what is the direction D is 
 going. This is the direction and the motivation for it. Whether 
 DIP 1025 is the correct move in that direction or not is 
 certainly worthy of technical discussion.

 BTW, after not having memory corruption bugs for years, I've 
 lost days of work in the last couple months looking for the 
 cause of two memory corruption problems in DMD.


How have you not had this problem before? DMD's code base is kind 
of horrible. Pointers to arrays that end up having to be accessed 
by (*p)[x]. A simple wrapper could easily have alleviated that 
entirely. It's kind of ironic that the D reference compiler 
showcases exactly how you shouldn't write D code.


 I had grown complacent.


And now you are over eager to make changes to change that 
complacency. These DIPs are being ejected out one after another. 
It seems like everything else is being thrown under the bus on 
this hellbent pursuit of safety.


 ... the former at compile time and the latter at runtime (as a 
 dynamic check is necessary). Hence, a long deprecation period 
 will be necessary.


Of all the things people would be willing to have such a big 
breaking changes for. Auto-decoding to name one. This is now 
where the line is drawn? Some of these decisions are being made 
too hastily IMO.
Nov 11 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 7:03 PM, Jab wrote:

 How have you not had this problem before?


Because I was careful.



 DMD's code base is kind of horrible. 
 Pointers to arrays that end up having to be accessed by (*p)[x]. A simple 
 wrapper could easily have alleviated that entirely. It's kind of ironic that
the 
 D reference compiler showcases exactly how you shouldn't write D code.


DMD was rather recently converted to D from a "C with Classes" codebase. Its 
structure very much reflects that heritage.



 Of all the things people would be willing to have such a big breaking changes 
 for. Auto-decoding to name one. This is now where the line is drawn? Some of 
 these decisions are being made too hastily IMO.


No decision has been made about DIP 1025. That's what this process is for. 
Everyone gets their say.
Nov 11 2019
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 11.11.19 23:28, Walter Bright wrote:

 
 We either get on the bus or get run over by the bus.


That's a false dichotomy. E.g., you could also eat the engine of the bus 
for dinner and then get stranded in the desert with heavy metal poisoning.
Nov 11 2019
prev sibling next sibling parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright wrote:

 The idea is to be able to guarantee memory safety. I've spoken 
 about this in my presentations and DConf, and that's clearly 
 what DIP25 and DIP1000 were targeted at, and my proposals for 
 an Ownership/Borrowing system for D.

 This is where the industry is moving. Even C++ is heading that 
 direction:

   
 http://open-std.org/JTC1/SC22/WG21/docs/papers/2018/p1179r0.pdf

 This proposal is written by Herb Sutter, and is no joke.

 We either get on the bus or get run over by the bus.


By the short bus.
Sorry, couldn't resist.
Nov 12 2019
prev sibling next sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright wrote:

 The idea is to be able to guarantee memory safety. I've spoken 
 about this in my presentations and DConf, and that's clearly 
 what DIP25 and DIP1000 were targeted at, and my proposals for 
 an Ownership/Borrowing system for D.

 This is where the industry is moving. Even C++ is heading that 
 direction:

   
 http://open-std.org/JTC1/SC22/WG21/docs/papers/2018/p1179r0.pdf

 This proposal is written by Herb Sutter, and is no joke.

 We either get on the bus or get run over by the bus.


golang is not going in that direction. They are not  safe but 
mostly because
they have chosen performance over safety.

To me D is great because it is not doing what everybody else is 
doing.
If D was only doing what c++ or rust are doing, D can only become 
as
good as those IMO. But D should be better.
Generative programming in combination with functional 
programming, is
awesome. Hardly anybody is doing that.
I'm not saying we should steal other peoples good ideas.
I'm trying to say we should steal interesting ideas/mistakes and 
improve on them.
For instance taking the c++ template meta-programming mistake and
making it a feature was a huge win.

Additionally, I think there is an elephant in the room. 
Multi-threading.
I just watched a talk on rust async/await.
https://www.youtube.com/watch?v=lJ3NC-R3gSI
It took them ~10 years to get the borrow-checker + multi 
threading thing
done/right, and compared to us they have infinite resources.
Is there a plan for D and the borrow-checker in place/planning?
Nov 12 2019
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 12 November 2019 at 09:12:51 UTC, Robert Schadek 
wrote:

 golang is not going in that direction. They are not  safe but 
 mostly because they have chosen performance over safety.


You have to make an effort to break Go. You can do that with most 
languages...


Python. Like Java it is much more
its own virtual environment (but unix influenced) than a system 
level programming language. Which makes sense for web-services.
Nov 12 2019
parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Tuesday, 12 November 2019 at 10:09:05 UTC, Ola Fosheim Grøstad 
wrote:


 Python. Like Java it is much more
 its own virtual environment (but unix influenced) than a system 
 level programming language. Which makes sense for web-services.


Hm, when I think of go I think of docker/kubernetes.
For me that is systems programming.
I see your point, but I think there are other ways to look at it.
You can actually get Microcontrollers that run python not c.

I think saying, and I'm not saying you say that, systems 
programming therefore
mmm is at least a "Hasty Generalization".
Nov 12 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 12 November 2019 at 10:25:40 UTC, Robert Schadek 
wrote:

 Hm, when I think of go I think of docker/kubernetes.
 For me that is systems programming.
 I see your point, but I think there are other ways to look at 
 it.
 You can actually get Microcontrollers that run python not c.


Yep, that is true.


 I think saying, and I'm not saying you say that, systems 
 programming therefore
 mmm is at least a "Hasty Generalization".


Yes.

I think the long term future even for embedded programming is 
high level languages that allows you to "bind" datastructures to 
low level representations where you need them (but otherwise 
leave it to the compiler).

So yes, how to do systems programming will hopefully evolve over 
time.

But for me there ought to be a separation between high level and 
low level constructs. In C++ all high level types are library 
only, that means the compiler can only do low level optimization. 
D has some room to innovate there, so I certainly am sympathetic 
to the idea that high level features should not be undermined. 
Maybe the question ought to be, can the high level features be 
strenghtened in a way that allows the compiler to be smarter and 
that allows the compiler to "understand" where low level 
assumptions are being made and where it has free hands to be 
smart. So rather than downgrading high level construct, upgrade 
them from runtime-constructs to compiler-constructs.

It is quite intriguing though that something as simple as slices 
can be so complex to figure out. I've written some variations of 
slices for C++ before they added it, and there are so many trade 
offs... Maybe one has to classifiy all the different use 
scenarios for slice-like types and then figure out some solution 
that allows the compiler to be more clever than in other 
languages.
Nov 12 2019
prev sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 12 November 2019 at 09:12:51 UTC, Robert Schadek 
wrote:

 On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright 
 wrote:

 Although malloc() is marked as  system, it actually is  safe. 
 It's free() that needs to be  system.


 I think that is a mute point.


(You mean moot point)


 malloc gives you a void*, you can't really do anything with 
 that void*
 in  safe code.
 You can't even cast it to int*, and it shouldn't.

 I know I sound like a broken record, but IMO mmm and  safe is 
 the wrong direction.


Thats what containers are for, to encapsulate the inherent 
unsafety and expose a safe interface.


 Generative programming in combination with functional 
 programming, is
 awesome. Hardly anybody is doing that.
 I'm not saying we should steal other peoples good ideas.
 I'm trying to say we should steal interesting ideas/mistakes 
 and improve on them.


We should definitely steal them, but we had better a) make sure 
they are good ideas, and b) understand what makes them good. 
Otherwise we risk doing what C++ did with static if/constexpr if, 
UFCS and many more.


 For instance taking the c++ template meta-programming mistake 
 and
 making it a feature was a huge win.

 Additionally, I think there is an elephant in the room. 
 Multi-threading.
 I just watched a talk on rust async/await.
 https://www.youtube.com/watch?v=lJ3NC-R3gSI
 It took them ~10 years to get the borrow-checker + multi 
 threading thing
 done/right, and compared to us they have infinite resources.
 Is there a plan for D and the borrow-checker in place/planning?


That would be DIP1024[1], which follows the same proof of safety 
by induction of  trusted leaf functions for encapsulating safe 
access to members with locks/atomics/etc.

[1]:https://github.com/dlang/DIPs/blob/master/DIPs/DIP1024.md
Nov 12 2019
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 12.11.19 11:37, Nicholas Wilson wrote:

 
 We should definitely steal them, but we had better a) make sure they are 
 good ideas, and b) understand what makes them good. Otherwise we risk 
 doing what C++ did with static if/constexpr if, UFCS and many more.


Actually it will be even worse because we don't follow C++'s 
backwards-compatibility mantra. To me it appears that Walter wants to 
dismantle the GC piece by piece.
Nov 12 2019
prev sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Tuesday, 12 November 2019 at 10:37:44 UTC, Nicholas Wilson 
wrote:

 (You mean moot point)


yes

 Thats what containers are for, to encapsulate the inherent 
 unsafety and expose a safe interface.


Okay, from where I'm standing that sounds like the problem is 
with malloc/slicing
malloced memory, and not so much the GC.
On the other hand, the more functional my D becomes, the less I 
feel the need
for containers at all.


 encapsulating safe access to members with locks/atomics/etc.

 [1]:https://github.com/dlang/DIPs/blob/master/DIPs/DIP1024.md


Thank you, I'll have another read.
Sounds like this whole  safe-borrowing-rc needs a top down design 
DIP to
understand the complete context.
Nov 12 2019
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 12 November 2019 at 11:46:30 UTC, Robert Schadek 
wrote:

 On Tuesday, 12 November 2019 at 10:37:44 UTC, Nicholas Wilson 
 wrote:

 Thats what containers are for, to encapsulate the inherent 
 unsafety and expose a safe interface.


 Okay, from where I'm standing that sounds like the problem is 
 with malloc/slicing
 malloced memory, and not so much the GC.
 On the other hand, the more functional my D becomes, the less I 
 feel the need
 for containers at all.


Perhaps containers is the wrong word. I meant in in the sense 
data structures intersection wrapping, e.g. DList!int or 
Atomic!int. Both of those are inherently unsafe (blobs of 
pointers and shared memory respectively) but the container 
provides a safe way to interact with the data encapsulated by it 
by disallowing outside access in an unchecked manner.
Nov 12 2019
parent Robert Schadek <rschadek symmetryinvestments.com> writes:
On Tuesday, 12 November 2019 at 13:08:08 UTC, Nicholas Wilson 
wrote:

 Perhaps containers is the wrong word. I meant in in the sense 
 data structures intersection wrapping, e.g. DList!int or 
 Atomic!int. Both of those are inherently unsafe (blobs of 
 pointers and shared memory respectively) but the container 
 provides a safe way to interact with the data encapsulated by 
 it by disallowing outside access in an unchecked manner.


I see.
Nov 12 2019
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright wrote:

 The idea is to be able to guarantee memory safety. I've spoken 
 about this in my presentations and DConf, and that's clearly 
 what DIP25 and DIP1000 were targeted at, and my proposals for 
 an Ownership/Borrowing system for D.

 This is where the industry is moving. Even C++ is heading that 
 direction:

   
 http://open-std.org/JTC1/SC22/WG21/docs/papers/2018/p1179r0.pdf

 This proposal is written by Herb Sutter, and is no joke.

 We either get on the bus or get run over by the bus.


I feel like if this proposal is part of the motivation for this 
DIP, then it should be mentioned in the DIP.
Nov 12 2019
prev sibling next sibling parent Uknown <sireeshkodali1 gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.


I get what the DIP is trying to do, but like Jonathon said, this 
solution is simply not worth the cost *at all*. Such a change 
would alienate the existing users, make a lot of good, working 
code break for no real reason, and cause a lot of other issues.

The better solution IMO is to instead set an allocator, either 
per thread or global, like std.experimental.allocator does. Then, 
change the runtime to allocate and expand slices by using that 
allocator, instead of the GC. The allocator could be GC by 
default. Maybe once that is done, slicing arbitrary pointers 
could be depreciated. Then the ` safe` aspect would be solved, 
and there would be practically 0 code breakage.

Something like this:

---a.d

import core.allocator; //hypothetical

void f(int[] x)  safe
{
     x ~= 0;
}

void main()  system
{
     int[] slice = (cast(int*)(malloc(10 * int.sizeof)))[0 .. 10]; 
//deprecated
     this_thread.allocator = Mallocator;
     int[] slice = new int[10];
     f(slice); //now safe, since mallocator is used instead of GC
}
Nov 11 2019
prev sibling next sibling parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.


"This change is a __necessary__ part of evolving D toward being 
memory safe without using a GC"

This is a bold statement, that needs to be addressed with a 
little of explanation of why there are no alternatives.

I think it's not acceptable to suffer such a huge pain, without 
having a clear understanding that there was a deep analysis on 
potential alternative solutions, and an explanation of why they 
are not sufficient for the scope.

Just to be clear, why  nogc is not enough?

Thanks
Nov 11 2019
parent reply Uknown <sireeshkodali1 gmail.com> writes:
On Monday, 11 November 2019 at 16:36:57 UTC, Paolo Invernizzi 
wrote:

 
 "This change is a __necessary__ part of evolving D toward being 
 memory safe without using a GC"

 This is a bold statement, that needs to be addressed with a 
 little of explanation of why there are no alternatives.

 I think it's not acceptable to suffer such a huge pain, without 
 having a clear understanding that there was a deep analysis on 
 potential alternative solutions, and an explanation of why they 
 are not sufficient for the scope.

 Just to be clear, why  nogc is not enough?

 Thanks


A lot of people are bringing it up, so I'll bite. The problem 
with  nogc is that it doesn't cover all cases. Imagine the code 
given in the DIP like this instead:

---lib.d

void f(int x[])  safe pure nothrow
{
     x ~= 0;
}

---main.d

void main()  safe
{
     import lib: f;
     import std.container : Array;

     Array!int x = [0, 1, 2, 3, 4];

     f(slice);
     // x's dtor will try to free an invalid pointer
}

Clearly here main does something that seems safe on the surface. 
But in actuality it is clearly unsafe code. And its hard to 
verify, because main and the libraries used are written by 
completely different people.
Nov 11 2019
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/11/19 11:47 AM, Uknown wrote:

 A lot of people are bringing it up, so I'll bite. The problem with  nogc 
 is that it doesn't cover all cases. Imagine the code given in the DIP 
 like this instead:
 
 ---lib.d
 
 void f(int x[])  safe pure nothrow
 {
      x ~= 0;
 }
 
 ---main.d
 
 void main()  safe
 {
      import lib: f;
      import std.container : Array;
 
      Array!int x = [0, 1, 2, 3, 4];
 
      f(slice);
      // x's dtor will try to free an invalid pointer
 }
 
 Clearly here main does something that seems safe on the surface. But in 
 actuality it is clearly unsafe code. And its hard to verify, because 
 main and the libraries used are written by completely different people.


No, you are misunderstanding a lot here.

1. f(slice), there is no symbol slice, I think maybe you mean x[]?

2. f's x is a *copy*, which means that appending to x here DOES NOT 
AFFECT main's x at all. Main's x will destroy perfectly fine, and all is 
well.

3. If  nogc is added to main, then it won't compile, because f cannot be 
 nogc. Which is quite the point people are making.

-Steve
Nov 11 2019
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 17:00:07 UTC, Steven Schveighoffer 
wrote:

 No, you are misunderstanding a lot here.


This is nitpicking.

If you are going to have a good story with interfacing with C++ 
libraries and frameworks... you don't want this to fail 
accidentally:

ownerpointer = normalize(initialize(allocate()))

Where normalize is a D function that calls other GC library code 
that is well-behaving, except the one that accidentally grows the 
slice because it works in the library-author's pure GC code.

You have to be able to call GC code even if you want to modify 
non-GC memory.


Anyway, it also isn't true that Dynamic Arrays don't have a type. 
It isn't untyped. But the typesystem it uses is completely 
different from the rest of the language.

It appears to be quasi-dynamic effects based, whereas the rest of 
the language is mostly static strongly typed. Meaning, the type 
changes as you use it. Does any other object in D do that?
Nov 11 2019
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 10:12:13 AM MST Ola Fosheim Grøstad via 
Digitalmars-d wrote:

 On Monday, 11 November 2019 at 17:00:07 UTC, Steven Schveighoffer

 wrote:

 No, you are misunderstanding a lot here.


 This is nitpicking.

 If you are going to have a good story with interfacing with C++
 libraries and frameworks... you don't want this to fail
 accidentally:

 ownerpointer = normalize(initialize(allocate()))

 Where normalize is a D function that calls other GC library code
 that is well-behaving, except the one that accidentally grows the
 slice because it works in the library-author's pure GC code.

 You have to be able to call GC code even if you want to modify
 non-GC memory.


 Anyway, it also isn't true that Dynamic Arrays don't have a type.
 It isn't untyped. But the typesystem it uses is completely
 different from the rest of the language.

 It appears to be quasi-dynamic effects based, whereas the rest of
 the language is mostly static strongly typed. Meaning, the type
 changes as you use it. Does any other object in D do that?


How on earth do D dynamic arrays change their type? int[] is always int[]. I
believe that the bit that Steven is talking about with regards to them being
untyped is that inside druntime, it uses void*, because the code isn't
templated - which arguably is something that should be fixed. Outside of
druntime's innards though, a dynamic array always has the same type. All it
really is is a struct containing a pointer and a size_t for length.

- Jonathan M Davis
Nov 11 2019
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/11/19 12:28 PM, Jonathan M Davis wrote:

 On Monday, November 11, 2019 10:12:13 AM MST Ola Fosheim Grøstad via
 Digitalmars-d wrote:

 Anyway, it also isn't true that Dynamic Arrays don't have a type.
 It isn't untyped. But the typesystem it uses is completely
 different from the rest of the language.

 It appears to be quasi-dynamic effects based, whereas the rest of
 the language is mostly static strongly typed. Meaning, the type
 changes as you use it. Does any other object in D do that?


 
 How on earth do D dynamic arrays change their type? int[] is always int[]. I
 believe that the bit that Steven is talking about with regards to them being
 untyped is that inside druntime, it uses void*, because the code isn't
 templated - which arguably is something that should be fixed. Outside of
 druntime's innards though, a dynamic array always has the same type. All it
 really is is a struct containing a pointer and a size_t for length.


auto arr = "hello".dup;

ubyte[] arr2 = cast(ubyte[]) arr;

arr2 ~= 1; // fine appends in-place (possibly)
arr ~= 'a'; // fine, makes a copy

The type doesn't matter to druntime, only the length of data. That is 
the point.

Of course, this all breaks down if the type has a postblit/destructor, 
which IS stored by druntime. But I'm not sure if there are any runtime 
mechanisms to prevent it (I haven't touched that code much since the 
posblit/dtor code was added).

-Steve
Nov 11 2019
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 17:28:09 UTC, Jonathan M Davis 
wrote:

 How on earth do D dynamic arrays change their type? int[] is 
 always int[].


Well, what we mean by "type" can of course be debated, but anyway:

Case 1:
     char[] a = new char[2];
     writeln(a.ptr);
     if (false) a.length = 1;
     a.length = 2;
     writeln(a.ptr);  // same pointer

Case 2:
     char[] a = new char[2];
     writeln(a.ptr);
     if (true) a.length = 1;
     a.length = 2;
     writeln(a.ptr); // different pointer

So, whether it extends inline depends what you did with the 
object dynamically. It isn't determined statically. Even though 
in this case that would be possible.

Kinda like the difference between an open and closed file, 
whether it is open or not depends on some dynamic external cause 
(if it exists on the files system).
Nov 11 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 11:54:04 AM MST Ola Fosheim Grøstad via 
Digitalmars-d wrote:

 On Monday, 11 November 2019 at 17:28:09 UTC, Jonathan M Davis

 wrote:

 How on earth do D dynamic arrays change their type? int[] is
 always int[].


 Well, what we mean by "type" can of course be debated, but anyway:

 Case 1:
      char[] a = new char[2];
      writeln(a.ptr);
      if (false) a.length = 1;
      a.length = 2;
      writeln(a.ptr);  // same pointer

 Case 2:
      char[] a = new char[2];
      writeln(a.ptr);
      if (true) a.length = 1;
      a.length = 2;
      writeln(a.ptr); // different pointer

 So, whether it extends inline depends what you did with the
 object dynamically. It isn't determined statically. Even though
 in this case that would be possible.

 Kinda like the difference between an open and closed file,
 whether it is open or not depends on some dynamic external cause
 (if it exists on the files system).


So what if the pointer's value changed? Objects have their members mutated
all the time. That doesn't make them different types. If you want to know
whether appending to a dynamic array will result in a new buffer being
allocated (with the elements being copied over, and the ptr member of the
dynamic array being mutated), then check whether the capacity is large
enough to contain the elements being appended. If it is, then the GC can
expand the dynamic array in-place. If it isn't, then the GC has to allocate
a new buffer, copy the elements over, and mutate the dynamic array's ptr and
length members to point to the new buffer. And that's the same regardless of
what kind of memory backs the dynamic array (it's just that there's never
enough capacity to grow in-place if the underlying buffer wasn't allocated
by the GC for dynamic arrays). I don't understand why you'd think that a
dynamic array being in a different state based on what you did with it would
in any way make it a different type. It's perfectly normal for the behavior
of an object to change based on its state.

- Jonathan M Davis
Nov 11 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 19:18:53 UTC, Jonathan M Davis 
wrote:

 I don't understand why you'd think that a dynamic array being 
 in a different state based on what you did with it would in any 
 way make it a different type. It's perfectly normal for the 
 behavior of an object to change based on its state.


Well, the conceptual state was exactly the same when accessed. So 
it is unusual for a core language type.

Anyway, there is no point in arguing this, as we first have to 
agree on what we mean by the word "type" at runtime vs "type" at 
compile time. Which would be very off topic...

(You could create a compile time ubertype that includes all other 
possible types, so that nothing can be said about any object at 
compile time, but you would still think that those objects had 
various types at runtime.)
Nov 11 2019
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/11/19 12:12 PM, Ola Fosheim Grøstad wrote:

 On Monday, 11 November 2019 at 17:00:07 UTC, Steven Schveighoffer wrote:

 No, you are misunderstanding a lot here.


 
 This is nitpicking.
 


Sorry, I'm not sure why it's nitpicking. The example shown does not show 
what the author wanted it to show.

When you have a function:

void f(int x[])  safe pure nothrow
{
     x ~= 0;
}

There is no way to call that function in a way that will break safety, 
or alter any external data. It's pure, only the shallowly passed in 
slice is affected, nothing outside. This is truly a misunderstanding by 
the author (Uknown) of what a slice actually is.

The original example in the DIP shows more of a problem, but  nogc is 
certainly a solution there, which is why people are bringing it up.


 If you are going to have a good story with interfacing with C++ 
 libraries and frameworks... you don't want this to fail accidentally:
 
 ownerpointer = normalize(initialize(allocate()))
 
 Where normalize is a D function that calls other GC library code that is 
 well-behaving, except the one that accidentally grows the slice because 
 it works in the library-author's pure GC code.


True, GC-using code could be valid to call if it doesn't affect your 
type. But  nogc IS a valid way to use slices and ensure that they never 
use the GC to grow.

But in fact, a better solution is to make sure the TYPE prevents it, and 
we absolutely have the mechanisms today to do this. An opt-in type 
(ironically, such as Array), would be sufficient to make sure you can 
still call gc-using code, without mucking with your type's ability to 
free the original pointer.

If the DIP said it wanted to introduce a type that enforces it will 
always point to the resource that is managed, that would be fine. It 
just can't overtake T[] to mean that. It's way way too disruptive. I'd 
be fine with new syntax as well (like T[new] or whatnot).

-Steve
Nov 11 2019
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 17:30:25 UTC, Steven Schveighoffer 
wrote:

 Sorry, I'm not sure why it's nitpicking. The example shown does 
 not show what the author wanted it to show.


Alright, alright, I thought it was obvious what he meant, but I 
guess that is because I already agreed with his core viewpoint.



 True, GC-using code could be valid to call if it doesn't affect 
 your type. But  nogc IS a valid way to use slices and ensure 
 that they never use the GC to grow.


Yes, but you need to be careful so you don't split the ecosystem 
in two. So people should be able to call the best libraries they 
can find (GC or not).

I understand that this is difficult, but I think the intent of 
this DIP is pointing in the right direction, even if it isn't 
bulletproof or ready.  So I hope it isn't dismissed outright.



 But in fact, a better solution is to make sure the TYPE 
 prevents it


Yes, a type that protects the underlying buffer somehow makes a 
lot of sense.

But then you need to encourage templated slicing somehow, so that 
you can write generic code that takes all kinds of slicing-types 
(and use introspection perhaps to choose how they are used).

Not so easy to get right, I guess... Hm.


 If the DIP said it wanted to introduce a type that enforces it 
 will always point to the resource that is managed, that would 
 be fine. It just can't overtake T[] to mean that. It's way way 
 too disruptive. I'd be fine with new syntax as well (like 
 T[new] or whatnot).


Ok, I understand.

Maybe a type that will share the binary interface of C++ span and 
string_view. They are supposed to be the ones to use in function 
C++ function signatures to increase interoperability between 
libraries etc.

Right now it is a bit difficult to say how span<> will be adopted 
in C++ libraries, but I would imagine that it might end up being 
used for matrices as well "span<span<T>>". So binary 
compatibility could make sense.
Nov 11 2019
parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 11 November 2019 at 18:13:45 UTC, Ola Fosheim Grøstad 
wrote:

 Maybe a type that will share the binary interface of C++ span 
 and string_view. They are supposed to be the ones to use in 
 function C++ function signatures to increase interoperability 
 between libraries etc.

 Right now it is a bit difficult to say how span<> will be 
 adopted in C++ libraries, but I would imagine that it might end 
 up being used for matrices as well "span<span<T>>". So binary 
 compatibility could make sense.


(A bit OT)

We tried to convince Walter to make extern(C++) void f(int[]) 
Just Workâ„¢, to no avail, because he though that it would b 
"unintuitive" for people to have include a header to have to use 
it. Go figure.

I doubt that span<span<T>> will be widely used, because jagged 
matrices are a bit of an odd choice of structure, 
span<string_view> might get a bit of use though.
Nov 11 2019
prev sibling parent reply Uknown <sireeshkodali1 gmail.com> writes:
On Monday, 11 November 2019 at 17:00:07 UTC, Steven Schveighoffer 
wrote:

 On 11/11/19 11:47 AM, Uknown wrote:

 A lot of people are bringing it up, so I'll bite. The problem 
 with  nogc is that it doesn't cover all cases. Imagine the 
 code given in the DIP like this instead:
 
 ---lib.d
 
 void f(int x[])  safe pure nothrow
 {
      x ~= 0;
 }
 
 ---main.d
 
 void main()  safe
 {
      import lib: f;
      import std.container : Array;
 
      Array!int x = [0, 1, 2, 3, 4];
 
      f(slice);
      // x's dtor will try to free an invalid pointer
 }
 
 Clearly here main does something that seems safe on the 
 surface. But in actuality it is clearly unsafe code. And its 
 hard to verify, because main and the libraries used are 
 written by completely different people.


 No, you are misunderstanding a lot here.

 1. f(slice), there is no symbol slice, I think maybe you mean 
 x[]?


yes, my mistake


 2. f's x is a *copy*, which means that appending to x here DOES 
 NOT AFFECT main's x at all. Main's x will destroy perfectly 
 fine, and all is well.


You're right, I shouldn't have used Array. I was just trying to 
point out how libraries could obscure such bugs. A different 
container with different semantics which exposes slices could 
cause issues.


 3. If  nogc is added to main, then it won't compile, because f 
 cannot be  nogc. Which is quite the point people are making.

 -Steve


The problem is that once the slice is created, there's no way to 
know how the underlying memory was allocated. And if you want to 
append to the slice, then that's hard. Because there's no way to 
append generically in the language without the GC. Don't get me 
wrong, I think this DIP is the wrong solution. But there is an 
issue to be addressed. We need a way to be able to append without 
invoking the GC. I personally think std.experimental.allocators + 
some other associated language modifications are the way forward.
Nov 11 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 11:03:35 AM MST Uknown via Digitalmars-d wrote:

 The problem is that once the slice is created, there's no way to
 know how the underlying memory was allocated. And if you want to
 append to the slice, then that's hard. Because there's no way to
 append generically in the language without the GC. Don't get me
 wrong, I think this DIP is the wrong solution. But there is an
 issue to be addressed. We need a way to be able to append without
 invoking the GC. I personally think std.experimental.allocators +
 some other associated language modifications are the way forward.


If you want to append to a dynamic array without using the GC, then just
allocate a new block of memory with malloc or whatever, copy the elements
from the dynamic array into it along with the elements being appended, slice
that block of memory, and assign that dynamic array to the first one. Of
course, you then have to code worrying about when it's safe to free that
memory, but if your code needs something like reference counting to manage
the memory, then it probably shouldn't be passing around dynamic arrays
anyway. Dynamic arrays which are slices of malloc-ed memory or static arrays
or whatnot work just fine so long as the code that does the slicing can know
when the dynamic arrays which are slices of that memory go out of scope so
that it can know when it's safe to free that memory, but if the dynamic
arrays being passed around in a way that the code doing the allocating can't
know when that memory isn't referenced anymore, then dynamic arrays are a
bad fit - and that's the case whether this DIP is accepted or not. If the
dynamic arrays are being passed around in a way that you don't know what
their lifetimes are, then they need to be GC allocated.

- Jonathan M Davis
Nov 11 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 18:15:13 UTC, Jonathan M Davis 
wrote:

 accepted or not. If the dynamic arrays are being passed around 
 in a way that you don't know what their lifetimes are, then 
 they need to be GC allocated.


In the Prior Work section the DIP states:

«Rust's notion of pointers owning the memory they point to, 
unless the pointer is borrowed, is equivalent with this DIP's 
notion of a slice "borrowing" a reference to another object which 
manages the memory.»

So it does seem to imply that slices are borrowed views.

Conceptually, dynamic arrays would be borrowed from the GC, thus 
you don't need to know their lifetimes.

Lifetimes would come into play for non-dynamic-arrays, I guess, 
so scope... perhaps.
Nov 11 2019
prev sibling next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/11/19 5:27 AM, Mike Parker wrote:

 This is the feedback thread for the first round of Community Review for 
 DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
 
 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee
6e4/DIPs/DIP1025.md 
 
 
 All review-related feedback on and discussion of the DIP should occur in 
 this thread. The review period will end at 11:59 PM ET on November 25, 
 or when I make a post declaring it complete.
 
 At the end of Round 1, if further review is deemed necessary, the DIP 
 will be scheduled for another round of Community Review. Otherwise, it 
 will be queued for the Final Review and Formal Assessment.
 
 Anyone intending to post feedback in this thread is expected to be 
 familiar with the reviewer guidelines:
 
 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md
 
 *Please stay on topic!*
 
 Thanks in advance to all who participate.


This is a question of terminology. The dynamic array type in D is 
specified to be a slice. This has extreme benefits that I think we all 
will lament if this goes away.

As I've said before, treating the slice type as the array type is super 
confusing and not technically true. A dynamic array owns its own data, 
the slice does not. In D, the dynamic array is created and maintained 
*typeless* in the GC. It's owned by the GC. A slice can point at a 
dynamic array, stack-allocated data, global/thread-local data, a 
malloc-allocated array, or any other piece of memory that you can 
possibly point at. As soon as you append to it, it NOW points at a 
GC-allocated array. This should be neither surprising, nor a problem, IF 
you adopt the right terminology.

A fix here could be to actually DEFINE the dynamic array type, and allow 
the user to fetch it, and understand how it works. You can continue to 
allow the slice to be an interface to it, or you can use it directly. 
Please email me if you want more details.

I 100% agree with Adam, just put  nogc if you don't want to use the GC. 
This is a huge breaking change, and the much better solution is to make 
the desired behavior opt-in. A middle ground could be a newly defined 
"slice" type that can be used to opt-in.

I'd say no to this change, unless we are talking D3. And if that's the 
case, let's start talking about all the other things we want to fix ;)

-Steve
Nov 11 2019
prev sibling next sibling parent reply Jon Degenhardt <jond noreply.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md


The DIP includes a discussion of std.array.appender:


 A workaround for deprecating:
 
 a ~= b;
 is to use std.array.appender ...


The DIP should describe whether this includes both 
std.array.Appender and std.array.RefAppender or just 
std.array.Appender.
Nov 11 2019
parent Jon Degenhardt <jond noreply.com> writes:
On Monday, 11 November 2019 at 18:01:03 UTC, Jon Degenhardt wrote:

 On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md


 The DIP includes a discussion of std.array.appender:


 A workaround for deprecating:
 
 a ~= b;
 is to use std.array.appender ...


 The DIP should describe whether this includes both 
 std.array.Appender and std.array.RefAppender or just 
 std.array.Appender.


To clarify my previous comment, std.array.appender (lower-case) 
is a convenience function that can create both std.array.Appender 
and std.array.RefAppender. However, it is not obvious that 
std.array.RefAppender would be supported after acceptance of this 
DIP.
Nov 11 2019
prev sibling next sibling parent reply Meta <jared771 gmail.com> writes:
I've never been comfortable with the fact that T[] can refer to 
any type of memory, GC-allocated or not (or T*, for that matter), 
but I think this DIP is not the right approach. It severely 
pessimizes one of the most common use cases of slices in order to 
prevent memory corruption when the slice does not refer to a GC'd 
array. While this could be a big gain, it is also definitely a 
huge loss in terms of how much code would be broken.

I completely agree, though, that the fact that ~ may change 
slices referring to non-GC'd memory to GC'd memory is not a good 
thing - I just think this DIP would break too much code.

I can't see a clear way forward, either, because at a fundamental 
level in the type system, D treats GC'd and non-GC'd pointers the 
same. Therefore, it would not be enough to introduce a new type 
that exclusively refers to GC-allocated memory (i.e., new T* / 
T[new], or whatever), because slicing would still have to be 
disabled for non-GC'd pointers, and appending would still have to 
be disabled for non-GC'd slices.

Would it be enough to put an assert in GC.realloc/extend/free 
that asserts the memory is GC-owned, maybe tied to a compiler 
switch?
Nov 11 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 12:50 PM, Meta wrote:

 Would it be enough to put an assert in GC.realloc/extend/free that asserts the 
 memory is GC-owned, maybe tied to a compiler switch?


Currently, for `a~=b` if `a` is not owned by the GC, it is copied to a new GC 
location by the runtime.
Nov 11 2019
parent Meta <jared771 gmail.com> writes:
On Monday, 11 November 2019 at 23:20:31 UTC, Walter Bright wrote:

 On 11/11/2019 12:50 PM, Meta wrote:

 Would it be enough to put an assert in GC.realloc/extend/free 
 that asserts the memory is GC-owned, maybe tied to a compiler 
 switch?


 Currently, for `a~=b` if `a` is not owned by the GC, it is 
 copied to a new GC location by the runtime.


Yes, my point being, if the concern is over manually-allocated 
pointers being invalidated by accidental reallocation by the GC, 
wouldn't the absolute least intrusive change possible be to put 
asserts in the relevant GC methods to make sure it's only 
reallocating/extending/freeing memory *it* owns? It would mean a 
crash at runtime only, and only in debug mode, but you get a 
semi-nice stack trace showing you the exact point you're trying 
to do something bad (assuming I properly understand the GC 
implementation, which I may not).
Nov 11 2019
prev sibling next sibling parent 12345swordy <alexanderheistermann gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.


I genuinely believe this is a wrong approach to this. The code 
breakage is potentially massive on this. It would be better to 
create a new dynamic array that only shrinks and never grow as a 
library solution.

Why can't his be done via a library?

-Alex
Nov 11 2019
prev sibling next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.


As noted by the other reviewers,
- the restrictions proposed by the DIP imply much breaking 
changes.
- the problem is a lack of distinction between slices and dynamic 
arrays.

To that I also add that
- the restrictions would be disruptive for a certain usage of D, 
especially shell-like programs, manipulating array of files, 
array of directories, program arguments, etc.

so maybe that

1. At the local scope or when this is semantically possible, the 
compiler could keep track of the difference between a dynamic 
array and a slice, preventing growing.
2. When not possible anymore, e.g function parameter, a  slice 
attribute could be used to apply the same restriction as in 1.

---
void func1(int[] arr)
{
     arr ~= 1;               // OK
     arr.length += 1;        // OK
     auto slice = arr[0..2];
     slice.length += 1;      // NG, local restriction (1.)
}

void func2( slice int[] arr)
{
     arr ~= 1;               // NG,  slice attribute (2.)
     arr.length += 1;        // NG,  slice attribute (2.)
     auto slice = arr[0..2];
     slice.length += 1;      // NG, local restriction (1.)
}
---

As an alternative, instead of 2. the new semantics could be 
applied only if the function is  safe (although this would give 
less control over individual parameters)

---
void func2(int[] arr)  // or  trusted
{
     arr ~= 1;               // OK, not  safe or  trusted
     arr.length += 1;        // OK, not  safe or  trusted
     auto slice = arr[0..2];
     slice.length += 1;      // NG, local restriction (1.)
}

void func3(int[] arr)  safe
{
     arr ~= 1;               // NG, not allowed in  safe func
     arr.length += 1;        // NG, not allowed in  safe func
     auto slice = arr[0..2];
     slice.length += 1;      // NG, local restriction (1.)
}
---

Of course ideally a full 1. would be better but then this 
requires complex DFA, which DMD doesn't know how to do yet, 
unless I'm not well informed.
Nov 11 2019
parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 11 November 2019 at 21:49:52 UTC, Basile B. wrote:


 2. When not possible anymore, e.g function parameter, a  slice 
 attribute could be used to apply the same restriction as in 1.


O God, please no! Pretty frankly, writing D cose is starting to 
became a masterclass of attribute handling: with move semantic, 
rvalue, live, and now that, I've practically lost the count!
Nov 11 2019
parent Basile B. <b2.temp gmx.com> writes:
On Monday, 11 November 2019 at 21:59:36 UTC, Paolo Invernizzi 
wrote:

 On Monday, 11 November 2019 at 21:49:52 UTC, Basile B. wrote:


 2. When not possible anymore, e.g function parameter, a  slice 
 attribute could be used to apply the same restriction as in 1.


 O God, please no! Pretty frankly, writing D cose is starting to 
 became a masterclass of attribute handling: with move semantic, 
 rvalue, live, and now that, I've practically lost the count!


TBH me too.  safe could be used instead but then much more 
breakage to expect.
Nov 11 2019
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 11.11.19 11:27, Mike Parker wrote:

 This is the feedback thread for the first round of Community Review for 
 DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
 
 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee
6e4/DIPs/DIP1025.md 
 
 
 All review-related feedback on and discussion of the DIP should occur in 
 this thread. The review period will end at 11:59 PM ET on November 25, 
 or when I make a post declaring it complete.
 
 At the end of Round 1, if further review is deemed necessary, the DIP 
 will be scheduled for another round of Community Review. Otherwise, it 
 will be queued for the Final Review and Formal Assessment.
 
 Anyone intending to post feedback in this thread is expected to be 
 familiar with the reviewer guidelines:
 
 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md
 
 *Please stay on topic!*
 
 Thanks in advance to all who participate.


Wrong season. April Fools' Day is in April.
Nov 11 2019
prev sibling next sibling parent Suleyman <sahmi.soulaimane gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
 [...]


I also find it a problem that dynamic arrays do not carry their 
allocator in their type, hence the problem with reallocation. It 
works for a GC only language and many people think of D like this 
and are not complaining about it, so their voice should be given 
the importance that it deserves. In addition, D also wants to 
compete in convenience with scripting languages. Clearly there is 
a good argument on both sides. A separation between a slice and 
an array which knows its allocator is needed, but the convenience 
hit is too big for those who never in their life called malloc, 
so it's hard to make a good decision on this.

In general, D has made good decisions so far regarding this 
issue. For people who just want the convenience then things are 
great, the GC is dominant, malloc is  system so they need to be 
careful with it. And for those who want to control memory 
allocation they have to use  nogc.

The problem comes when these two worlds talk to each other, the 
 nogc world and the convenience world, the medium they use is a 
slice. But I don't find it that much of a problem, since the 
convenience world will never manually allocate or free anything, 
and the  nogc world cannot assume anything about who allocated a 
slice, that's the beauty of the slice, that it's very portable.
Nov 11 2019
prev sibling next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.


I understand where this proposal comes from, but it would 
certainly cause a lot of breakage and it would cripple the 
universal feature common to all programming languages, arrays. 
Without filling the void this proposal leaves behind, I don't see 
any support for it.

I would personally look at the conflation of dynamic arrays and 
slices. If you could somehow separate the 2 concepts and THEN 
constrain slices to only be able to shrink, that might work. (or 
keep them conflated but make borrowed slices non appendable.) (or 
make the implicit explicit by somehow requiring `.array` before 
appending).

On the other hand, the fact that they are conflated also brings 
benefits, and you certainly don't want to loose them. It is a 
tough nut to crack.

OT:

In general I agree with the sentiment D's language features 
oftentimes depend too much on an all-knowing garbage collector. 
Maybe there is a way that behaviour could be made customisable 
for people who need that, instead of having everyone bear the 
costs. (e.g. I have done a lot of betterC programming lately, and 
while I can use my own custom allocators, things like closures 
still depend on a GC.)
Nov 12 2019
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 12 November 2019 at 08:41:52 UTC, Sebastiaan Koppe 
wrote:

 I would personally look at the conflation of dynamic arrays and 
 slices. If you could somehow separate the 2 concepts and THEN 
 constrain slices to only be able to shrink, that might work. 
 (or keep them conflated but make borrowed slices non 
 appendable.) (or make the implicit explicit by somehow 
 requiring `.array` before appending).

 On the other hand, the fact that they are conflated also brings 
 benefits, and you certainly don't want to loose them. It is a 
 tough nut to crack.


I'm starting to think that the issue is not that they are 
conflated, but that it isn't clear whether "dynamic" GC arrays  
are values/object, high-level/low-level or 
language-constructs/runtime-constructs.

If concatenation expressions work with high-level values 
conceptually, and slices are views on those values, and GC-arrays 
are high-level-nonruntime-constucts, then things might become 
more clear.

Then leave to the optimizer to estimate and preallocate a buffer 
size (or with hints).

I don't think there is anything wrong with having high level 
concepts, but then they should be truly high level so that the 
optimizer can do the work and there should be some clear 
delineation between high-level and  low-level types. (You could 
still allow low level operations temporarily, but the life-time 
of that would have to be made formal, so that the optimizer has 
free hands when the low-level work is done).

But then you also need another low-level concept for views on an 
address-bound object/buffer/memory.

So it is a tough nut, yes.



 In general I agree with the sentiment D's language features 
 oftentimes depend too much on an all-knowing garbage collector.


Yes, but is also cultural, in the sense that some people rely 
100% on the garbage collector. So how can  the type system ensure 
that the high quality libraries they write are usable in the more 
general programming context?

If all programmers only relied on the GC by 30%, then maybe this 
would be less of an issue, as they would think differently about 
the use context of the library.  The rather weakly typed C++ 
ecosystem works because there is some shared understanding of the 
use context and conventions.

So that is an extra challenge for D. Maybe static analysis could 
help. Some innovation that can prove properties down the call 
chain fromt he call site (rather than all those attributes on the 
function signatures).
Nov 12 2019
prev sibling parent IGotD- <nise nise.com> writes:
On Tuesday, 12 November 2019 at 08:41:52 UTC, Sebastiaan Koppe 
wrote:

 I would personally look at the conflation of dynamic arrays and 
 slices. If you could somehow separate the 2 concepts and THEN 
 constrain slices to only be able to shrink, that might work.


Yes, this DIP needs to be suspended until we know what to do with 
the divorce between dynamic arrays and slices.
Nov 12 2019
prev sibling next sibling parent Tim <tim.dlang t-online.de> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.


If you create a slice from manually managed memory, you probably 
also have to make sure the slice does not outlive the memory. 
This means the slice has to be passed as scope to functions (or 
some pure functions). The DIP should only apply to scope slices, 
because other slices are probably allocated on the GC.
Nov 12 2019
prev sibling next sibling parent reply Dukc <ajieskola gmail.com> writes:
If we are to do this, we need to limit the breakage somehow. A 
few ideas, that could perhaps be used in some combination:

-Instead of disallowing appending on arrays, change the garbage 
collector so that it'll never append in place. Of course, this 
has the potential to be a performance killer so alone it won't be 
enough.

-Only disable GC appending in  safe or  live

-Come up with a way to mark some code or module to diallow GC 
appending, but do not disable it everywhere. Something that won't 
require yet another attribute in every function declaration.

-`scope` slices can be appended on.

Rust is already infamous for being hard to get into because of 
the borrow checker. So we definitely should not implement 
everything it does the same way. This DIP in it's current form 
would cause so much damage that if we fall behind because we 
won't do it, so be it.
Nov 12 2019
parent reply Jab <jab_293 gmall.com> writes:
On Tuesday, 12 November 2019 at 12:30:42 UTC, Dukc wrote:

 This DIP in it's current form would cause so much damage that 
 if we fall behind because we won't do it, so be it.


It is going to fall behind either way. This DIP focuses on 
falling behind Rust/C++ safety with manual memory management. But 
it completely ignores falling behind on simplicity/easy of 
use/unrestricted use with other languages that have a GC. The 
focus seems to only be on Rust/C++ lately. By trying to be safe 
with manual memory management you lose what the GC offers, this 
is in part due to the fact that GC pointers and malloc pointers 
are indistinguishable. It's a clash between two different 
ideologies that don't mix well.
Nov 12 2019
next sibling parent bachmeier <no spam.net> writes:
On Tuesday, 12 November 2019 at 17:09:30 UTC, Jab wrote:

 On Tuesday, 12 November 2019 at 12:30:42 UTC, Dukc wrote:

 This DIP in it's current form would cause so much damage that 
 if we fall behind because we won't do it, so be it.


 It is going to fall behind either way. This DIP focuses on 
 falling behind Rust/C++ safety with manual memory management. 
 But it completely ignores falling behind on simplicity/easy of 
 use/unrestricted use with other languages that have a GC. The 
 focus seems to only be on Rust/C++ lately. By trying to be safe 
 with manual memory management you lose what the GC offers, this 
 is in part due to the fact that GC pointers and malloc pointers 
 are indistinguishable. It's a clash between two different 
 ideologies that don't mix well.


D will fall behind in terms of stability. The language is 20 
years old. It shouldn't be breaking the thousands and thousands 
of lines of code I've already written unless we're moving to D 
3.0 over a period of several years. It's hard to see how the 
language recovers from making a move like this with no planning 
and no communication. I don't see how anyone could recommend 
using D after something like this, except maybe to people you 
really don't like and aren't your boss.
Nov 12 2019
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 12.11.19 18:09, Jab wrote:

 On Tuesday, 12 November 2019 at 12:30:42 UTC, Dukc wrote:

 This DIP in it's current form would cause so much damage that if we 
 fall behind because we won't do it, so be it.


 
 It is going to fall behind either way. This DIP focuses on falling 
 behind Rust/C++ safety with manual memory management. But it completely 
 ignores falling behind on simplicity/easy of use/unrestricted use with 
 other languages that have a GC.


I agree with this.


 The focus seems to only be on Rust/C++ 
 lately. By trying to be safe with manual memory management you lose what 
 the GC offers, this is in part due to the fact that GC pointers and 
 malloc pointers are indistinguishable.


That's why the language needs to provide tools like borrowing, such that 
libraries can easily implement  trusted smart pointers that carry around 
allocator/ownership information. In  safe code, any accessible raw 
pointer always has to have unlimited lifetime. (In Rust, raw pointers 
are entirely inaccessible in safe code, because Rust has no built-in GC.)


 It's a clash between two different ideologies that don't mix well.


I don't agree with this. We can have both. We can enforce mutable 
aliasing constraints for memory accessible through shared, e.g., tracing 
GC-owned, pointers using runtime checks. (Rust also does this if you 
want to mutate memory through shared, e.g., reference counted, pointers.)
Nov 12 2019
prev sibling next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md


I don't know.
C++ has std::vector and std::string_view, but our slices do these 
two things. It makes things easier sometimes but in the end you 
have to track ownership in your head, especially if you use  nogc 
and -betterC.

Does "dynamic arrays ARE slices" really hurt us?

This change would break a great number of libraries, as said in 
the thread. Personally I have zero interest in more memory safety.

On the other hand, maybe we can go "leap of faith" on this.
It would also help if there was a good std::vector replacement in 
the stdlib, with explicit .capacity.
Nov 12 2019
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 12 November 2019 at 13:10:06 UTC, Guillaume Piolat 
wrote:

 [snip]

 I don't know.
 C++ has std::vector and std::string_view, but our slices do 
 these two things. It makes things easier sometimes but in the 
 end you have to track ownership in your head, especially if you 
 use  nogc and -betterC.

 Does "dynamic arrays ARE slices" really hurt us?

 [snip]


I keep getting a little mixed up when referring to slices and 
dynamic arrays. So some
T[] x;
is a dynamic array and
auto y = x[a..b];
or
auto y = x[];
is a slicing operation on x. And right now, slicing x also 
returns a dynamic array unless there is some opSlice defined that 
does something different.

So one option would be to keep dynamic arrays as they are and 
change the slicing operations to return a slice type. They would 
be like dynamic arrays with the exception that you can't append 
to them or change the length. However, this would probably lead 
to a lot of breakage, even if you could have a .array function 
that can easily convert a slice to an array.

What about instead something like below with a type that has 
appending disabled. I had tried to make length const, but I 
figured it was easier to just return data's length. I'm not sure 
if this works in the more general case though. Regardless, the 
benefit is that this is completely opt-in and requires no 
code-breakage.

```
struct View(T : U[], U) {
     T data;
     alias data this;

     this(T x) {
      	data = x;
     }

      disable void opOpAssign(string op)(U rhs)
         if (op == "~")
     {
         data ~= rhs;
     }

      disable void opOpAssign(string op)(U[] rhs)
         if (op == "~")
     {
         data ~= rhs;
     }

      disable T[] opBinary(string op)(U rhs)
         if (op == "~")
     {
         return data ~ rhs;
     }

      disable T[] opBinary(string op)(U[] rhs)
         if (op == "~")
     {
         return data ~ rhs;
     }

     size_t length() {
      	return data.length;
     }
}

auto view(T : U[], U)(T x) {
	return View!(T, U)(x);
}

void main() {
     int[] x = [0, 3, 5];
     auto y = x[0..2].view;
}
```
Nov 12 2019
parent jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 12 November 2019 at 15:59:57 UTC, jmh530 wrote:

 [snip]

      disable T[] opBinary(string op)(U rhs)
         if (op == "~")
     {
         return data ~ rhs;
     }

      disable T[] opBinary(string op)(U[] rhs)
         if (op == "~")
     {
         return data ~ rhs;
     }

 [snip]


Small error here
T[] opBinary
should be
T opBinary
Nov 12 2019
prev sibling next sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md


See here for my response: 
https://gist.github.com/John-Colvin/c9c0b79bc9d47ed8d57ec9e959d0542b


Included below for convenience and record:


The DIP provides 2 examples to justify itself:

1) unpredictable mutable aliasing after appends

2) unpredictable changes of ownership caused by appends

They can share the same cause (`~=`), but are two distinct 
symptoms, and can also have other causes. They have a unifying 
aspect which is unpredictable aliasing.

The breakage of the DIP is immense, so there needs to be a 
commensurately large upside and no reasonable alternative path 
with less breakage.




```D
int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
slice ~= 1;
free(slice.ptr); // Oops!
```

`free` is not an ` safe` operation and requires some conditions 
to be met for correct use. These conditions are as follows:
  - `free` happens after all accesses to the allocation referenced 
by the passed pointer.
  - the allocation referenced by the passed pointer was made by 
`{m,c,re}alloc`

as such, you can't really ever call `free` without completely 
trusting the path by which the data travelled, from inception 
(`{m,c,re}alloc`) to the call to `free`. This may be achieved by 
a variety of ways (e.g. a `MallocSlice` type that encapsulates 
the slice, or by having the path be very short and doing it 
manually) but fundamentally if I write a function taking a slice 
and then freeing the pointer, I cannot mark it as ` trusted`, 
regardless of whether `~=` is allowed or not.

That `~=` will cause a new array to be allocated unpredictably is 
not the problem, the problem is that it can happen at all. If we 
look at it probabilistically: the problem is not that the 
probability of re-allocation is in `(0, 1)`, the problem is that 
it's not `0`. *Someone can write `slice = slice ~ 1` and get the 
same behaviour, so my considerations when calling `free` have not 
changed by disallowing `~=`.*



```D
enum { dead, alive }
int[] cat = new int[6];
cat[5] = alive;
int[] b = cat;
b ~= 1;      // may or may not move b to new location
b[5] = dead; // indeterminate whether cat[5] is dead or alive
```

This is a sticky one. `~=` applied to data where there are >1 
references and >0 are mutable is liable to cause problems of 
unpredictable action at a distance. Removing `~=` makes it 
slightly harder to do this, but seeing as it's so very easy to do 
the same thing in other ways (`a = b ? a : (a ~ 2)` and many 
other trivial examples) it arguably doesn't matter much. I could 
believe it makes it less likely to do by accident.



Overall, this change seems to not buy anyone much, but costs a 
lot. It correctly identifies a problem (or 2 problems, depending 
how you  look at it), but the proposed solution does not appear 
on a simple reading to solve the problem.

The DIP does not provide any examples of what is directly enabled 
by the change and I was not able to infer it from the current 
text.

*The author should explain what previously impossible/unsafe code 
can now be made possible/safe given the proposed change.*
Nov 12 2019
parent reply Jab <jab_293 gmall.com> writes:
On Tuesday, 12 November 2019 at 15:54:22 UTC, John Colvin wrote:

 *The author should explain what previously impossible/unsafe 
 code can now be made possible/safe given the proposed change.*


Looking at the examples again, do they even show memory 
corruption?


This would be a memory leak, not memory corruption:

   int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
   slice ~= 1;
   free(slice.ptr); // Oops!



This doesn't show memory corruption either, its a potential logic 
bug as it doesn't consider a side effect of array concatenation.


   enum { dead, alive }
   int[] cat = new int[6];
   cat[5] = alive;
   int[] b = cat;
   b ~= 1;      // may or may not move b to new location
   b[5] = dead; // indeterminate whether cat[5] is dead or alive


Neither of these examples are pertinent to  safe, which is aimed 
at reducing/removing memory corruption.

This would be a more relevant problem for  safe:

   int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
   free(slice.ptr);
   slice ~= 1;       // use after free

As far as I'm aware none of the current previous DIPs resolve 
this do they?
Nov 12 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 12 November 2019 at 17:25:06 UTC, Jab wrote:

 This would be a memory leak, not memory corruption:

   int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
   slice ~= 1;
   free(slice.ptr); // Oops!


Depends on the implementation of the garbage collector and C free.

If it frees the beginning of block that the GC has allocated 
using malloc then it would indeed cause memory corruption as soon 
at somebody else get the same block from malloc.
Nov 12 2019
prev sibling next sibling parent Tobias Pankrath <tobias pankrath.net> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":


https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md




I am against it.

1. One of D's main selling points to me is productivity. We 
currently do not have the machinery in place to be as productive 
without the GC as with it. For example std.allocator is still 
experimental and there seems to be no movement on that front.

First put the alternatives in place in the meantime there is 
 nogc.

2.  nogc

3. If we remove arrays the language has no useful 
builtin/included container anymore. std.container is no 
alternative, e.g.: https://issues.dlang.org/show_bug.cgi?id=13791
Nov 12 2019
prev sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":


This review round is now officially terminated. The DIP author 
has withdrawn the DIP from consideration. I've marked the DIP as 
Withdrawn and summarized the review, as well as the author's 
reason for withdrawal:

https://github.com/dlang/DIPs/blob/master/DIPs/other/DIP1025.md

I've also updated the DIP process documentation to describe the 
Withdrawn status:

https://github.com/dlang/DIPs/blob/master/docs/process-reviews.md#withdrawn-and-superseded-dips
Nov 13 2019
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 13.11.19 10:07, Mike Parker wrote:

 On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community Review 
 for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":


 
 This review round is now officially terminated. The DIP author has 
 withdrawn the DIP from consideration. I've marked the DIP as Withdrawn 
 and summarized the review, as well as the author's reason for withdrawal:
 
 https://github.com/dlang/DIPs/blob/master/DIPs/other/DIP1025.md
 
 I've also updated the DIP process documentation to describe the 
 Withdrawn status:
 
 https://github.com/dlang/DIPs/blob/master/docs/process-reviews.md#withdrawn
and-superseded-dips 
 


Thanks!
Nov 13 2019
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, November 13, 2019 2:07:04 AM MST Mike Parker via Digitalmars-d 
wrote:

 On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:

 This is the feedback thread for the first round of Community



 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":


 This review round is now officially terminated. The DIP author
 has withdrawn the DIP from consideration. I've marked the DIP as
 Withdrawn and summarized the review, as well as the author's
 reason for withdrawal:

 https://github.com/dlang/DIPs/blob/master/DIPs/other/DIP1025.md

 I've also updated the DIP process documentation to describe the
 Withdrawn status:

 https://github.com/dlang/DIPs/blob/master/docs/process-reviews.md#withdraw
 n-and-superseded-dips


Thank you, Walter.

- Jonathan M Davis
Nov 13 2019