www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP 1025--Dynamic Arrays Only Shrink, Never Grow--Community Review

reply Mike Parker <aldacron gmail.com> writes:
This is the feedback thread for the first round of Community 
Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

All review-related feedback on and discussion of the DIP should 
occur in this thread. The review period will end at 11:59 PM ET 
on November 25, or when I make a post declaring it complete.

At the end of Round 1, if further review is deemed necessary, the 
DIP will be scheduled for another round of Community Review. 
Otherwise, it will be queued for the Final Review and Formal 
Assessment.

Anyone intending to post feedback in this thread is expected to 
be familiar with the reviewer guidelines:

https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

*Please stay on topic!*

Thanks in advance to all who participate.
Nov 11 2019
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 3:27:26 AM MST Mike Parker via Digitalmars-d 
wrote:
 This is the feedback thread for the first round of Community
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e
 4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should
 occur in this thread. The review period will end at 11:59 PM ET
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, the
 DIP will be scheduled for another round of Community Review.
 Otherwise, it will be queued for the Final Review and Formal
 Assessment.

 Anyone intending to post feedback in this thread is expected to
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.
Ouch. This would be a _huge_ breaking change. The ability to grow dynamic arrays is used all over the place. It's also one of the great benefits of D's dynamic array IMHO. The fact that you append to a dynamic array without caring where it came from can be incredibly useful, and if you really care about checking whether it has the capacity to grow or whether it's managed by the GC, we have functions to check for that. How would this interact with stuff like assumeSafeAppend? There are code bases that take advantage of the combination of assumeSafeAppend so that they don't have to worry about managing allocations (since ~= takes care of it all), but they can still reuse the dynamic array and thus avoid reallocations if the GC-managed memory block that the dynamic array is a slice of is large enough to hold whatever data is appended. It looks to me like change would break the code in all such code bases. As for Appender, it's designed around the idea that you're going to build an array and then use it rather than you're going to grow it or shrink it as needed, so it doesn't work anywhere near as well when dealing with an array shrinking, let alone growing and shrinking. It also does not work for the use case where you're passing dynamic arrays around and then appending to them, since then you don't have access to the Appender, just the dynamic array. Also, the ability to append to a dynamic array without worrying about whether it allocates or not such that the array can grow in place when it can (thus avoiding unnecessary allocations) but still have reallocations occur when it needs to is something that isn't going to work with Appender. In my experience, code operating on strings in particular takes advantage of being able to slice and append to dynamic arrays with impunity - which works particularly well with immutable, since then you don't have to worry about any of the characters being mutated. Changing such code to use Appender or ~ would be far more complicated and is likely to cause a lot more unnecessary allocations to occur. Also, I fail to see how getting rid of the ability to append and reallocate dynamic arrays really helps with managing memory when the memory that the dynamic array is a slice of is not allocated by the GC. Dynamic arrays aren't reference-counted. You can have slices throughout the code which all refer to the same memory even if they never do an append operation. All of the tools that you'd need to use in terms of scope or pure or coding conventions or whatever to know whether a slice of that memory exists anywhere (and thus whether it's safe to free it) would be the same whether it can be appended to or not. And if you want to disable the ability to append to a dynamic array, you can always just mark the code as nogc. Why do programmers who _do_ want to use the GC in their code need to be punished by making dynamic array worse, when anyone who doesn't want to use the GC can already explicitly add restrictions forbidding it? Honestly, this seems like a prime case for how trying to make D work better with code that doesn't want to use the GC is making D worse for code that does want to use the GC. I really, really, really hope that this change does not make it into the language. From where I sit, it would make the language hands down worse. In fact, as far as usability goes, it would make D's strings _worse_ than C++'s strings. Sure, we'd still be able to get substrings more efficiently than C++ can, but not being able to append to a string makes it a pretty terrible string. At least C++'s strings can do that efficiently. IMHO, where D's dynamic arrays and strings sit right now as far as memory allocation goes is the sweet spot. They're efficient at both growing and shrinking. If I wanted a language that was catering to the non-GC case over the GC case, I'd use C++. Improvements to D that make it work better with nogc code without compromising code that uses the GC seems like a good idea to me, but this very much compromises the language for code that uses the GC. _Please_ don't do this. - Jonathan M Davis
Nov 11 2019
next sibling parent bachmeier <no spam.net> writes:
On Monday, 11 November 2019 at 11:17:26 UTC, Jonathan M Davis 
wrote:

 Honestly, this seems like a prime case for how trying to make D 
 work better with code that doesn't want to use the GC is making 
 D worse for code that does want to use the GC. I really, 
 really, really hope that this change does not make it into the 
 language. From where I sit, it would make the language hands 
 down worse. In fact, as far as usability goes, it would make 
 D's strings _worse_ than C++'s strings. Sure, we'd still be 
 able to get substrings more efficiently than C++ can, but not 
 being able to append to a string makes it a pretty terrible 
 string. At least C++'s strings can do that efficiently.

 IMHO, where D's dynamic arrays and strings sit right now as far 
 as memory allocation goes is the sweet spot. They're efficient 
 at both growing and shrinking. If I wanted a language that was 
 catering to the non-GC case over the GC case, I'd use C++. 
 Improvements to D that make it work better with  nogc code 
 without compromising code that uses the GC seems like a good 
 idea to me, but this very much compromises the language for 
 code that uses the GC. _Please_ don't do this.
I'll move on from D if this change is incorporated. It's a terrible idea that would break code that shouldn't be broken. It would also be a sign that D is effectively giving up on GC in an attempt to compete with Rust, and as such is no longer suited for general purpose programming.
Nov 11 2019
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 11:17:26 UTC, Jonathan M Davis 
wrote:
 D's strings _worse_ than C++'s strings. Sure, we'd still be 
 able to get substrings more efficiently than C++ can, but not 
 being able to append to a string makes it a pretty terrible 
 string. At least C++'s strings can do that efficiently.
C++17 differentiate between the owner of the string: https://en.cppreference.com/w/cpp/string/basic_string And the "borrowed" slice of a string: https://en.cppreference.com/w/cpp/string/basic_string_view In C++20 this is extended to contiguous buffers with std::span.
Nov 11 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 3:17 AM, Jonathan M Davis wrote:
 Ouch. This would be a _huge_ breaking change. The ability to grow dynamic
 arrays is used all over the place. It's also one of the great benefits of
 D's dynamic array IMHO. The fact that you append to a dynamic array without
 caring where it came from can be incredibly useful, and if you really care
 about checking whether it has the capacity to grow or whether it's managed
 by the GC, we have functions to check for that.
While: a ~= b; would no longer be allowed, a = a ~ b; would be. Yes, it is a breaking change, but the fix for when that behavior is desired is pretty simple.
Nov 11 2019
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/11/19 4:32 PM, Walter Bright wrote:
 On 11/11/2019 3:17 AM, Jonathan M Davis wrote:
 Ouch. This would be a _huge_ breaking change. The ability to grow dynamic
 arrays is used all over the place. It's also one of the great benefits of
 D's dynamic array IMHO. The fact that you append to a dynamic array 
 without
 caring where it came from can be incredibly useful, and if you really 
 care
 about checking whether it has the capacity to grow or whether it's 
 managed
 by the GC, we have functions to check for that.
While:    a ~= b; would no longer be allowed,    a = a ~ b; would be. Yes, it is a breaking change, but the fix for when that behavior is desired is pretty simple.
This is not the same. Appending has amortized cost, while concatenation is not. Expect to have D applications which use this approach for updating explode in runtime. -Steve
Nov 11 2019
prev sibling next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
The rationale is severely lacking: Any issue related to the GC is completely orthogonal to any issues with memory safety. The first example is buggy, but is system because it uses free, and thus the onus is on the user. The second example is well defined because ~= copies if the capacity is exceeded[1]. The prior work section lists no references and does not present any logical connection to the ideas presented in the rationale. The breaking changes are massive and not well argued for. The suggested workaround of 1) a = a ~ b; in place of 2) a ~= b; is functional identical, thus any issues present in 2 are also present in 1. The suggestion that doing so in a loop would create more garbage is false as it is functionally identical. The author should revisit the core assumptions made and rewrite the DIP accordingly with references for prior art. If the issue in example one is considered bad enough to warrant doing something about then that issue should be tacked specifically, starting perhaps with a druntime / GC option to later statistics/log when an unmanaged slice is appended to. [1]: void main() { import std.stdio; enum { dead, alive } foreach (i;0 .. 10000) { int[] cat = new int[i+1]; cat[i] = alive; writeln(cat[i]); int[] b = cat; b ~= 1; b[i] = dead; writeln(cat[i]); if (i % 1000) { import core.memory; GC.collect(); } } }
Nov 11 2019
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Monday, 11 November 2019 at 11:26:49 UTC, Nicholas Wilson 
wrote:
 is functional identical, thus any issues present in 2 are also 
 present in 1. The suggestion that doing so in a loop would 
 create more garbage is false as it is functionally identical.
There is a difference: a = a ~ b is guaranteed to make a copy while a ~= b might just use spare capacity of the memory that slice `a` refers to. https://dlang.org/spec/arrays.html#array-concatenation This is what the dead/alive example is supposed to express. It is still true however that the rationale is not very convincing. The first example indeed doesn't show any benefits of the DIP: ``` int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10]; slice = slice ~ 1; // now guaranteed to make a copy free(slice.ptr); // Still oops ``` The following claim in the DIP is also unsubstantiated:
 This change is a necessary part of D evolving towards being 
 memory safe without using
 a GC.
I would like to see an example of memory corruption in ` safe` code that can happen because of slice appending.
Nov 11 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 5:13 AM, Dennis wrote:
 On Monday, 11 November 2019 at 11:26:49 UTC, Nicholas Wilson wrote:
 is functional identical, thus any issues present in 2 are also present in 1. 
 The suggestion that doing so in a loop would create more garbage is false as 
 it is functionally identical.
There is a difference: a = a ~ b is guaranteed to make a copy while a ~= b might just use spare capacity of the memory that slice `a` refers to. https://dlang.org/spec/arrays.html#array-concatenation
That's right. I had overlooked that.
 This is what the dead/alive example is supposed to express.
 It is still true however that the rationale is not very convincing.
 The first example indeed doesn't show any benefits of the DIP:
 
 ```
 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 slice = slice ~ 1; // now guaranteed to make a copy
 free(slice.ptr); // Still oops
 ```
Imagine these 3 lines are spread out over a large code base.
 The following claim in the DIP is also unsubstantiated:
 
 This change is a necessary part of D evolving towards being memory safe 
 without using
 a GC.
Memory safety cannot be achieved without control over who is the owner of memory.
 I would like to see an example of memory corruption in ` safe` code that can 
 happen because of slice appending.
trusted void myfree(void* p) { free(p); } free(slice.ptr);
Nov 11 2019
next sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright wrote:
 On 11/11/2019 5:13 AM, Dennis wrote:
 [...]
That's right. I had overlooked that.
 [...]
Imagine these 3 lines are spread out over a large code base.
 [...]
Memory safety cannot be achieved without control over who is the owner of memory.
 [...]
trusted void myfree(void* p) { free(p); } free(slice.ptr);
That's not safe code, it's trusted, it's not mechanically checkable, and all the horrors summoned in a programmers mind can happens here. My impression is that this is not the point at all.
Nov 11 2019
prev sibling next sibling parent Exil <Exil gmall.com> writes:
On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright wrote:
 This is what the dead/alive example is supposed to express.
 It is still true however that the rationale is not very 
 convincing.
 The first example indeed doesn't show any benefits of the DIP:
 
 ```
 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 slice = slice ~ 1; // now guaranteed to make a copy
 free(slice.ptr); // Still oops
 ```
Imagine these 3 lines are spread out over a large code base.
You realize he changed the code right? That is still valid code that exhibits the same problem even after this DIP gets integrated into DMD.
 The following claim in the DIP is also unsubstantiated:
 
 This change is a necessary part of D evolving towards being 
 memory safe without using
 a GC.
Memory safety cannot be achieved without control over who is the owner of memory.
So is the GC being removed from D? Is this the ultimate goal? Cause the convenience of the language that is provided by using a GC is being removed along with these DIPs. At some point there's not going to be a reason to the GC because all of its' conveniences will have been removed at this rate.
Nov 11 2019
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 11.11.19 22:41, Walter Bright wrote:
 
 Memory safety cannot be achieved without control over who is the owner 
 of memory.
The GC is a perfectly fine owner of memory. Please take a step back and stop gutting the language.
Nov 11 2019
prev sibling next sibling parent Dennis <dkorpel gmail.com> writes:
On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright wrote:
 That's right. I had overlooked that.
It looked like you were aware since you mentioned "although that will generate more GC garbage if used in a loop" when referring to the workaround of `a = a ~ b`.
 Imagine these 3 lines are spread out over a large code base.
With or without this DIP, that would be problematic.
 On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright 
 wrote:
 Memory safety cannot be achieved without control over who is 
 the owner of memory.

 I would like to see an example of memory corruption in ` safe` 
 code that can happen because of slice appending.
trusted void myfree(void* p) { free(p); } free(slice.ptr);
free cannot be trusted on a raw pointer type. In that definition, you can (from safe code) give `myFree` a pointer to a local variable, global variable or GC pointer, all of which may corrupt memory. (And I'm ignoring the lack of live here. Without live you can obviously add double free / use-after-free to that list.) The only way to make `free` safe is to encapsulate it in a struct or class that ensures the pointer that is being freed was allocated with malloc. If you have a custom type, then concatenation like ~= will not work unless you implement it, so disabling it on the built-in slice type does not do much good. In fact, you can consider ~= only syntactic sugar for a call to _d_arrayappendcTX. You can disable the ~= construct, but everywhere you would use that operator you can still do something functionally equivalent and change what kind of data (i.e. stack memory, heap memory) the slice refers to. So the current proposal helps discouraging certain code, but it does not prevent anything.
Nov 11 2019
prev sibling next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 2:41:14 PM MST Walter Bright via Digitalmars-d 
wrote:
 Memory safety cannot be achieved without control over who is the owner of
 memory.
And how does this DIP help with that? Dynamic arrays do not know or care where their memory came from, and they do not track lifetimes in any way, shape, or form. Doing anything that might append to them then guarantees that they're allocated by the GC, because that's the only way that they can grow. Without doing one of those operations, you would have to actually look at the ptr value of the dynamic array and ask the GC whether it's owned by the GC. Getting rid of the append operations does not change that. It's still impossible to know who owns the memory by simply looking at the type, and code that passes slices of non-GC allocated memory around has to be aware of the lifetime of those dynamic arrays and what's done with them to know whether any dynamic arrays referring to that memory still exist. The type system doesn't help with that beyond stuff like scope and pure limiting what can be done with the dynamic array that gets passed in. And getting rid of the append operations doesn't change that. Sure, right now, if you pass a dynamic array which is a slice of malloc-ed memory to a function, that function could append to that dynamic array and result in a reallocation. Why is that a problem? It's not like that's going to cause a memory leak unless it was your only pointer to that memory, and if that's the case, having the dynamic array shrink on you would be just as big a problem, if not bigger. Either way, you have to know what the code is doing with the dynamic array if you're going to try to determine when it's safe to free that memory, because dynamic arrays can normally be passed around and copied with impunity, and they're not reference-counted. Their very nature pretty requires that if you're going to have a dynamic array which slices non-GC-allocated memory, you have to carefully control what you do with it, because nothing about the dynamic array manages the lifetime of the memory it points to. That's the case whether appending is allowed or not. If for some reason, it is actually a problem for a piece of code if a dynamic array ever gets appended to, then requiring that the code be nogc fixes that. It's then not possible to pass the dynamic array to any functions which might append to it. Really, code that's using dynamic arrays which are slices of anything other than GC-allocated memory has to be very limited in how it passes them around regardless of whether appending is possible, because dynamic arrays don't manage their own memory at all. Code that wants to do much passing around of malloc-ed memory to anything other than a pure function where it's clear that the slices can't be squirreled away in any of the arguments or return value is going to tend to need to wrap that memory in something other than a dynamic array in order to manage its lifetime - either that, or it needs to be written in a way that the programmer can clearly know that anything that even might contain a reference to that memory is long gone before they're looking to free that memory (e.g. because it's allocated at the beginning of the program and freed at the end). It looks to me like you're crippling the normal use case in attempt to deal with a less common one. And it doesn't look to me like it's really fixing anything for that less common use case. Honestly, if it really is the case that we have to lose the ability to append to dynamic arrays in order to get safety for malloc-ed memory, then I'd rather give up on safety for malloc-ed memory and keep the ability to append to dynamic arrays. But from what I can tell, the only argument that the DIP has for why appending to dynamic arrays even could be a problem is because it might be problematic if a dynamic array which slices non-GC-allocated memory ends up being copied into newly allocated GC memory, because it's appended to. I dispute that that's a real problem, and even if it is one, nogc prevents it without any language changes. It should be clear to anyone seriously using dynamic arrays that int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10]; slice ~= 1; free(slice.ptr); // Oops! is a problem. It's a given that appending can cause a reallocation. But even if appending weren't allowed, you could still have something like int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10]; slice = slice[1 .. $]; free(slice.ptr); // Oops! and have problems, because the dynamic array isn't slicing the beginning of that block of memory anymore. So, allowing a dynamic array to shrink (or even be assigned to at all) causes similar problems as appending in this case. Either way, relying on the dynamic array itself to know which pointer to use to free that memory is just plain bad practice unless the code is definitely never going to do anything that would mutate the dynamic array itself (be it slicing, appending, or assignment from another array). In general, if code is going to slice malloc-ed memory to get a dynamic array, it should be keeping around a separate pointer which refers to the block that was allocated so that it can free it when no more dynamic arrays exist which point to that memory rather than relying on the dynamic array for that information. Either way, the general inability to track the lifetime of dynamic arrays outside of things like scope and pure means that you can't rely on the type system to track the lifetime of malloc-ed memory outside of functions that simply take an argument and return a result (with scope and pure used to ensure that no slices of that memory are holed away somewhere). As for code such as enum { dead, alive } int[] cat = new int[6]; cat[5] = alive; int[] b = cat; b ~= 1; // may or may not move b to new location b[5] = dead; // indeterminate whether cat[5] is dead or alive So what? It's a given that appending to a dynamic array could cause a reallocation. If that's a problem, the code can check the array's capacity first to see if appending would cause a reallocation. This behavior of arrays is nothing new and isn't specific to dynamic arrays which are slices of memory which wasn't allocated by the GC. If the code actually cares about whether a reallocation takes place, it can easily check whether it will or has taken place, and it can easily prevent it from taking place by requiring that the code be nogc. The DIP does nothing to show why this behavior is actually a problem for safe code. - Jonathan M Davis
Nov 11 2019
parent Andrea Fontana <nospam example.com> writes:
On Monday, 11 November 2019 at 22:40:44 UTC, Jonathan M Davis 
wrote:
 On Monday, November 11, 2019 2:41:14 PM MST Walter Bright via 
 Digitalmars-d wrote:
 It should be clear to anyone seriously using dynamic arrays that

 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 slice ~= 1;
 free(slice.ptr); // Oops!

 is a problem. It's a given that appending can cause a 
 reallocation. But even if appending weren't allowed, you could 
 still have something like

 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 slice = slice[1 .. $];
 free(slice.ptr); // Oops!

 and have problems, because the dynamic array isn't slicing the 
 beginning of that block of memory anymore.
+1 It was my first thought
Nov 12 2019
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright wrote:
 On 11/11/2019 5:13 AM, Dennis wrote:
 ```
 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 slice = slice ~ 1; // now guaranteed to make a copy
 free(slice.ptr); // Still oops
 ```
Imagine these 3 lines are spread out over a large code base.
People doing manual memory management without RAII deserve the codebase complexity they get.
 The following claim in the DIP is also unsubstantiated:
 
 This change is a necessary part of D evolving towards being 
 memory safe without using
 a GC.
Memory safety cannot be achieved without control over who is the owner of memory.
 I would like to see an example of memory corruption in ` safe` 
 code that can happen because of slice appending.
trusted void myfree(void* p) { free(p); } free(slice.ptr);
As noted elsewhere that is a broken trusted, and therefore a user fault, not a language fault. This is no different to breaking the type system in any other way.
Nov 11 2019
prev sibling parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Monday, 11 November 2019 at 21:41:14 UTC, Walter Bright wrote:
 On 11/11/2019 5:13 AM, Dennis wrote:
 On Monday, 11 November 2019 at 11:26:49 UTC, Nicholas Wilson 
 wrote:
 is functional identical, thus any issues present in 2 are 
 also present in 1. The suggestion that doing so in a loop 
 would create more garbage is false as it is functionally 
 identical.
There is a difference: a = a ~ b is guaranteed to make a copy while a ~= b might just use spare capacity of the memory that slice `a` refers to. https://dlang.org/spec/arrays.html#array-concatenation
That's right. I had overlooked that.
 This is what the dead/alive example is supposed to express.
 It is still true however that the rationale is not very 
 convincing.
 The first example indeed doesn't show any benefits of the DIP:
 
 ```
 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 slice = slice ~ 1; // now guaranteed to make a copy
 free(slice.ptr); // Still oops
 ```
Imagine these 3 lines are spread out over a large code base.
 The following claim in the DIP is also unsubstantiated:
 
 This change is a necessary part of D evolving towards being 
 memory safe without using
 a GC.
Memory safety cannot be achieved without control over who is the owner of memory.
I thought the GC knows if the memory is in its control or not? Why not prohibit the concatenation after the GC checked if it is memory it has no view over. That slice was allocated outside or points on the stack so stop with an error. It's not compile time, but better than nothing, but doesn't break all the apps that want to live a simple life of GC memory handling. If I had wanted to make my programmers life difficult I would have chosen Rust or C++.
 I would like to see an example of memory corruption in ` safe` 
 code that can happen because of slice appending.
trusted void myfree(void* p) { free(p); } free(slice.ptr);
cheating doesn't count. trusted in the same vein ad casting, is explicit security overriding, you get what you called for.
Nov 11 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 3:26 AM, Nicholas Wilson wrote:
 The breaking changes are massive and not well argued for.
 The suggested workaround of
 
 1) a = a ~ b;
 
 in place of
 
 2) a ~= b;
 
 is functional identical, thus any issues present in 2 are also present in 1.
The point is making it obvious that the GC is being used, i.e. it is intentional behavior.
Nov 11 2019
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 2:34:39 PM MST Walter Bright via Digitalmars-d 
wrote:
 On 11/11/2019 3:26 AM, Nicholas Wilson wrote:
 The breaking changes are massive and not well argued for.
 The suggested workaround of

 1) a = a ~ b;

 in place of

 2) a ~= b;

 is functional identical, thus any issues present in 2 are also present
 in 1.
The point is making it obvious that the GC is being used, i.e. it is intentional behavior.
Both ~ and ~= use the GC, and it should be obvious that they do, since they can't work if they don't. Using ~ isn't any clearer than ~=, and ~ has worse performance when a dynamic array is able to grow in-place. - Jonathan M Davis
Nov 11 2019
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/11/19 4:54 PM, Jonathan M Davis wrote:
 On Monday, November 11, 2019 2:34:39 PM MST Walter Bright via Digitalmars-d
 wrote:
 On 11/11/2019 3:26 AM, Nicholas Wilson wrote:
 The breaking changes are massive and not well argued for.
 The suggested workaround of

 1) a = a ~ b;

 in place of

 2) a ~= b;

 is functional identical, thus any issues present in 2 are also present
 in 1.
The point is making it obvious that the GC is being used, i.e. it is intentional behavior.
Both ~ and ~= use the GC, and it should be obvious that they do, since they can't work if they don't. Using ~ isn't any clearer than ~=, and ~ has worse performance when a dynamic array is able to grow in-place.
Yes. And the more I see this example, the worse job it does as an example of why this DIP is needed. In order for the DIP to guarantee the pointer of a slice is still the pointer to memory to be freed, you have to *disable assignment*. Do we want to do that too? It also must disable popFront, or really any operation except for opIndex. Here's another example of an error like the item in the DIP, which is totally possible even with the DIP: nogc: // just to emphasize int[] slice = (cast(int*)malloc(10 * int.sizeof))[0 .. 10]; version(bad) { int[1] arr; slice = arr[]; } else version(alsoBad) { slice = slice [1 .. $]; // slice doesn't grow! } free(slice.ptr); // Oops; -Steve
Nov 11 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 22:04:17 UTC, Steven Schveighoffer 
wrote:
 In order for the DIP to guarantee the pointer of a slice is 
 still the pointer to memory to be freed, you have to *disable 
 assignment*. Do we want to do that too?

 It also must disable popFront, or really any operation except 
 for opIndex.
Actually, I think the example I gave was bad, since the DIP states that a slice is for borrowing, and then it should not be used for ownership. I retract it. If a slice is a strictly borrowed view then you can have popFront etc, but not extending it. This would basically be the same as C++ span with dynamic extents. Then you would need something else for ownership of non-GC memory that can provide borrowed slices. And yet again something else for write-buffer-views (e.g. a view on a field in a struct that should be written to, reallocation is bad in that situation). So you need basically 3 different types. I think…
Nov 11 2019
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 21:54:23 UTC, Jonathan M Davis 
wrote:
 Both ~ and ~= use the GC, and it should be obvious that they 
 do, since they can't work if they don't. Using ~ isn't any 
 clearer than ~=, and ~ has worse performance when a dynamic 
 array is able to grow in-place.
a~b is easier to deal with in code analysis than a~=b because the latter has unknown aliasing properties. The compiler knows more when you can either say "will always reallocate" or "will never reallocate". An optimizer could always do it in place if it is known to be safe. Especially in a loop or in a series of concatenations.
Nov 11 2019
prev sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 11 November 2019 at 21:34:39 UTC, Walter Bright wrote:
 On 11/11/2019 3:26 AM, Nicholas Wilson wrote:
 The breaking changes are massive and not well argued for.
 The suggested workaround of
 
 1) a = a ~ b;
 
 in place of
 
 2) a ~= b;
 
 is functional identical, thus any issues present in 2 are also 
 present in 1.
The point is making it obvious that the GC is being used, i.e. it is intentional behavior.
and ~= doesn't? As noted elsewhere there is no safety issues with it because the GC will copy the slice or fill up available capacity. This entire line of reasoning is a non-sequiter.
Nov 11 2019
prev sibling next sibling parent FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.
In my homebrew D-lite language, I used T[~] as the built-in type for "an array that carries capacity information, owns memory (T[auto~] but nevermind that) and can be appended to." This seemed to work well. I definitely think that if we deprecate ~, such a type should be built-in, because this is vital functionality. appender is just too weird a type (it's a weird class-struct hybrid??) to carry this functionality. Simple things must be easy. We don't have to import std.array for T[], we shouldn't have to import it for T[~].
Nov 11 2019
prev sibling next sibling parent reply Dukc <ajieskola gmail.com> writes:
I'll repeat what I said in the draft review, because I don't know 
whether my feedback wasn't noticed. In addition, I'll be more 
explicit.

In no practical way we can implement this. The breakage would 
most likely dwarf breaking autodecoding.

However, the above only applies to breaking appending to slices 
in general. I agree that if the slice is not either:

1. GC-allocated
2. referring to a static array.
3. referring to an initialization value of an array of `static` 
storage class

...then the chances of somebody appending to it are low enough 
that breaking might be viable. Even then, somebody might append 
to a mallocated slice by GC if there is another reference to the 
original slice lying around, but for me that sounds rare enough 
that finding memory leaks could justify the breakage.

I see two alternatives. We can simply discourage growing slices 
with the garbage collector. A transition switch would be 
introduced, but no removing `~` or `slice.length += 2` totally, 
or at least not for many years. We also could introduce a new 
appending operator, say >< (along with ><=), which behaves just 
like ~ but refuses to work with slices, and encourage to use that 
istead of the present append operator.

The other alternative is that slice growing by GC would check 
where the array is referring to. First, if it refers to null, the 
static memory area or the stack, check passed. Second, if it 
refers to anything the GC controls, check passed. Otherwise, it's 
assumed it's pointing to mallocated memory, or just corrupt - 
program terminated with an unrecoverable error. For  system code, 
the compiler could forgo these checks with the same rules as 
array bounds checks can be forgoed. Growing a slice of mallocated 
memory where the check is omitted would be undefined behaviour.
Nov 11 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 4:33:00 AM MST Dukc via Digitalmars-d wrote:
 I'll repeat what I said in the draft review, because I don't know
 whether my feedback wasn't noticed. In addition, I'll be more
 explicit.

 In no practical way we can implement this. The breakage would
 most likely dwarf breaking autodecoding.

 However, the above only applies to breaking appending to slices
 in general. I agree that if the slice is not either:

 1. GC-allocated
 2. referring to a static array.
 3. referring to an initialization value of an array of `static`
 storage class

 ...then the chances of somebody appending to it are low enough
 that breaking might be viable. Even then, somebody might append
 to a mallocated slice by GC if there is another reference to the
 original slice lying around, but for me that sounds rare enough
 that finding memory leaks could justify the breakage.

 I see two alternatives. We can simply discourage growing slices
 with the garbage collector. A transition switch would be
 introduced, but no removing `~` or `slice.length += 2` totally,
 or at least not for many years. We also could introduce a new
 appending operator, say >< (along with ><=), which behaves just
 like ~ but refuses to work with slices, and encourage to use that
 istead of the present append operator.

 The other alternative is that slice growing by GC would check
 where the array is referring to. First, if it refers to null, the
 static memory area or the stack, check passed. Second, if it
 refers to anything the GC controls, check passed. Otherwise, it's
 assumed it's pointing to mallocated memory, or just corrupt -
 program terminated with an unrecoverable error. For  system code,
 the compiler could forgo these checks with the same rules as
 array bounds checks can be forgoed. Growing a slice of mallocated
 memory where the check is omitted would be undefined behaviour.
IMHO, the ability to append to a dynamic array without caring whether it's backed by GC-allocated memory, malloc-ed memory, stack allocated memory, or whatever is a huge benefit. Right now, you can pass a dynamic array that isn't backed by the GC to a function, and it all just works even if the function needs to do a reallocation to make the array large enough. Doing so is not necessarily a bug and allows you to use non-GC allocated memory where appropriate while still being able to grow the array, and it does so without the code having to care about what kind of memory is currently backing the dynamic array. And if the code _does_ care for some reason, the array's capacity property will tell you whether the array can be grown in-place, and multiple functions in core.memory can be used to determine whether the dynamic array is a slice of GC-allocated memory. - Jonathan M Davis
Nov 11 2019
parent reply Dukc <ajieskola gmail.com> writes:
On Monday, 11 November 2019 at 11:51:09 UTC, Jonathan M Davis 
wrote:
 IMHO, the ability to append to a dynamic array without caring 
 whether it's backed by GC-allocated memory, malloc-ed memory, 
 stack allocated memory, or whatever is a huge benefit.
Yeah... And I definitely think all of these should continue to exist with the possible exception of malloced memory.
 Right now, you can pass a dynamic array that isn't backed by 
 the GC to a function, and it all just works even if the 
 function needs to do a reallocation to make the array large 
 enough. Doing so is not necessarily a bug and allows you to use 
 non-GC allocated memory where appropriate while still being 
 able to grow the array, and it does so without the code having 
 to care about what kind of memory is currently backing the 
 dynamic array.
But chances are that if you grow a malloced slice bu GC, you cause a memory leak, and that's what Walter is trying to solve. On the other hand, there should be a way for the caller to explicitly allow a function to grow a GC-allocated slice, without touching the function, and my idea does not solve that. I dunno...
Nov 11 2019
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 5:15:23 AM MST Dukc via Digitalmars-d wrote:
 On Monday, 11 November 2019 at 11:51:09 UTC, Jonathan M Davis

 wrote:
 IMHO, the ability to append to a dynamic array without caring
 whether it's backed by GC-allocated memory, malloc-ed memory,
 stack allocated memory, or whatever is a huge benefit.
Yeah... And I definitely think all of these should continue to exist with the possible exception of malloced memory.
 Right now, you can pass a dynamic array that isn't backed by
 the GC to a function, and it all just works even if the
 function needs to do a reallocation to make the array large
 enough. Doing so is not necessarily a bug and allows you to use
 non-GC allocated memory where appropriate while still being
 able to grow the array, and it does so without the code having
 to care about what kind of memory is currently backing the
 dynamic array.
But chances are that if you grow a malloced slice bu GC, you cause a memory leak, and that's what Walter is trying to solve. On the other hand, there should be a way for the caller to explicitly allow a function to grow a GC-allocated slice, without touching the function, and my idea does not solve that. I dunno...
Whether you risk causing a memory leak or not doesn't really have anything to do with whether reallocations can occur. The code needs to be aware of when the malloc-ed memory is definitely no longer used and then manually free it. It's certainly true that the lack of reference counting makes it harder to write code that uses slices of malloc-ed memory, but having slices of malloc-ed memory poses exactly the same risks of keeping those slices around too long whether ~= works on them or not. You either write the code in a way that you know when the slices are no longer used so that you know when you can free the memory, or you don't slice non-GC-allocated memory. - Jonathan M Davis
Nov 11 2019
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
This will make for stronger typing and is therefore a-good-thing.
Nov 11 2019
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
«The author is unaware of any slices in other languages that don't also carry a .capacity property to keep track of how much the slice can grow before it must be reallocated.» C++20 span: https://en.cppreference.com/w/cpp/container/span Maybe you can achieve binary compatibility, somehow.
Nov 11 2019
prev sibling next sibling parent ixid <adamsibson gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
These changes are becoming too big for a production language. They appear to be driven by the reasonable desire to fully separate D from the GC and add a stronger model for memory safety but with this extent of change would be more appropriate for a test version of D to develop the whole rather than piecemeal and potentially release as a D3.0 if the changes become sufficiently major when taken together.
Nov 11 2019
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 [snip]
In that first example, the following int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10]; doesn't actually compile. You would compile it as auto slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10]; So actually auto slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10]; slice ~= 1; generates an error because you can't append an int to an int*.
Nov 11 2019
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 11 November 2019 at 13:25:35 UTC, jmh530 wrote:
 In that first example, the following
 int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
 doesn't actually compile.

 You would compile it as
 auto slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
It should probably just be int[] slice = (cast(int*)malloc(10 * int.sizeof))[0 .. 10]; Since the type certainly should be int[], and it will compile.
Nov 11 2019
parent jmh530 <john.michael.hall gmail.com> writes:
On Monday, 11 November 2019 at 13:26:56 UTC, Adam D. Ruppe wrote:
 [snip]

 It should probably just be

 int[] slice = (cast(int*)malloc(10 * int.sizeof))[0 .. 10];


 Since the type certainly should be int[], and it will compile.
Ah, thanks. So I could probably re-write this as ``` int* data = cast(int*)malloc(10 * int.sizeof); int[] slice = data[0 .. 10]; ``` So maybe the issue is with being able to slice arbitrary pointers? I don't know... It all seems tied in to the ability to conflate a malloc'ed slice with a GC'ed slice. If anything, I don't think it should be easy to do this.
Nov 11 2019
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Prior art fails to mention  nogc, which already prohibits these 
operations in D today. So does -betterC. These work without 
breaking nearly every D module I have ever seen for marginal 
benefit at best.
Nov 11 2019
next sibling parent reply bachmeier <no spam.net> writes:
On Monday, 11 November 2019 at 13:33:35 UTC, Adam D. Ruppe wrote:
 Prior art fails to mention  nogc, which already prohibits these 
 operations in D today. So does -betterC. These work without 
 breaking nearly every D module I have ever seen for marginal 
 benefit at best.
The "marginal benefit at best" depends. If the goal is to move away from the GC and make using D without the GC the priority, as seems to be the case based on this and other recent work, the benefit is not marginal. Some communication about the overall goal here would be nice. Is there even a plan?
Nov 11 2019
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 11 November 2019 at 14:02:36 UTC, bachmeier wrote:
 If the goal is to move away from the GC and make using D 
 without the GC the priority,
This already works - and you can statically enforce it on a per-function level by nogc. Or on a per-type basis with a trivial wrapper struct (except null doesn't implicitly construct - we should fix THAT). Or on a per-project basis with -betterC. All this proposal does is break code that hasn't opted into those existing options.
 Some communication about the overall goal here would be nice. 
 Is there even a plan?
I agree with this regardless though.
Nov 11 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 6:22 AM, Adam D. Ruppe wrote:
 This already works - and you can statically enforce it on a per-function level 
 by  nogc. Or on a per-type basis with a trivial wrapper struct (except null 
 doesn't implicitly construct - we should fix THAT). Or on a per-project basis 
 with -betterC.
 
 All this proposal does is break code that hasn't opted into those existing
options.
Global solutions do exist, but they don't allow mixed mode code. The idea is to make code mechanically auditable. It isn't auditable if creating a wrapper is expected.
 Some communication about the overall goal here would be nice. Is there even a 
 plan?
I agree with this regardless though.
Be able to mechanically verify that code does not have memory safety bugs. We may not be able to get there 100%, but the closer the better.
Nov 11 2019
prev sibling parent =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:
On 2019-11-11 13:33:35 +0000, Adam D. Ruppe said:

 Prior art fails to mention  nogc, which already prohibits these 
 operations in D today. So does -betterC. These work without breaking 
 nearly every D module I have ever seen for marginal benefit at best.
Yes, that's a very good point and gives one everything to manually manage memory and be sure the GC doesn't jump in. If a mixed memory management model is used, the programmer needs to understand the side-effects and what happens if the GC jumps in and take full responsibility. So, I don't like this change at all. KISS. -- Robert M. Münch http://www.saphirion.com smarter | better | faster
Nov 11 2019
prev sibling next sibling parent reply IGotD- <nise nise.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.
"Enlarging a slice, using the append operator or by setting the .length property, makes use of the garbage collector. This fosters the incorrect notion that D slices require the GC." The title mentions arrays but the rest of the text is about slices. I need one clarification, this DIP regards slices not arrays? So an array is being able to be resized just as before. It's just that this will no longer apply for slices? If this only regards slices, then I think this is a sound change, even if it is breaking. A slice should not really know the underlying algorithm how the data is stored. It could be an array, it could be part of a buffer or a view into any other structure. Limiting the slice so that it cannot grow beyond its original boundaries makes sense. If this include arrays, that they aren't able to grow then I'm against this change but again it doesn't make any sense. Then it is not a dynamic array.
Nov 11 2019
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Monday, 11 November 2019 at 14:48:54 UTC, IGotD- wrote:
 I need one clarification, this DIP regards slices not arrays? 
 So an array is being able to be resized just as before. It's 
 just that this will no longer apply for slices?
Maybe this article will help: https://dlang.org/articles/d-array-article.html#responsibility
 So where is the true dynamic array type in D? It's hidden by 
 the runtime, and in fact, has no formal type. Slices are good 
 enough, and as it turns out, the runtime is smart enough about 
 what you want to do with the data, that you almost never notice 
 dynamic arrays are missing as a full-fledged type. In fact, 
 most D coders consider the D slice to be the dynamic array type 
 -- it's even listed as a dynamic array type in the spec! The 
 lack of ownership is very subtle and easy to miss.
Nov 11 2019
parent reply IGotD- <nise nise.com> writes:
On Monday, 11 November 2019 at 14:56:22 UTC, Dennis wrote:
 On Monday, 11 November 2019 at 14:48:54 UTC, IGotD- wrote:
 I need one clarification, this DIP regards slices not arrays? 
 So an array is being able to be resized just as before. It's 
 just that this will no longer apply for slices?
Maybe this article will help: https://dlang.org/articles/d-array-article.html#responsibility
 So where is the true dynamic array type in D? It's hidden by 
 the runtime, and in fact, has no formal type. Slices are good 
 enough, and as it turns out, the runtime is smart enough about 
 what you want to do with the data, that you almost never 
 notice dynamic arrays are missing as a full-fledged type. In 
 fact, most D coders consider the D slice to be the dynamic 
 array type -- it's even listed as a dynamic array type in the 
 spec! The lack of ownership is very subtle and easy to miss.
Does this mean if this DIP goes though, a "real" distinctive dynamic array implementation must be done in order facilitate the functionality that has previously been done by D-slices?
Nov 11 2019
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 11 November 2019 at 15:00:50 UTC, IGotD- wrote:
 Does this mean if this DIP goes though, a "real" distinctive 
 dynamic array implementation must be done in order facilitate 
 the functionality that has previously been done by D-slices?
Reminds me of the old T[new] thing from like.... dconf 2007 or something. What's old is new again.
Nov 11 2019
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 7:48:54 AM MST IGotD- via Digitalmars-d wrote:
 On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee5
 6e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should
 occur in this thread. The review period will end at 11:59 PM ET
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary,
 the DIP will be scheduled for another round of Community
 Review. Otherwise, it will be queued for the Final Review and
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.
"Enlarging a slice, using the append operator or by setting the .length property, makes use of the garbage collector. This fosters the incorrect notion that D slices require the GC." The title mentions arrays but the rest of the text is about slices. I need one clarification, this DIP regards slices not arrays? So an array is being able to be resized just as before. It's just that this will no longer apply for slices? If this only regards slices, then I think this is a sound change, even if it is breaking. A slice should not really know the underlying algorithm how the data is stored. It could be an array, it could be part of a buffer or a view into any other structure. Limiting the slice so that it cannot grow beyond its original boundaries makes sense. If this include arrays, that they aren't able to grow then I'm against this change but again it doesn't make any sense. Then it is not a dynamic array.
Dynamic arrays _are_ slices. T[] is a dynamic array which is a slice of a block of memory which may or may not be GC-allocated. Some folks do try to talk like dynamic arrays are the GC-allocated memory block that a slice refers to, but that's not the definition used by the language. As far as the language is concerned, T[] is a dynamic array no matter what memory it refers to, and the language has no specific term for a memory block allocated by the GC which is sliced by a dynamic array. This DIP makes it so that it would no longer be possible to use any operations on a dynamic array which would increase its size, whereas right now those operations always work in code that isn't nogc no matter what memory the dynamic array is a slice of. It's just that in the case where the dynamic array has no capacity to grow, you always get a reallocation. Whether the dynamic array referred to GC-allocated memory or not is irrelevant except that if the memory is not GC-allocated, there definitely isn't room to grow the dynamic array in-place, whereas if it was GC-allocated then there might be room to grow in-place. In either case, if there isn't room to grow in-place, then the GC allocates a new memory block, copies the elements into the new memory block, and alters the dynamic array to point to the new memory block. - Jonathan M Davis
Nov 11 2019
prev sibling next sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
The Rationale of this DIP is lacking IMO.
It highlights a problem, that the DIP will fix, but doesn't 
explain the greater
context of the problem.
The last few DIPs started from a fairly low level IMO.
To me it is not clear where this journey should go.
What is the long term payoff for this pain?
Are we aiming for rust like memory management? C++ like? Python 
like?
Something in between?

I think this needs to addressed before this DIP can be discussed.

I know this pushes the bounds of the review guidelines, but I 
think this
DIP starts a journey in the wrong direction.
IMO D's problem is not the GC nor ~=.
I think it is the "wrong to assume" that manual memory management 
(mmm) should
play nice with the GC.
The GC should be the  safe default.
If you use mmm you made your bed, don't force me to reshuffle my 
GC bed.
I think mmm and safety are very hard to consolidate.
We should spend our resources on making range based, functional, 
programming
in combination with the GC  safe and easy.

Another valid way to solve the problem given in the rationale of 
the DIP would
be to disallow the use of malloc or the cast.
Nov 11 2019
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Monday, 11 November 2019 at 15:11:32 UTC, Robert Schadek wrote:
 

 Another valid way to solve the problem given in the rationale 
 of the DIP would
 be to disallow the use of malloc or the cast.
malloc is also only allowed in system code.
Nov 11 2019
next sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Monday, 11 November 2019 at 15:24:48 UTC, jmh530 wrote:
 malloc is also only allowed in  system code.
Now I'm even less sure what this DIP tries to achieve in the long run.
Nov 11 2019
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 11 November 2019 at 15:29:21 UTC, Robert Schadek wrote:
 Now I'm even less sure what this DIP tries to achieve in the 
 long run.
The question isn't the allocation itself, but rather the slice. void main() { int[5] buffer; // or malloc or whatever foo(buffer[]); // this is OK } safe void foo(scope int[] a) { a ~= 0; // and this now reallocates on the GC } but i'd argue this is the *right* thing to do. and if you don't want the GC function, do nogc on it, now it is a compile error. If we were saying that T[] shouldn't have a ~= operator at all... well i can kinda see the point, you could use a container object instead. But a lot of times the current behavior is perfectly sane. Like this example has no memory leak, no out of bounds access, and is convenient to use. If anything maybe we should make nogc more granular so like have independent nosliceappend or whatever. But like meh.
Nov 11 2019
parent Robert Schadek <rschadek symmetryinvestments.com> writes:
On Monday, 11 November 2019 at 15:56:02 UTC, Adam D. Ruppe wrote:
I was not necessarily trying to make that point.

I was merely trying to say, that IMO, mmm should not be considered
a necessarily desirable concept, especially when  safety is the 
goal.

In other words, approxEqual(!mmm, "Less problems making D  safe")
Nov 11 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 7:24 AM, jmh530 wrote:
 malloc is also only allowed in  system code.
Although malloc() is marked as system, it actually is safe. It's free() that needs to be system.
Nov 11 2019
next sibling parent reply Jab <jab_293 gmall.com> writes:
On Tuesday, 12 November 2019 at 03:08:28 UTC, Walter Bright wrote:
 On 11/11/2019 7:24 AM, jmh530 wrote:
 malloc is also only allowed in  system code.
Although malloc() is marked as system, it actually is safe. It's free() that needs to be system.
You can only really consider malloc safe cause "void*" is completely useless in safe, and casting it to a different type is what is unsafe. By those constraints free() could also be considered "safe". If you do cast the pointer to another type, then both are not safe.
Nov 11 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 7:35 PM, Jab wrote:
 On Tuesday, 12 November 2019 at 03:08:28 UTC, Walter Bright wrote:
 On 11/11/2019 7:24 AM, jmh530 wrote:
 malloc is also only allowed in  system code.
Although malloc() is marked as system, it actually is safe. It's free() that needs to be system.
You can only really consider malloc safe cause "void*" is completely useless in safe, and casting it to a different type is what is unsafe. By those constraints free() could also be considered "safe". If you do cast the pointer to another type, then both are not safe.
The cast of the pointer returned from malloc() is indeed unsafe, but the malloc() call is safe.
Nov 11 2019
prev sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Tuesday, 12 November 2019 at 03:08:28 UTC, Walter Bright wrote:
 Although malloc() is marked as  system, it actually is  safe. 
 It's free() that needs to be  system.
I think that is a mute point. malloc gives you a void*, you can't really do anything with that void* in safe code. You can't even cast it to int*, and it shouldn't. I know I sound like a broken record, but IMO mmm and safe is the wrong direction.
Nov 12 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, November 12, 2019 1:43:27 AM MST Robert Schadek via Digitalmars-
d wrote:
 On Tuesday, 12 November 2019 at 03:08:28 UTC, Walter Bright wrote:
 Although malloc() is marked as  system, it actually is  safe.
 It's free() that needs to be  system.
I think that is a mute point. malloc gives you a void*, you can't really do anything with that void* in safe code. You can't even cast it to int*, and it shouldn't. I know I sound like a broken record, but IMO mmm and safe is the wrong direction.
Honestly, I don't think that it can really be done without seriously hamstringing ourselves. Dynamic arrays and manually managed memory are a _terrible_ combination, because dynamic arrays do nothing to manage lifetimes. They're really not any different from passing naked pointers around as far as lifetime tracking goes. This means that the only way that you can automatically manage that is to restrict where they can be stored. scope does that, but the result is that you can pretty much just pass the data into a function that operates on it and doesn't store it. It doesn't even allow you to do something as simple as have the function return a range wrapper like a lot of Phobos does. And real world code typically needs to store stuff, not just pass it around. Maybe what Walter is looking to do with live will let it go a bit further, but I don't see how it could possibly allow anywhere near the freedom you get when passing around dynamic arrays that are slices of GC-allocated memory. You end up throwing all kinds of complications into the mix just to try to get a bit more safety for non-GC-allocated memory. Maybe some of those complications are worth it, but at some point, it's just too much extra machinery for too little benefit, and I'd argue that if it harms common use patterns of GC-allocated objects (which this DIP definitely does), then it's going to far. Personally, I don't think that it makes much sense to try to improve passing around naked slices of malloc-ed memory. The lack of lifetime management makes it so that you can't possibly guarantee that it's safe to free the memory without severely restricting what's allowed, and in general, I'd strongly argue that in most cases, it's bad practice to pass around naked slices of malloc-ed memory unless what the code is deing is restricted enough that the programmer knows what's being done with that memory so that they can know when it's safe to free it. Some use of features like scope may help make it more manageable, but I think that it's folly to expect code to do something like auto arr = (cast(int*)malloc(length * int.sizeof))[0 .. length]; pass that memory around, and then later do free(arr.ptr); and have that be safe. Outside of simple examples, to pass around malloc-ed memory safely, it needs to be wrapped in a type that manages its lifetime (e.g. a smart pointer). Language improvements (like DIP 1000) may make it easier to manage when pointers or references to that memory temporarily escape the smart pointer in order to operate on it, but to pass around and store that memory like you'd frequently do with GC-allocated memory, it needs to be wrapped in something that manages its lifetime, just like you'd do in languages like C++. Regardless, even if all agreed that we wanted to improve the safety of malloc-ing a naked dynamic array, passing it around, and then freeing it, I don't see how this DIP really helps with that. As has been pointed out already in this thread, simply mutating the dynamic array itself screws with your ability to get the pointer from it to free (showing that you really should be storing a pointer for that separately regardless of what's going on with appending), and appending doesn't cause any safety problems. It just makes it harder to track whether the dynamic array still refers to any portion of that malloc-ed memory, but even then, it doesn't make it impossible to do so, and if you want to forbid appending, you can simply require that the code be nogc. So, the DIP fails to even do what it's trying to do, and in the process, it hurts very common use cases. - Jonathan M Davis
Nov 12 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 12 November 2019 at 11:12:30 UTC, Jonathan M Davis 
wrote:
 Dynamic arrays and manually managed memory are a _terrible_ 
 combination, because dynamic arrays do nothing to manage 
 lifetimes. They're really not any different from passing naked 
 pointers around as far as lifetime tracking goes. This means 
 that the only way that you can automatically manage that is to 
 restrict where they can be stored. scope does that, but the 
 result is that you can pretty much just pass the data into a 
 function that operates on it and doesn't store it.
One option, albeit unorthodox, is to require heap-stored slices to be "concrete" and therefore "encode" the properties of what it is referencing. Whereas you could allow slices used in the call-graph to be "abstract" (templated, but without the syntax) and then infer their concrete properties using static analysis. So, the counter-argument would be "combinatorial explosion", but if the compiler deduce that the actual function that is being compiled generate the same code for a set of concrete slice types then you can emit a single copy of the compiled code. Anyway, that ought to be sufficient to distinguish GC, reference counted and RAII objects. And it would also allow D slices to differ from C++ slices, but also allow you to write one function to cover both scenarios.
Nov 12 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
The idea is to be able to guarantee memory safety. I've spoken about this in my 
presentations and DConf, and that's clearly what DIP25 and DIP1000 were
targeted 
at, and my proposals for an Ownership/Borrowing system for D.

This is where the industry is moving. Even C++ is heading that direction:

   http://open-std.org/JTC1/SC22/WG21/docs/papers/2018/p1179r0.pdf

This proposal is written by Herb Sutter, and is no joke.

We either get on the bus or get run over by the bus.
Nov 11 2019
next sibling parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright wrote:

 We either get on the bus or get run over by the bus.
Frankly speaking, I've read such argumentations a lot of times along many years in the D history: the concurrency chapter of the D Programming Language is here to testify that this type of argumentation should be avoided, in a technical discussion. Let's move forward from that, and let's try to find a sound solution to the memory safety problem, but without killing the language in the attempt. There's no hurry: D is _already_ a safe language, thanks to your intuition of 20 years ago to bake the GC.
Nov 11 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 2:33 PM, Paolo Invernizzi wrote:
 On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright wrote:
 
 We either get on the bus or get run over by the bus.
Frankly speaking, I've read such argumentations a lot of times along many years in the D history: the concurrency chapter of the D Programming Language is here to testify that this type of argumentation should be avoided, in a technical discussion. Let's move forward from that, and let's try to find a sound solution to the memory safety problem, but without killing the language in the attempt. There's no hurry: D is _already_ a safe language, thanks to your intuition of 20 years ago to bake the GC.
D is quite a bit safer currently than C or C++, but we've been a bit complacent about that advantage and are at risk of falling behind. I am replying to the question about what is the direction D is going. This is the direction and the motivation for it. Whether DIP 1025 is the correct move in that direction or not is certainly worthy of technical discussion. BTW, after not having memory corruption bugs for years, I've lost days of work in the last couple months looking for the cause of two memory corruption problems in DMD. I had grown complacent.
Nov 11 2019
parent reply Jab <jab_293 gmall.com> writes:
On Tuesday, 12 November 2019 at 00:51:28 UTC, Walter Bright wrote:
 On 11/11/2019 2:33 PM, Paolo Invernizzi wrote:
 On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright 
 wrote:
 
 We either get on the bus or get run over by the bus.
Frankly speaking, I've read such argumentations a lot of times along many years in the D history: the concurrency chapter of the D Programming Language is here to testify that this type of argumentation should be avoided, in a technical discussion. Let's move forward from that, and let's try to find a sound solution to the memory safety problem, but without killing the language in the attempt. There's no hurry: D is _already_ a safe language, thanks to your intuition of 20 years ago to bake the GC.
D is quite a bit safer currently than C or C++, but we've been a bit complacent about that advantage and are at risk of falling behind. I am replying to the question about what is the direction D is going. This is the direction and the motivation for it. Whether DIP 1025 is the correct move in that direction or not is certainly worthy of technical discussion. BTW, after not having memory corruption bugs for years, I've lost days of work in the last couple months looking for the cause of two memory corruption problems in DMD.
How have you not had this problem before? DMD's code base is kind of horrible. Pointers to arrays that end up having to be accessed by (*p)[x]. A simple wrapper could easily have alleviated that entirely. It's kind of ironic that the D reference compiler showcases exactly how you shouldn't write D code.
 I had grown complacent.
And now you are over eager to make changes to change that complacency. These DIPs are being ejected out one after another. It seems like everything else is being thrown under the bus on this hellbent pursuit of safety.
 ... the former at compile time and the latter at runtime (as a 
 dynamic check is necessary). Hence, a long deprecation period 
 will be necessary.
Of all the things people would be willing to have such a big breaking changes for. Auto-decoding to name one. This is now where the line is drawn? Some of these decisions are being made too hastily IMO.
Nov 11 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 7:03 PM, Jab wrote:
 How have you not had this problem before?
Because I was careful.
 DMD's code base is kind of horrible. 
 Pointers to arrays that end up having to be accessed by (*p)[x]. A simple 
 wrapper could easily have alleviated that entirely. It's kind of ironic that
the 
 D reference compiler showcases exactly how you shouldn't write D code.
DMD was rather recently converted to D from a "C with Classes" codebase. Its structure very much reflects that heritage.
 Of all the things people would be willing to have such a big breaking changes 
 for. Auto-decoding to name one. This is now where the line is drawn? Some of 
 these decisions are being made too hastily IMO.
No decision has been made about DIP 1025. That's what this process is for. Everyone gets their say.
Nov 11 2019
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 11.11.19 23:28, Walter Bright wrote:
 
 We either get on the bus or get run over by the bus.
That's a false dichotomy. E.g., you could also eat the engine of the bus for dinner and then get stranded in the desert with heavy metal poisoning.
Nov 11 2019
prev sibling next sibling parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright wrote:
 The idea is to be able to guarantee memory safety. I've spoken 
 about this in my presentations and DConf, and that's clearly 
 what DIP25 and DIP1000 were targeted at, and my proposals for 
 an Ownership/Borrowing system for D.

 This is where the industry is moving. Even C++ is heading that 
 direction:

   
 http://open-std.org/JTC1/SC22/WG21/docs/papers/2018/p1179r0.pdf

 This proposal is written by Herb Sutter, and is no joke.

 We either get on the bus or get run over by the bus.
By the short bus. Sorry, couldn't resist.
Nov 12 2019
prev sibling next sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright wrote:
 The idea is to be able to guarantee memory safety. I've spoken 
 about this in my presentations and DConf, and that's clearly 
 what DIP25 and DIP1000 were targeted at, and my proposals for 
 an Ownership/Borrowing system for D.

 This is where the industry is moving. Even C++ is heading that 
 direction:

   
 http://open-std.org/JTC1/SC22/WG21/docs/papers/2018/p1179r0.pdf

 This proposal is written by Herb Sutter, and is no joke.

 We either get on the bus or get run over by the bus.
golang is not going in that direction. They are not safe but mostly because they have chosen performance over safety. To me D is great because it is not doing what everybody else is doing. If D was only doing what c++ or rust are doing, D can only become as good as those IMO. But D should be better. Generative programming in combination with functional programming, is awesome. Hardly anybody is doing that. I'm not saying we should steal other peoples good ideas. I'm trying to say we should steal interesting ideas/mistakes and improve on them. For instance taking the c++ template meta-programming mistake and making it a feature was a huge win. Additionally, I think there is an elephant in the room. Multi-threading. I just watched a talk on rust async/await. https://www.youtube.com/watch?v=lJ3NC-R3gSI It took them ~10 years to get the borrow-checker + multi threading thing done/right, and compared to us they have infinite resources. Is there a plan for D and the borrow-checker in place/planning?
Nov 12 2019
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 12 November 2019 at 09:12:51 UTC, Robert Schadek 
wrote:
 golang is not going in that direction. They are not  safe but 
 mostly because they have chosen performance over safety.
You have to make an effort to break Go. You can do that with most languages... Python. Like Java it is much more its own virtual environment (but unix influenced) than a system level programming language. Which makes sense for web-services.
Nov 12 2019
parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Tuesday, 12 November 2019 at 10:09:05 UTC, Ola Fosheim Grøstad 
wrote:

 Python. Like Java it is much more
 its own virtual environment (but unix influenced) than a system 
 level programming language. Which makes sense for web-services.
Hm, when I think of go I think of docker/kubernetes. For me that is systems programming. I see your point, but I think there are other ways to look at it. You can actually get Microcontrollers that run python not c. I think saying, and I'm not saying you say that, systems programming therefore mmm is at least a "Hasty Generalization".
Nov 12 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 12 November 2019 at 10:25:40 UTC, Robert Schadek 
wrote:
 Hm, when I think of go I think of docker/kubernetes.
 For me that is systems programming.
 I see your point, but I think there are other ways to look at 
 it.
 You can actually get Microcontrollers that run python not c.
Yep, that is true.
 I think saying, and I'm not saying you say that, systems 
 programming therefore
 mmm is at least a "Hasty Generalization".
Yes. I think the long term future even for embedded programming is high level languages that allows you to "bind" datastructures to low level representations where you need them (but otherwise leave it to the compiler). So yes, how to do systems programming will hopefully evolve over time. But for me there ought to be a separation between high level and low level constructs. In C++ all high level types are library only, that means the compiler can only do low level optimization. D has some room to innovate there, so I certainly am sympathetic to the idea that high level features should not be undermined. Maybe the question ought to be, can the high level features be strenghtened in a way that allows the compiler to be smarter and that allows the compiler to "understand" where low level assumptions are being made and where it has free hands to be smart. So rather than downgrading high level construct, upgrade them from runtime-constructs to compiler-constructs. It is quite intriguing though that something as simple as slices can be so complex to figure out. I've written some variations of slices for C++ before they added it, and there are so many trade offs... Maybe one has to classifiy all the different use scenarios for slice-like types and then figure out some solution that allows the compiler to be more clever than in other languages.
Nov 12 2019
prev sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 12 November 2019 at 09:12:51 UTC, Robert Schadek 
wrote:
 On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright 
 wrote:
 Although malloc() is marked as  system, it actually is  safe. 
 It's free() that needs to be  system.
I think that is a mute point.
(You mean moot point)
 malloc gives you a void*, you can't really do anything with 
 that void*
 in  safe code.
 You can't even cast it to int*, and it shouldn't.

 I know I sound like a broken record, but IMO mmm and  safe is 
 the wrong direction.
Thats what containers are for, to encapsulate the inherent unsafety and expose a safe interface.
 Generative programming in combination with functional 
 programming, is
 awesome. Hardly anybody is doing that.
 I'm not saying we should steal other peoples good ideas.
 I'm trying to say we should steal interesting ideas/mistakes 
 and improve on them.
We should definitely steal them, but we had better a) make sure they are good ideas, and b) understand what makes them good. Otherwise we risk doing what C++ did with static if/constexpr if, UFCS and many more.
 For instance taking the c++ template meta-programming mistake 
 and
 making it a feature was a huge win.

 Additionally, I think there is an elephant in the room. 
 Multi-threading.
 I just watched a talk on rust async/await.
 https://www.youtube.com/watch?v=lJ3NC-R3gSI
 It took them ~10 years to get the borrow-checker + multi 
 threading thing
 done/right, and compared to us they have infinite resources.
 Is there a plan for D and the borrow-checker in place/planning?
That would be DIP1024[1], which follows the same proof of safety by induction of trusted leaf functions for encapsulating safe access to members with locks/atomics/etc. [1]:https://github.com/dlang/DIPs/blob/master/DIPs/DIP1024.md
Nov 12 2019
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 12.11.19 11:37, Nicholas Wilson wrote:
 
 We should definitely steal them, but we had better a) make sure they are 
 good ideas, and b) understand what makes them good. Otherwise we risk 
 doing what C++ did with static if/constexpr if, UFCS and many more.
Actually it will be even worse because we don't follow C++'s backwards-compatibility mantra. To me it appears that Walter wants to dismantle the GC piece by piece.
Nov 12 2019
prev sibling parent reply Robert Schadek <rschadek symmetryinvestments.com> writes:
On Tuesday, 12 November 2019 at 10:37:44 UTC, Nicholas Wilson 
wrote:
 (You mean moot point)
yes
 Thats what containers are for, to encapsulate the inherent 
 unsafety and expose a safe interface.
Okay, from where I'm standing that sounds like the problem is with malloc/slicing malloced memory, and not so much the GC. On the other hand, the more functional my D becomes, the less I feel the need for containers at all.
 encapsulating safe access to members with locks/atomics/etc.

 [1]:https://github.com/dlang/DIPs/blob/master/DIPs/DIP1024.md
Thank you, I'll have another read. Sounds like this whole safe-borrowing-rc needs a top down design DIP to understand the complete context.
Nov 12 2019
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 12 November 2019 at 11:46:30 UTC, Robert Schadek 
wrote:
 On Tuesday, 12 November 2019 at 10:37:44 UTC, Nicholas Wilson 
 wrote:
 Thats what containers are for, to encapsulate the inherent 
 unsafety and expose a safe interface.
Okay, from where I'm standing that sounds like the problem is with malloc/slicing malloced memory, and not so much the GC. On the other hand, the more functional my D becomes, the less I feel the need for containers at all.
Perhaps containers is the wrong word. I meant in in the sense data structures intersection wrapping, e.g. DList!int or Atomic!int. Both of those are inherently unsafe (blobs of pointers and shared memory respectively) but the container provides a safe way to interact with the data encapsulated by it by disallowing outside access in an unchecked manner.
Nov 12 2019
parent Robert Schadek <rschadek symmetryinvestments.com> writes:
On Tuesday, 12 November 2019 at 13:08:08 UTC, Nicholas Wilson 
wrote:
 Perhaps containers is the wrong word. I meant in in the sense 
 data structures intersection wrapping, e.g. DList!int or 
 Atomic!int. Both of those are inherently unsafe (blobs of 
 pointers and shared memory respectively) but the container 
 provides a safe way to interact with the data encapsulated by 
 it by disallowing outside access in an unchecked manner.
I see.
Nov 12 2019
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Monday, 11 November 2019 at 22:28:05 UTC, Walter Bright wrote:
 The idea is to be able to guarantee memory safety. I've spoken 
 about this in my presentations and DConf, and that's clearly 
 what DIP25 and DIP1000 were targeted at, and my proposals for 
 an Ownership/Borrowing system for D.

 This is where the industry is moving. Even C++ is heading that 
 direction:

   
 http://open-std.org/JTC1/SC22/WG21/docs/papers/2018/p1179r0.pdf

 This proposal is written by Herb Sutter, and is no joke.

 We either get on the bus or get run over by the bus.
I feel like if this proposal is part of the motivation for this DIP, then it should be mentioned in the DIP.
Nov 12 2019
prev sibling next sibling parent Uknown <sireeshkodali1 gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.
I get what the DIP is trying to do, but like Jonathon said, this solution is simply not worth the cost *at all*. Such a change would alienate the existing users, make a lot of good, working code break for no real reason, and cause a lot of other issues. The better solution IMO is to instead set an allocator, either per thread or global, like std.experimental.allocator does. Then, change the runtime to allocate and expand slices by using that allocator, instead of the GC. The allocator could be GC by default. Maybe once that is done, slicing arbitrary pointers could be depreciated. Then the ` safe` aspect would be solved, and there would be practically 0 code breakage. Something like this: ---a.d import core.allocator; //hypothetical void f(int[] x) safe { x ~= 0; } void main() system { int[] slice = (cast(int*)(malloc(10 * int.sizeof)))[0 .. 10]; //deprecated this_thread.allocator = Mallocator; int[] slice = new int[10]; f(slice); //now safe, since mallocator is used instead of GC }
Nov 11 2019
prev sibling next sibling parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.
"This change is a __necessary__ part of evolving D toward being memory safe without using a GC" This is a bold statement, that needs to be addressed with a little of explanation of why there are no alternatives. I think it's not acceptable to suffer such a huge pain, without having a clear understanding that there was a deep analysis on potential alternative solutions, and an explanation of why they are not sufficient for the scope. Just to be clear, why nogc is not enough? Thanks
Nov 11 2019
parent reply Uknown <sireeshkodali1 gmail.com> writes:
On Monday, 11 November 2019 at 16:36:57 UTC, Paolo Invernizzi 
wrote:
 
 "This change is a __necessary__ part of evolving D toward being 
 memory safe without using a GC"

 This is a bold statement, that needs to be addressed with a 
 little of explanation of why there are no alternatives.

 I think it's not acceptable to suffer such a huge pain, without 
 having a clear understanding that there was a deep analysis on 
 potential alternative solutions, and an explanation of why they 
 are not sufficient for the scope.

 Just to be clear, why  nogc is not enough?

 Thanks
A lot of people are bringing it up, so I'll bite. The problem with nogc is that it doesn't cover all cases. Imagine the code given in the DIP like this instead: ---lib.d void f(int x[]) safe pure nothrow { x ~= 0; } ---main.d void main() safe { import lib: f; import std.container : Array; Array!int x = [0, 1, 2, 3, 4]; f(slice); // x's dtor will try to free an invalid pointer } Clearly here main does something that seems safe on the surface. But in actuality it is clearly unsafe code. And its hard to verify, because main and the libraries used are written by completely different people.
Nov 11 2019
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/11/19 11:47 AM, Uknown wrote:
 A lot of people are bringing it up, so I'll bite. The problem with  nogc 
 is that it doesn't cover all cases. Imagine the code given in the DIP 
 like this instead:
 
 ---lib.d
 
 void f(int x[])  safe pure nothrow
 {
      x ~= 0;
 }
 
 ---main.d
 
 void main()  safe
 {
      import lib: f;
      import std.container : Array;
 
      Array!int x = [0, 1, 2, 3, 4];
 
      f(slice);
      // x's dtor will try to free an invalid pointer
 }
 
 Clearly here main does something that seems safe on the surface. But in 
 actuality it is clearly unsafe code. And its hard to verify, because 
 main and the libraries used are written by completely different people.
No, you are misunderstanding a lot here. 1. f(slice), there is no symbol slice, I think maybe you mean x[]? 2. f's x is a *copy*, which means that appending to x here DOES NOT AFFECT main's x at all. Main's x will destroy perfectly fine, and all is well. 3. If nogc is added to main, then it won't compile, because f cannot be nogc. Which is quite the point people are making. -Steve
Nov 11 2019
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 17:00:07 UTC, Steven Schveighoffer 
wrote:
 No, you are misunderstanding a lot here.
This is nitpicking. If you are going to have a good story with interfacing with C++ libraries and frameworks... you don't want this to fail accidentally: ownerpointer = normalize(initialize(allocate())) Where normalize is a D function that calls other GC library code that is well-behaving, except the one that accidentally grows the slice because it works in the library-author's pure GC code. You have to be able to call GC code even if you want to modify non-GC memory. Anyway, it also isn't true that Dynamic Arrays don't have a type. It isn't untyped. But the typesystem it uses is completely different from the rest of the language. It appears to be quasi-dynamic effects based, whereas the rest of the language is mostly static strongly typed. Meaning, the type changes as you use it. Does any other object in D do that?
Nov 11 2019
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 10:12:13 AM MST Ola Fosheim Grøstad via 
Digitalmars-d wrote:
 On Monday, 11 November 2019 at 17:00:07 UTC, Steven Schveighoffer

 wrote:
 No, you are misunderstanding a lot here.
This is nitpicking. If you are going to have a good story with interfacing with C++ libraries and frameworks... you don't want this to fail accidentally: ownerpointer = normalize(initialize(allocate())) Where normalize is a D function that calls other GC library code that is well-behaving, except the one that accidentally grows the slice because it works in the library-author's pure GC code. You have to be able to call GC code even if you want to modify non-GC memory. Anyway, it also isn't true that Dynamic Arrays don't have a type. It isn't untyped. But the typesystem it uses is completely different from the rest of the language. It appears to be quasi-dynamic effects based, whereas the rest of the language is mostly static strongly typed. Meaning, the type changes as you use it. Does any other object in D do that?
How on earth do D dynamic arrays change their type? int[] is always int[]. I believe that the bit that Steven is talking about with regards to them being untyped is that inside druntime, it uses void*, because the code isn't templated - which arguably is something that should be fixed. Outside of druntime's innards though, a dynamic array always has the same type. All it really is is a struct containing a pointer and a size_t for length. - Jonathan M Davis
Nov 11 2019
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/11/19 12:28 PM, Jonathan M Davis wrote:
 On Monday, November 11, 2019 10:12:13 AM MST Ola Fosheim Grøstad via
 Digitalmars-d wrote:
 Anyway, it also isn't true that Dynamic Arrays don't have a type.
 It isn't untyped. But the typesystem it uses is completely
 different from the rest of the language.

 It appears to be quasi-dynamic effects based, whereas the rest of
 the language is mostly static strongly typed. Meaning, the type
 changes as you use it. Does any other object in D do that?
How on earth do D dynamic arrays change their type? int[] is always int[]. I believe that the bit that Steven is talking about with regards to them being untyped is that inside druntime, it uses void*, because the code isn't templated - which arguably is something that should be fixed. Outside of druntime's innards though, a dynamic array always has the same type. All it really is is a struct containing a pointer and a size_t for length.
auto arr = "hello".dup; ubyte[] arr2 = cast(ubyte[]) arr; arr2 ~= 1; // fine appends in-place (possibly) arr ~= 'a'; // fine, makes a copy The type doesn't matter to druntime, only the length of data. That is the point. Of course, this all breaks down if the type has a postblit/destructor, which IS stored by druntime. But I'm not sure if there are any runtime mechanisms to prevent it (I haven't touched that code much since the posblit/dtor code was added). -Steve
Nov 11 2019
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 17:28:09 UTC, Jonathan M Davis 
wrote:
 How on earth do D dynamic arrays change their type? int[] is 
 always int[].
Well, what we mean by "type" can of course be debated, but anyway: Case 1: char[] a = new char[2]; writeln(a.ptr); if (false) a.length = 1; a.length = 2; writeln(a.ptr); // same pointer Case 2: char[] a = new char[2]; writeln(a.ptr); if (true) a.length = 1; a.length = 2; writeln(a.ptr); // different pointer So, whether it extends inline depends what you did with the object dynamically. It isn't determined statically. Even though in this case that would be possible. Kinda like the difference between an open and closed file, whether it is open or not depends on some dynamic external cause (if it exists on the files system).
Nov 11 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 11:54:04 AM MST Ola Fosheim Grøstad via 
Digitalmars-d wrote:
 On Monday, 11 November 2019 at 17:28:09 UTC, Jonathan M Davis

 wrote:
 How on earth do D dynamic arrays change their type? int[] is
 always int[].
Well, what we mean by "type" can of course be debated, but anyway: Case 1: char[] a = new char[2]; writeln(a.ptr); if (false) a.length = 1; a.length = 2; writeln(a.ptr); // same pointer Case 2: char[] a = new char[2]; writeln(a.ptr); if (true) a.length = 1; a.length = 2; writeln(a.ptr); // different pointer So, whether it extends inline depends what you did with the object dynamically. It isn't determined statically. Even though in this case that would be possible. Kinda like the difference between an open and closed file, whether it is open or not depends on some dynamic external cause (if it exists on the files system).
So what if the pointer's value changed? Objects have their members mutated all the time. That doesn't make them different types. If you want to know whether appending to a dynamic array will result in a new buffer being allocated (with the elements being copied over, and the ptr member of the dynamic array being mutated), then check whether the capacity is large enough to contain the elements being appended. If it is, then the GC can expand the dynamic array in-place. If it isn't, then the GC has to allocate a new buffer, copy the elements over, and mutate the dynamic array's ptr and length members to point to the new buffer. And that's the same regardless of what kind of memory backs the dynamic array (it's just that there's never enough capacity to grow in-place if the underlying buffer wasn't allocated by the GC for dynamic arrays). I don't understand why you'd think that a dynamic array being in a different state based on what you did with it would in any way make it a different type. It's perfectly normal for the behavior of an object to change based on its state. - Jonathan M Davis
Nov 11 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 19:18:53 UTC, Jonathan M Davis 
wrote:
 I don't understand why you'd think that a dynamic array being 
 in a different state based on what you did with it would in any 
 way make it a different type. It's perfectly normal for the 
 behavior of an object to change based on its state.
Well, the conceptual state was exactly the same when accessed. So it is unusual for a core language type. Anyway, there is no point in arguing this, as we first have to agree on what we mean by the word "type" at runtime vs "type" at compile time. Which would be very off topic... (You could create a compile time ubertype that includes all other possible types, so that nothing can be said about any object at compile time, but you would still think that those objects had various types at runtime.)
Nov 11 2019
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/11/19 12:12 PM, Ola Fosheim Grøstad wrote:
 On Monday, 11 November 2019 at 17:00:07 UTC, Steven Schveighoffer wrote:
 No, you are misunderstanding a lot here.
This is nitpicking.
Sorry, I'm not sure why it's nitpicking. The example shown does not show what the author wanted it to show. When you have a function: void f(int x[]) safe pure nothrow { x ~= 0; } There is no way to call that function in a way that will break safety, or alter any external data. It's pure, only the shallowly passed in slice is affected, nothing outside. This is truly a misunderstanding by the author (Uknown) of what a slice actually is. The original example in the DIP shows more of a problem, but nogc is certainly a solution there, which is why people are bringing it up.
 If you are going to have a good story with interfacing with C++ 
 libraries and frameworks... you don't want this to fail accidentally:
 
 ownerpointer = normalize(initialize(allocate()))
 
 Where normalize is a D function that calls other GC library code that is 
 well-behaving, except the one that accidentally grows the slice because 
 it works in the library-author's pure GC code.
True, GC-using code could be valid to call if it doesn't affect your type. But nogc IS a valid way to use slices and ensure that they never use the GC to grow. But in fact, a better solution is to make sure the TYPE prevents it, and we absolutely have the mechanisms today to do this. An opt-in type (ironically, such as Array), would be sufficient to make sure you can still call gc-using code, without mucking with your type's ability to free the original pointer. If the DIP said it wanted to introduce a type that enforces it will always point to the resource that is managed, that would be fine. It just can't overtake T[] to mean that. It's way way too disruptive. I'd be fine with new syntax as well (like T[new] or whatnot). -Steve
Nov 11 2019
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 17:30:25 UTC, Steven Schveighoffer 
wrote:
 Sorry, I'm not sure why it's nitpicking. The example shown does 
 not show what the author wanted it to show.
Alright, alright, I thought it was obvious what he meant, but I guess that is because I already agreed with his core viewpoint.
 True, GC-using code could be valid to call if it doesn't affect 
 your type. But  nogc IS a valid way to use slices and ensure 
 that they never use the GC to grow.
Yes, but you need to be careful so you don't split the ecosystem in two. So people should be able to call the best libraries they can find (GC or not). I understand that this is difficult, but I think the intent of this DIP is pointing in the right direction, even if it isn't bulletproof or ready. So I hope it isn't dismissed outright.
 But in fact, a better solution is to make sure the TYPE 
 prevents it
Yes, a type that protects the underlying buffer somehow makes a lot of sense. But then you need to encourage templated slicing somehow, so that you can write generic code that takes all kinds of slicing-types (and use introspection perhaps to choose how they are used). Not so easy to get right, I guess... Hm.
 If the DIP said it wanted to introduce a type that enforces it 
 will always point to the resource that is managed, that would 
 be fine. It just can't overtake T[] to mean that. It's way way 
 too disruptive. I'd be fine with new syntax as well (like 
 T[new] or whatnot).
Ok, I understand. Maybe a type that will share the binary interface of C++ span and string_view. They are supposed to be the ones to use in function C++ function signatures to increase interoperability between libraries etc. Right now it is a bit difficult to say how span<> will be adopted in C++ libraries, but I would imagine that it might end up being used for matrices as well "span<span<T>>". So binary compatibility could make sense.
Nov 11 2019
parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 11 November 2019 at 18:13:45 UTC, Ola Fosheim Grøstad 
wrote:
 Maybe a type that will share the binary interface of C++ span 
 and string_view. They are supposed to be the ones to use in 
 function C++ function signatures to increase interoperability 
 between libraries etc.

 Right now it is a bit difficult to say how span<> will be 
 adopted in C++ libraries, but I would imagine that it might end 
 up being used for matrices as well "span<span<T>>". So binary 
 compatibility could make sense.
(A bit OT) We tried to convince Walter to make extern(C++) void f(int[]) Just Workâ„¢, to no avail, because he though that it would b "unintuitive" for people to have include a header to have to use it. Go figure. I doubt that span<span<T>> will be widely used, because jagged matrices are a bit of an odd choice of structure, span<string_view> might get a bit of use though.
Nov 11 2019
prev sibling parent reply Uknown <sireeshkodali1 gmail.com> writes:
On Monday, 11 November 2019 at 17:00:07 UTC, Steven Schveighoffer 
wrote:
 On 11/11/19 11:47 AM, Uknown wrote:
 A lot of people are bringing it up, so I'll bite. The problem 
 with  nogc is that it doesn't cover all cases. Imagine the 
 code given in the DIP like this instead:
 
 ---lib.d
 
 void f(int x[])  safe pure nothrow
 {
      x ~= 0;
 }
 
 ---main.d
 
 void main()  safe
 {
      import lib: f;
      import std.container : Array;
 
      Array!int x = [0, 1, 2, 3, 4];
 
      f(slice);
      // x's dtor will try to free an invalid pointer
 }
 
 Clearly here main does something that seems safe on the 
 surface. But in actuality it is clearly unsafe code. And its 
 hard to verify, because main and the libraries used are 
 written by completely different people.
No, you are misunderstanding a lot here. 1. f(slice), there is no symbol slice, I think maybe you mean x[]?
yes, my mistake
 2. f's x is a *copy*, which means that appending to x here DOES 
 NOT AFFECT main's x at all. Main's x will destroy perfectly 
 fine, and all is well.
You're right, I shouldn't have used Array. I was just trying to point out how libraries could obscure such bugs. A different container with different semantics which exposes slices could cause issues.
 3. If  nogc is added to main, then it won't compile, because f 
 cannot be  nogc. Which is quite the point people are making.

 -Steve
The problem is that once the slice is created, there's no way to know how the underlying memory was allocated. And if you want to append to the slice, then that's hard. Because there's no way to append generically in the language without the GC. Don't get me wrong, I think this DIP is the wrong solution. But there is an issue to be addressed. We need a way to be able to append without invoking the GC. I personally think std.experimental.allocators + some other associated language modifications are the way forward.
Nov 11 2019
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, November 11, 2019 11:03:35 AM MST Uknown via Digitalmars-d wrote:
 The problem is that once the slice is created, there's no way to
 know how the underlying memory was allocated. And if you want to
 append to the slice, then that's hard. Because there's no way to
 append generically in the language without the GC. Don't get me
 wrong, I think this DIP is the wrong solution. But there is an
 issue to be addressed. We need a way to be able to append without
 invoking the GC. I personally think std.experimental.allocators +
 some other associated language modifications are the way forward.
If you want to append to a dynamic array without using the GC, then just allocate a new block of memory with malloc or whatever, copy the elements from the dynamic array into it along with the elements being appended, slice that block of memory, and assign that dynamic array to the first one. Of course, you then have to code worrying about when it's safe to free that memory, but if your code needs something like reference counting to manage the memory, then it probably shouldn't be passing around dynamic arrays anyway. Dynamic arrays which are slices of malloc-ed memory or static arrays or whatnot work just fine so long as the code that does the slicing can know when the dynamic arrays which are slices of that memory go out of scope so that it can know when it's safe to free that memory, but if the dynamic arrays being passed around in a way that the code doing the allocating can't know when that memory isn't referenced anymore, then dynamic arrays are a bad fit - and that's the case whether this DIP is accepted or not. If the dynamic arrays are being passed around in a way that you don't know what their lifetimes are, then they need to be GC allocated. - Jonathan M Davis
Nov 11 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 11 November 2019 at 18:15:13 UTC, Jonathan M Davis 
wrote:
 accepted or not. If the dynamic arrays are being passed around 
 in a way that you don't know what their lifetimes are, then 
 they need to be GC allocated.
In the Prior Work section the DIP states: «Rust's notion of pointers owning the memory they point to, unless the pointer is borrowed, is equivalent with this DIP's notion of a slice "borrowing" a reference to another object which manages the memory.» So it does seem to imply that slices are borrowed views. Conceptually, dynamic arrays would be borrowed from the GC, thus you don't need to know their lifetimes. Lifetimes would come into play for non-dynamic-arrays, I guess, so scope... perhaps.
Nov 11 2019
prev sibling next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/11/19 5:27 AM, Mike Parker wrote:
 This is the feedback thread for the first round of Community Review for 
 DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
 
 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee
6e4/DIPs/DIP1025.md 
 
 
 All review-related feedback on and discussion of the DIP should occur in 
 this thread. The review period will end at 11:59 PM ET on November 25, 
 or when I make a post declaring it complete.
 
 At the end of Round 1, if further review is deemed necessary, the DIP 
 will be scheduled for another round of Community Review. Otherwise, it 
 will be queued for the Final Review and Formal Assessment.
 
 Anyone intending to post feedback in this thread is expected to be 
 familiar with the reviewer guidelines:
 
 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md
 
 *Please stay on topic!*
 
 Thanks in advance to all who participate.
This is a question of terminology. The dynamic array type in D is specified to be a slice. This has extreme benefits that I think we all will lament if this goes away. As I've said before, treating the slice type as the array type is super confusing and not technically true. A dynamic array owns its own data, the slice does not. In D, the dynamic array is created and maintained *typeless* in the GC. It's owned by the GC. A slice can point at a dynamic array, stack-allocated data, global/thread-local data, a malloc-allocated array, or any other piece of memory that you can possibly point at. As soon as you append to it, it NOW points at a GC-allocated array. This should be neither surprising, nor a problem, IF you adopt the right terminology. A fix here could be to actually DEFINE the dynamic array type, and allow the user to fetch it, and understand how it works. You can continue to allow the slice to be an interface to it, or you can use it directly. Please email me if you want more details. I 100% agree with Adam, just put nogc if you don't want to use the GC. This is a huge breaking change, and the much better solution is to make the desired behavior opt-in. A middle ground could be a newly defined "slice" type that can be used to opt-in. I'd say no to this change, unless we are talking D3. And if that's the case, let's start talking about all the other things we want to fix ;) -Steve
Nov 11 2019
prev sibling next sibling parent reply Jon Degenhardt <jond noreply.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md
The DIP includes a discussion of std.array.appender:
 A workaround for deprecating:
 
 a ~= b;
 is to use std.array.appender ...
The DIP should describe whether this includes both std.array.Appender and std.array.RefAppender or just std.array.Appender.
Nov 11 2019
parent Jon Degenhardt <jond noreply.com> writes:
On Monday, 11 November 2019 at 18:01:03 UTC, Jon Degenhardt wrote:
 On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md
The DIP includes a discussion of std.array.appender:
 A workaround for deprecating:
 
 a ~= b;
 is to use std.array.appender ...
The DIP should describe whether this includes both std.array.Appender and std.array.RefAppender or just std.array.Appender.
To clarify my previous comment, std.array.appender (lower-case) is a convenience function that can create both std.array.Appender and std.array.RefAppender. However, it is not obvious that std.array.RefAppender would be supported after acceptance of this DIP.
Nov 11 2019
prev sibling next sibling parent reply Meta <jared771 gmail.com> writes:
I've never been comfortable with the fact that T[] can refer to 
any type of memory, GC-allocated or not (or T*, for that matter), 
but I think this DIP is not the right approach. It severely 
pessimizes one of the most common use cases of slices in order to 
prevent memory corruption when the slice does not refer to a GC'd 
array. While this could be a big gain, it is also definitely a 
huge loss in terms of how much code would be broken.

I completely agree, though, that the fact that ~ may change 
slices referring to non-GC'd memory to GC'd memory is not a good 
thing - I just think this DIP would break too much code.

I can't see a clear way forward, either, because at a fundamental 
level in the type system, D treats GC'd and non-GC'd pointers the 
same. Therefore, it would not be enough to introduce a new type 
that exclusively refers to GC-allocated memory (i.e., new T* / 
T[new], or whatever), because slicing would still have to be 
disabled for non-GC'd pointers, and appending would still have to 
be disabled for non-GC'd slices.

Would it be enough to put an assert in GC.realloc/extend/free 
that asserts the memory is GC-owned, maybe tied to a compiler 
switch?
Nov 11 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/11/2019 12:50 PM, Meta wrote:
 Would it be enough to put an assert in GC.realloc/extend/free that asserts the 
 memory is GC-owned, maybe tied to a compiler switch?
Currently, for `a~=b` if `a` is not owned by the GC, it is copied to a new GC location by the runtime.
Nov 11 2019
parent Meta <jared771 gmail.com> writes:
On Monday, 11 November 2019 at 23:20:31 UTC, Walter Bright wrote:
 On 11/11/2019 12:50 PM, Meta wrote:
 Would it be enough to put an assert in GC.realloc/extend/free 
 that asserts the memory is GC-owned, maybe tied to a compiler 
 switch?
Currently, for `a~=b` if `a` is not owned by the GC, it is copied to a new GC location by the runtime.
Yes, my point being, if the concern is over manually-allocated pointers being invalidated by accidental reallocation by the GC, wouldn't the absolute least intrusive change possible be to put asserts in the relevant GC methods to make sure it's only reallocating/extending/freeing memory *it* owns? It would mean a crash at runtime only, and only in debug mode, but you get a semi-nice stack trace showing you the exact point you're trying to do something bad (assuming I properly understand the GC implementation, which I may not).
Nov 11 2019
prev sibling next sibling parent 12345swordy <alexanderheistermann gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.
I genuinely believe this is a wrong approach to this. The code breakage is potentially massive on this. It would be better to create a new dynamic array that only shrinks and never grow as a library solution. Why can't his be done via a library? -Alex
Nov 11 2019
prev sibling next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.
As noted by the other reviewers, - the restrictions proposed by the DIP imply much breaking changes. - the problem is a lack of distinction between slices and dynamic arrays. To that I also add that - the restrictions would be disruptive for a certain usage of D, especially shell-like programs, manipulating array of files, array of directories, program arguments, etc. so maybe that 1. At the local scope or when this is semantically possible, the compiler could keep track of the difference between a dynamic array and a slice, preventing growing. 2. When not possible anymore, e.g function parameter, a slice attribute could be used to apply the same restriction as in 1. --- void func1(int[] arr) { arr ~= 1; // OK arr.length += 1; // OK auto slice = arr[0..2]; slice.length += 1; // NG, local restriction (1.) } void func2( slice int[] arr) { arr ~= 1; // NG, slice attribute (2.) arr.length += 1; // NG, slice attribute (2.) auto slice = arr[0..2]; slice.length += 1; // NG, local restriction (1.) } --- As an alternative, instead of 2. the new semantics could be applied only if the function is safe (although this would give less control over individual parameters) --- void func2(int[] arr) // or trusted { arr ~= 1; // OK, not safe or trusted arr.length += 1; // OK, not safe or trusted auto slice = arr[0..2]; slice.length += 1; // NG, local restriction (1.) } void func3(int[] arr) safe { arr ~= 1; // NG, not allowed in safe func arr.length += 1; // NG, not allowed in safe func auto slice = arr[0..2]; slice.length += 1; // NG, local restriction (1.) } --- Of course ideally a full 1. would be better but then this requires complex DFA, which DMD doesn't know how to do yet, unless I'm not well informed.
Nov 11 2019
parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 11 November 2019 at 21:49:52 UTC, Basile B. wrote:

 2. When not possible anymore, e.g function parameter, a  slice 
 attribute could be used to apply the same restriction as in 1.
O God, please no! Pretty frankly, writing D cose is starting to became a masterclass of attribute handling: with move semantic, rvalue, live, and now that, I've practically lost the count!
Nov 11 2019
parent Basile B. <b2.temp gmx.com> writes:
On Monday, 11 November 2019 at 21:59:36 UTC, Paolo Invernizzi 
wrote:
 On Monday, 11 November 2019 at 21:49:52 UTC, Basile B. wrote:

 2. When not possible anymore, e.g function parameter, a  slice 
 attribute could be used to apply the same restriction as in 1.
O God, please no! Pretty frankly, writing D cose is starting to became a masterclass of attribute handling: with move semantic, rvalue, live, and now that, I've practically lost the count!
TBH me too. safe could be used instead but then much more breakage to expect.
Nov 11 2019
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 11.11.19 11:27, Mike Parker wrote:
 This is the feedback thread for the first round of Community Review for 
 DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
 
 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee
6e4/DIPs/DIP1025.md 
 
 
 All review-related feedback on and discussion of the DIP should occur in 
 this thread. The review period will end at 11:59 PM ET on November 25, 
 or when I make a post declaring it complete.
 
 At the end of Round 1, if further review is deemed necessary, the DIP 
 will be scheduled for another round of Community Review. Otherwise, it 
 will be queued for the Final Review and Formal Assessment.
 
 Anyone intending to post feedback in this thread is expected to be 
 familiar with the reviewer guidelines:
 
 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md
 
 *Please stay on topic!*
 
 Thanks in advance to all who participate.
Wrong season. April Fools' Day is in April.
Nov 11 2019
prev sibling next sibling parent Suleyman <sahmi.soulaimane gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
 [...]
I also find it a problem that dynamic arrays do not carry their allocator in their type, hence the problem with reallocation. It works for a GC only language and many people think of D like this and are not complaining about it, so their voice should be given the importance that it deserves. In addition, D also wants to compete in convenience with scripting languages. Clearly there is a good argument on both sides. A separation between a slice and an array which knows its allocator is needed, but the convenience hit is too big for those who never in their life called malloc, so it's hard to make a good decision on this. In general, D has made good decisions so far regarding this issue. For people who just want the convenience then things are great, the GC is dominant, malloc is system so they need to be careful with it. And for those who want to control memory allocation they have to use nogc. The problem comes when these two worlds talk to each other, the nogc world and the convenience world, the medium they use is a slice. But I don't find it that much of a problem, since the convenience world will never manually allocate or free anything, and the nogc world cannot assume anything about who allocated a slice, that's the beauty of the slice, that it's very portable.
Nov 11 2019
prev sibling next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.
I understand where this proposal comes from, but it would certainly cause a lot of breakage and it would cripple the universal feature common to all programming languages, arrays. Without filling the void this proposal leaves behind, I don't see any support for it. I would personally look at the conflation of dynamic arrays and slices. If you could somehow separate the 2 concepts and THEN constrain slices to only be able to shrink, that might work. (or keep them conflated but make borrowed slices non appendable.) (or make the implicit explicit by somehow requiring `.array` before appending). On the other hand, the fact that they are conflated also brings benefits, and you certainly don't want to loose them. It is a tough nut to crack. OT: In general I agree with the sentiment D's language features oftentimes depend too much on an all-knowing garbage collector. Maybe there is a way that behaviour could be made customisable for people who need that, instead of having everyone bear the costs. (e.g. I have done a lot of betterC programming lately, and while I can use my own custom allocators, things like closures still depend on a GC.)
Nov 12 2019
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 12 November 2019 at 08:41:52 UTC, Sebastiaan Koppe 
wrote:
 I would personally look at the conflation of dynamic arrays and 
 slices. If you could somehow separate the 2 concepts and THEN 
 constrain slices to only be able to shrink, that might work. 
 (or keep them conflated but make borrowed slices non 
 appendable.) (or make the implicit explicit by somehow 
 requiring `.array` before appending).

 On the other hand, the fact that they are conflated also brings 
 benefits, and you certainly don't want to loose them. It is a 
 tough nut to crack.
I'm starting to think that the issue is not that they are conflated, but that it isn't clear whether "dynamic" GC arrays are values/object, high-level/low-level or language-constructs/runtime-constructs. If concatenation expressions work with high-level values conceptually, and slices are views on those values, and GC-arrays are high-level-nonruntime-constucts, then things might become more clear. Then leave to the optimizer to estimate and preallocate a buffer size (or with hints). I don't think there is anything wrong with having high level concepts, but then they should be truly high level so that the optimizer can do the work and there should be some clear delineation between high-level and low-level types. (You could still allow low level operations temporarily, but the life-time of that would have to be made formal, so that the optimizer has free hands when the low-level work is done). But then you also need another low-level concept for views on an address-bound object/buffer/memory. So it is a tough nut, yes.
 In general I agree with the sentiment D's language features 
 oftentimes depend too much on an all-knowing garbage collector.
Yes, but is also cultural, in the sense that some people rely 100% on the garbage collector. So how can the type system ensure that the high quality libraries they write are usable in the more general programming context? If all programmers only relied on the GC by 30%, then maybe this would be less of an issue, as they would think differently about the use context of the library. The rather weakly typed C++ ecosystem works because there is some shared understanding of the use context and conventions. So that is an extra challenge for D. Maybe static analysis could help. Some innovation that can prove properties down the call chain fromt he call site (rather than all those attributes on the function signatures).
Nov 12 2019
prev sibling parent IGotD- <nise nise.com> writes:
On Tuesday, 12 November 2019 at 08:41:52 UTC, Sebastiaan Koppe 
wrote:
 I would personally look at the conflation of dynamic arrays and 
 slices. If you could somehow separate the 2 concepts and THEN 
 constrain slices to only be able to shrink, that might work.
Yes, this DIP needs to be suspended until we know what to do with the divorce between dynamic arrays and slices.
Nov 12 2019
prev sibling next sibling parent Tim <tim.dlang t-online.de> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md

 All review-related feedback on and discussion of the DIP should 
 occur in this thread. The review period will end at 11:59 PM ET 
 on November 25, or when I make a post declaring it complete.

 At the end of Round 1, if further review is deemed necessary, 
 the DIP will be scheduled for another round of Community 
 Review. Otherwise, it will be queued for the Final Review and 
 Formal Assessment.

 Anyone intending to post feedback in this thread is expected to 
 be familiar with the reviewer guidelines:

 https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

 *Please stay on topic!*

 Thanks in advance to all who participate.
If you create a slice from manually managed memory, you probably also have to make sure the slice does not outlive the memory. This means the slice has to be passed as scope to functions (or some pure functions). The DIP should only apply to scope slices, because other slices are probably allocated on the GC.
Nov 12 2019
prev sibling next sibling parent reply Dukc <ajieskola gmail.com> writes:
If we are to do this, we need to limit the breakage somehow. A 
few ideas, that could perhaps be used in some combination:

-Instead of disallowing appending on arrays, change the garbage 
collector so that it'll never append in place. Of course, this 
has the potential to be a performance killer so alone it won't be 
enough.

-Only disable GC appending in  safe or  live

-Come up with a way to mark some code or module to diallow GC 
appending, but do not disable it everywhere. Something that won't 
require yet another attribute in every function declaration.

-`scope` slices can be appended on.

Rust is already infamous for being hard to get into because of 
the borrow checker. So we definitely should not implement 
everything it does the same way. This DIP in it's current form 
would cause so much damage that if we fall behind because we 
won't do it, so be it.
Nov 12 2019
parent reply Jab <jab_293 gmall.com> writes:
On Tuesday, 12 November 2019 at 12:30:42 UTC, Dukc wrote:
 This DIP in it's current form would cause so much damage that 
 if we fall behind because we won't do it, so be it.
It is going to fall behind either way. This DIP focuses on falling behind Rust/C++ safety with manual memory management. But it completely ignores falling behind on simplicity/easy of use/unrestricted use with other languages that have a GC. The focus seems to only be on Rust/C++ lately. By trying to be safe with manual memory management you lose what the GC offers, this is in part due to the fact that GC pointers and malloc pointers are indistinguishable. It's a clash between two different ideologies that don't mix well.
Nov 12 2019
next sibling parent bachmeier <no spam.net> writes:
On Tuesday, 12 November 2019 at 17:09:30 UTC, Jab wrote:
 On Tuesday, 12 November 2019 at 12:30:42 UTC, Dukc wrote:
 This DIP in it's current form would cause so much damage that 
 if we fall behind because we won't do it, so be it.
It is going to fall behind either way. This DIP focuses on falling behind Rust/C++ safety with manual memory management. But it completely ignores falling behind on simplicity/easy of use/unrestricted use with other languages that have a GC. The focus seems to only be on Rust/C++ lately. By trying to be safe with manual memory management you lose what the GC offers, this is in part due to the fact that GC pointers and malloc pointers are indistinguishable. It's a clash between two different ideologies that don't mix well.
D will fall behind in terms of stability. The language is 20 years old. It shouldn't be breaking the thousands and thousands of lines of code I've already written unless we're moving to D 3.0 over a period of several years. It's hard to see how the language recovers from making a move like this with no planning and no communication. I don't see how anyone could recommend using D after something like this, except maybe to people you really don't like and aren't your boss.
Nov 12 2019
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 12.11.19 18:09, Jab wrote:
 On Tuesday, 12 November 2019 at 12:30:42 UTC, Dukc wrote:
 This DIP in it's current form would cause so much damage that if we 
 fall behind because we won't do it, so be it.
It is going to fall behind either way. This DIP focuses on falling behind Rust/C++ safety with manual memory management. But it completely ignores falling behind on simplicity/easy of use/unrestricted use with other languages that have a GC.
I agree with this.
 The focus seems to only be on Rust/C++ 
 lately. By trying to be safe with manual memory management you lose what 
 the GC offers, this is in part due to the fact that GC pointers and 
 malloc pointers are indistinguishable.
That's why the language needs to provide tools like borrowing, such that libraries can easily implement trusted smart pointers that carry around allocator/ownership information. In safe code, any accessible raw pointer always has to have unlimited lifetime. (In Rust, raw pointers are entirely inaccessible in safe code, because Rust has no built-in GC.)
 It's a clash between two different ideologies that don't mix well.
I don't agree with this. We can have both. We can enforce mutable aliasing constraints for memory accessible through shared, e.g., tracing GC-owned, pointers using runtime checks. (Rust also does this if you want to mutate memory through shared, e.g., reference counted, pointers.)
Nov 12 2019
prev sibling next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md
I don't know. C++ has std::vector and std::string_view, but our slices do these two things. It makes things easier sometimes but in the end you have to track ownership in your head, especially if you use nogc and -betterC. Does "dynamic arrays ARE slices" really hurt us? This change would break a great number of libraries, as said in the thread. Personally I have zero interest in more memory safety. On the other hand, maybe we can go "leap of faith" on this. It would also help if there was a good std::vector replacement in the stdlib, with explicit .capacity.
Nov 12 2019
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 12 November 2019 at 13:10:06 UTC, Guillaume Piolat 
wrote:
 [snip]

 I don't know.
 C++ has std::vector and std::string_view, but our slices do 
 these two things. It makes things easier sometimes but in the 
 end you have to track ownership in your head, especially if you 
 use  nogc and -betterC.

 Does "dynamic arrays ARE slices" really hurt us?

 [snip]
I keep getting a little mixed up when referring to slices and dynamic arrays. So some T[] x; is a dynamic array and auto y = x[a..b]; or auto y = x[]; is a slicing operation on x. And right now, slicing x also returns a dynamic array unless there is some opSlice defined that does something different. So one option would be to keep dynamic arrays as they are and change the slicing operations to return a slice type. They would be like dynamic arrays with the exception that you can't append to them or change the length. However, this would probably lead to a lot of breakage, even if you could have a .array function that can easily convert a slice to an array. What about instead something like below with a type that has appending disabled. I had tried to make length const, but I figured it was easier to just return data's length. I'm not sure if this works in the more general case though. Regardless, the benefit is that this is completely opt-in and requires no code-breakage. ``` struct View(T : U[], U) { T data; alias data this; this(T x) { data = x; } disable void opOpAssign(string op)(U rhs) if (op == "~") { data ~= rhs; } disable void opOpAssign(string op)(U[] rhs) if (op == "~") { data ~= rhs; } disable T[] opBinary(string op)(U rhs) if (op == "~") { return data ~ rhs; } disable T[] opBinary(string op)(U[] rhs) if (op == "~") { return data ~ rhs; } size_t length() { return data.length; } } auto view(T : U[], U)(T x) { return View!(T, U)(x); } void main() { int[] x = [0, 3, 5]; auto y = x[0..2].view; } ```
Nov 12 2019
parent jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 12 November 2019 at 15:59:57 UTC, jmh530 wrote:
 [snip]

      disable T[] opBinary(string op)(U rhs)
         if (op == "~")
     {
         return data ~ rhs;
     }

      disable T[] opBinary(string op)(U[] rhs)
         if (op == "~")
     {
         return data ~ rhs;
     }

 [snip]
Small error here T[] opBinary should be T opBinary
Nov 12 2019
prev sibling next sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

 https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md
See here for my response: https://gist.github.com/John-Colvin/c9c0b79bc9d47ed8d57ec9e959d0542b Included below for convenience and record: The DIP provides 2 examples to justify itself: 1) unpredictable mutable aliasing after appends 2) unpredictable changes of ownership caused by appends They can share the same cause (`~=`), but are two distinct symptoms, and can also have other causes. They have a unifying aspect which is unpredictable aliasing. The breakage of the DIP is immense, so there needs to be a commensurately large upside and no reasonable alternative path with less breakage. ```D int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10]; slice ~= 1; free(slice.ptr); // Oops! ``` `free` is not an ` safe` operation and requires some conditions to be met for correct use. These conditions are as follows: - `free` happens after all accesses to the allocation referenced by the passed pointer. - the allocation referenced by the passed pointer was made by `{m,c,re}alloc` as such, you can't really ever call `free` without completely trusting the path by which the data travelled, from inception (`{m,c,re}alloc`) to the call to `free`. This may be achieved by a variety of ways (e.g. a `MallocSlice` type that encapsulates the slice, or by having the path be very short and doing it manually) but fundamentally if I write a function taking a slice and then freeing the pointer, I cannot mark it as ` trusted`, regardless of whether `~=` is allowed or not. That `~=` will cause a new array to be allocated unpredictably is not the problem, the problem is that it can happen at all. If we look at it probabilistically: the problem is not that the probability of re-allocation is in `(0, 1)`, the problem is that it's not `0`. *Someone can write `slice = slice ~ 1` and get the same behaviour, so my considerations when calling `free` have not changed by disallowing `~=`.* ```D enum { dead, alive } int[] cat = new int[6]; cat[5] = alive; int[] b = cat; b ~= 1; // may or may not move b to new location b[5] = dead; // indeterminate whether cat[5] is dead or alive ``` This is a sticky one. `~=` applied to data where there are >1 references and >0 are mutable is liable to cause problems of unpredictable action at a distance. Removing `~=` makes it slightly harder to do this, but seeing as it's so very easy to do the same thing in other ways (`a = b ? a : (a ~ 2)` and many other trivial examples) it arguably doesn't matter much. I could believe it makes it less likely to do by accident. Overall, this change seems to not buy anyone much, but costs a lot. It correctly identifies a problem (or 2 problems, depending how you look at it), but the proposed solution does not appear on a simple reading to solve the problem. The DIP does not provide any examples of what is directly enabled by the change and I was not able to infer it from the current text. *The author should explain what previously impossible/unsafe code can now be made possible/safe given the proposed change.*
Nov 12 2019
parent reply Jab <jab_293 gmall.com> writes:
On Tuesday, 12 November 2019 at 15:54:22 UTC, John Colvin wrote:
 *The author should explain what previously impossible/unsafe 
 code can now be made possible/safe given the proposed change.*
Looking at the examples again, do they even show memory corruption? This would be a memory leak, not memory corruption: int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10]; slice ~= 1; free(slice.ptr); // Oops! This doesn't show memory corruption either, its a potential logic bug as it doesn't consider a side effect of array concatenation. enum { dead, alive } int[] cat = new int[6]; cat[5] = alive; int[] b = cat; b ~= 1; // may or may not move b to new location b[5] = dead; // indeterminate whether cat[5] is dead or alive Neither of these examples are pertinent to safe, which is aimed at reducing/removing memory corruption. This would be a more relevant problem for safe: int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10]; free(slice.ptr); slice ~= 1; // use after free As far as I'm aware none of the current previous DIPs resolve this do they?
Nov 12 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 12 November 2019 at 17:25:06 UTC, Jab wrote:
 This would be a memory leak, not memory corruption:

   int[] slice = cast(int*)malloc(10 * int.sizeof)[0 .. 10];
   slice ~= 1;
   free(slice.ptr); // Oops!
Depends on the implementation of the garbage collector and C free. If it frees the beginning of block that the GC has allocated using malloc then it would indeed cause memory corruption as soon at somebody else get the same block from malloc.
Nov 12 2019
prev sibling next sibling parent Tobias Pankrath <tobias pankrath.net> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":

https://github.com/dlang/DIPs/blob/1b525ec4c914c06bc286c1a6dc93bf1533ee56e4/DIPs/DIP1025.md
I am against it. 1. One of D's main selling points to me is productivity. We currently do not have the machinery in place to be as productive without the GC as with it. For example std.allocator is still experimental and there seems to be no movement on that front. First put the alternatives in place in the meantime there is nogc. 2. nogc 3. If we remove arrays the language has no useful builtin/included container anymore. std.container is no alternative, e.g.: https://issues.dlang.org/show_bug.cgi?id=13791
Nov 12 2019
prev sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
This review round is now officially terminated. The DIP author has withdrawn the DIP from consideration. I've marked the DIP as Withdrawn and summarized the review, as well as the author's reason for withdrawal: https://github.com/dlang/DIPs/blob/master/DIPs/other/DIP1025.md I've also updated the DIP process documentation to describe the Withdrawn status: https://github.com/dlang/DIPs/blob/master/docs/process-reviews.md#withdrawn-and-superseded-dips
Nov 13 2019
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 13.11.19 10:07, Mike Parker wrote:
 On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community Review 
 for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
This review round is now officially terminated. The DIP author has withdrawn the DIP from consideration. I've marked the DIP as Withdrawn and summarized the review, as well as the author's reason for withdrawal: https://github.com/dlang/DIPs/blob/master/DIPs/other/DIP1025.md I've also updated the DIP process documentation to describe the Withdrawn status: https://github.com/dlang/DIPs/blob/master/docs/process-reviews.md#withdrawn and-superseded-dips
Thanks!
Nov 13 2019
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, November 13, 2019 2:07:04 AM MST Mike Parker via Digitalmars-d 
wrote:
 On Monday, 11 November 2019 at 10:27:26 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community
 Review for DIP 1025, "Dynamic Arrays Only Shrink, Never Grow":
This review round is now officially terminated. The DIP author has withdrawn the DIP from consideration. I've marked the DIP as Withdrawn and summarized the review, as well as the author's reason for withdrawal: https://github.com/dlang/DIPs/blob/master/DIPs/other/DIP1025.md I've also updated the DIP process documentation to describe the Withdrawn status: https://github.com/dlang/DIPs/blob/master/docs/process-reviews.md#withdraw n-and-superseded-dips
Thank you, Walter. - Jonathan M Davis
Nov 13 2019