www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Append to dynamic array that was allocated via calloc

reply John Burton <john.burton jbmail.com> writes:
I can create a "slice" using non-gc allocated memory.

         int* ptr = cast(int*)calloc(int.sizeof, 10);
         int[] data = ptr[0..10];

If I don't want a memory leak I have to call free(ptr) somewhere 
as it won't be GC collected when data or ptr go out of scope. I 
presume there is nothing wrong with doing the above, other than 
perhaps there being better ways (and the memory leak if not 
free'd)

If I then write this :-

         data ~= 1;

What happens? It seems to successfully append an extra value to 
the array. It appears to "work" when I try it in my compiler but 
I don't understand how. Will this be trying to write beyond the 
memory I calloc'ed?
Jul 25 2017
next sibling parent Dukc <ajieskola gmail.com> writes:
On Tuesday, 25 July 2017 at 12:40:13 UTC, John Burton wrote:

 What happens? It seems to successfully append an extra value to 
 the array. It appears to "work" when I try it in my compiler 
 but I don't understand how. Will this be trying to write beyond 
 the memory I calloc'ed?


The language makes no guarantee. It may append in-place if the 
operating system finds out that the memory after it is not in 
use. But it may also allocate a whole new array (with gc) and 
copy the whole thing there. And since the original array was 
allocated manually, that results in a memory leak. Unless you 
check for that by storing the appended array in a new variable 
and compare their pointers.

The why part here is for efficiency reasons. We do not want to 
copy every time one appends, but if there's something in way that 
just needs to be done.

I think the correct way here is to use realloc(). Or 
std.container.Array. It is well possible there are other good 
options too.
Jul 25 2017
prev sibling next sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Tuesday, 25 July 2017 at 12:40:13 UTC, John Burton wrote:

 I can create a "slice" using non-gc allocated memory.

         int* ptr = cast(int*)calloc(int.sizeof, 10);
         int[] data = ptr[0..10];

 If I don't want a memory leak I have to call free(ptr) 
 somewhere as it won't be GC collected when data or ptr go out 
 of scope. I presume there is nothing wrong with doing the 
 above, other than perhaps there being better ways (and the 
 memory leak if not free'd)

 If I then write this :-

         data ~= 1;

 What happens? It seems to successfully append an extra value to 
 the array. It appears to "work" when I try it in my compiler 
 but I don't understand how. Will this be trying to write beyond 
 the memory I calloc'ed?


This should give you the answer:

writefln("Before: ptr = %s capacity = %s", slice.ptr, 
slice.capacity);
slice ~= 1;
writefln("After: ptr = %s capacity = %s", slice.ptr, 
slice.capacity);

It shows that before the append, the capacity is 0. That 
indicates that any append will cause a new allocation -- from the 
GC. The next writefln verifies this by showing a different value 
for ptr and a new capacity of 15.

In order for this to work, you'll need to manually manage the 
length and track the capacity yourself. If all you want is to 
allocate space for 10 ints, but not 10 actual ints, then 
something like this:

size_t capacity = 10;
int* ints = cast(int*)malloc(int.sizeof * capacity);
int[] slice = ints[0 .. 10];
slice.length = 0;
slice ~= 1;
--capacity;

Then reallocate the array when capacity reaches 0. Or just use 
std.container.array.Array which does all this for you.
Jul 25 2017
parent John Burton <john.burton jbmail.com> writes:
On Tuesday, 25 July 2017 at 13:24:36 UTC, Mike Parker wrote:

 On Tuesday, 25 July 2017 at 12:40:13 UTC, John Burton wrote:

 [...]


 This should give you the answer:

 writefln("Before: ptr = %s capacity = %s", slice.ptr, 
 slice.capacity);
 slice ~= 1;
 writefln("After: ptr = %s capacity = %s", slice.ptr, 
 slice.capacity);

 It shows that before the append, the capacity is 0. That 
 indicates that any append will cause a new allocation -- from 
 the GC. The next writefln verifies this by showing a different 
 value for ptr and a new capacity of 15.

 In order for this to work, you'll need to manually manage the 
 length and track the capacity yourself. If all you want is to 
 allocate space for 10 ints, but not 10 actual ints, then 
 something like this:

 size_t capacity = 10;
 int* ints = cast(int*)malloc(int.sizeof * capacity);
 int[] slice = ints[0 .. 10];
 slice.length = 0;
 slice ~= 1;
 --capacity;

 Then reallocate the array when capacity reaches 0. Or just use 
 std.container.array.Array which does all this for you.


Ok so it sounds like this is "safe" in that it will copy my data 
into GC memory and all work safely. I'll need to somehow keep 
track of my original memory and free it to avoid a memory leak...
(I don't plan to do any of this, but I wanted to understand)(
Jul 25 2017
prev sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 7/25/17 8:40 AM, John Burton wrote:

 I can create a "slice" using non-gc allocated memory.
 
          int* ptr = cast(int*)calloc(int.sizeof, 10);
          int[] data = ptr[0..10];
 
 If I don't want a memory leak I have to call free(ptr) somewhere as it 
 won't be GC collected when data or ptr go out of scope. I presume there 
 is nothing wrong with doing the above, other than perhaps there being 
 better ways (and the memory leak if not free'd)
 
 If I then write this :-
 
          data ~= 1;
 
 What happens? It seems to successfully append an extra value to the 
 array. It appears to "work" when I try it in my compiler but I don't 
 understand how. Will this be trying to write beyond the memory I calloc'ed?


What happens is the runtime detects that data is NOT pointing at an 
appendable GC-allocated block.

So it reallocates the whole block in the GC, and appends your element.

The original pointer is lost from data, and leaked (assuming you aren't 
manually freeing ptr somewhere).

I'd recommend using std.container.Array[1], which should do the right 
thing with malloc'd memory, and give you the nice operators such as ~=.

Or you have to resort to C calls for everything, if you don't want to 
deal with that.

Optionally, if you have a defined lifetime for this array in a scope, 
you can do this:

int *ptr = ...; // your calloc call
scope(exit) free(ptr); // ensure it is freed. Make SURE you don't change 
ptr inside the function.

Now you can use ~= on data, and it should still free the ptr at the end 
of the scope/function.

-Steve
Jul 25 2017