=?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:

We know that most of the time memory is allocated more than the 
requested amount. Is there a way to take advantage of that extra 
trailing space? (And potentially the pages that come after that.)

import core.memory;

void main()
{
     const count = 1;

     // I think there is extra capacity beyond the 'count' elements
     int* ptr = cast(int*)GC.malloc(count * int.sizeof);
     int[] arr = ptr[0 .. count];

     assert(arr.capacity == 0);
     arr.assumeSafeAppend;
     assert(arr.capacity == 0);    // still 0. :(
}

This issue puts std.array.array to a disadvantage compared to proper 
slices because array() involves the following call chain, the last of 
which does call GC.malloc:

   trustedAllocateArray
   uninitializedArray
   arrayAllocImpl

As a result, iota(10).array.assumeSafeAppend ends up having 0 capacity. :(

Ali

"monarch_dodra" <monarchdodra gmail.com> writes:

On Monday, 19 May 2014 at 06:08:18 UTC, Ali Çehreli wrote:
 We know that most of the time memory is allocated more than the 
 requested amount. Is there a way to take advantage of that 
 extra trailing space? (And potentially the pages that come 
 after that.)

 import core.memory;

 void main()
 {
     const count = 1;

     // I think there is extra capacity beyond the 'count' 
 elements
     int* ptr = cast(int*)GC.malloc(count * int.sizeof);
     int[] arr = ptr[0 .. count];

     assert(arr.capacity == 0);
     arr.assumeSafeAppend;
     assert(arr.capacity == 0);    // still 0. :(
 }

 This issue puts std.array.array to a disadvantage compared to 
 proper slices because array() involves the following call 
 chain, the last of which does call GC.malloc:

   trustedAllocateArray
   uninitializedArray
   arrayAllocImpl

 As a result, iota(10).array.assumeSafeAppend ends up having 0 
 capacity. :(

 Ali

Recently, a new function in druntime was added: "_d_newarrayU".

This void allocates a new array *with* appendable information. We 
can hope it will be given a more formal and public interface, and 
it would then be useable by array and/or Appender, to produce 
slices that have appendable data.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Mon, 19 May 2014 09:44:44 -0400, monarch_dodra <monarchdodra gmail.co=
m>  =

wrote:

 On Monday, 19 May 2014 at 06:08:18 UTC, Ali =C3=87ehreli wrote:
 We know that most of the time memory is allocated more than the  =


 requested amount. Is there a way to take advantage of that extra  =


 trailing space? (And potentially the pages that come after that.)

 import core.memory;

 void main()
 {
     const count =3D 1;

     // I think there is extra capacity beyond the 'count' elements
     int* ptr =3D cast(int*)GC.malloc(count * int.sizeof);
     int[] arr =3D ptr[0 .. count];

     assert(arr.capacity =3D=3D 0);
     arr.assumeSafeAppend;
     assert(arr.capacity =3D=3D 0);    // still 0. :(
 }


This is because a block must contain 'used' information in order for  =

capacity and assumeSafeAppend to work. This is enabled not only by addin=
g  =

the flag APPENDABLE to the allocation (or you can set the attribute  =

later), but you must ALSO properly set up the 'used' field. This is best=
  =

left to druntime. The mechanism to store the used size may change in the=
  =

future.

Is there a reason you wouldn't want to do int[] arr =3D new int[count]? =
It's  =

technically the same call.

 This issue puts std.array.array to a disadvantage compared to proper =


 =

 slices because array() involves the following call chain, the last of=


  =

 which does call GC.malloc:

   trustedAllocateArray
   uninitializedArray
   arrayAllocImpl


This is a bug. arrayAllocImpl should alloc using the proper functions an=
d  =

flags.

auto arr =3D array(1, 2, 3);
assert(arr.capacity !=3D 0); // should pass

 Recently, a new function in druntime was added: "_d_newarrayU".

 This void allocates a new array *with* appendable information. We can =

 =

 hope it will be given a more formal and public interface, and it would=

  =

 then be useable by array and/or Appender, to produce slices that have =

 =

 appendable data.

Cool, I didn't know this!

-Steve

=?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:

On 05/19/2014 06:55 AM, Steven Schveighoffer wrote:

 On Mon, 19 May 2014 09:44:44 -0400, monarch_dodra

 Recently, a new function in druntime was added: "_d_newarrayU".


 Cool, I didn't know this!

Thank you both! This information may be mentioned during a lightning 
talk at DConf. ;)

Ali

"monarch_dodra" <monarchdodra gmail.com> writes:

On Monday, 19 May 2014 at 13:55:00 UTC, Steven Schveighoffer 
wrote:
 On Monday, 19 May 2014 at 06:08:18 UTC, Ali Çehreli wrote:
 This issue puts std.array.array to a disadvantage compared to 
 proper slices because array() involves the following call 
 chain, the last of which does call GC.malloc:

  trustedAllocateArray
  uninitializedArray
  arrayAllocImpl


 This is a bug. arrayAllocImpl should alloc using the proper 
 functions and flags.

Well, Yes and no. The issue is that there is no interface 
available to achieve this, that wouldn't completely destroy what 
`array` is going for anyways. So it's more of a design issue than 
a bug proper.

https://issues.dlang.org/show_bug.cgi?id=12444
https://github.com/D-Programming-Language/phobos/pull/2044

The only way I'd know (currently) to make it work, is with 
reserve+assumeSafeAppend. But the issue with that approach is 
that it's not pure (because of the whole purity with global GC 
side effects deal).

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Mon, 19 May 2014 14:46:59 -0400, monarch_dodra <monarchdodra gmail.co=
m>  =

wrote:

 On Monday, 19 May 2014 at 13:55:00 UTC, Steven Schveighoffer wrote:
 On Monday, 19 May 2014 at 06:08:18 UTC, Ali =C3=87ehreli wrote:
 This issue puts std.array.array to a disadvantage compared to prope=




r  =

 slices because array() involves the following call chain, the last =




of  =

 which does call GC.malloc:

  trustedAllocateArray
  uninitializedArray
  arrayAllocImpl


 This is a bug. arrayAllocImpl should alloc using the proper functions=


  =

 and flags.

 Well, Yes and no. The issue is that there is no interface available to=

  =

 achieve this, that wouldn't completely destroy what `array` is going f=

or  =

 anyways. So it's more of a design issue than a bug proper.

Yes, I understand. When allocating an array, you need to initialize the =
 =

array properly, and the standard library should not have the  =

implementation details of the array management leaked into it. I think y=
ou  =

can replace the GC.malloc call with the new one you mentioned, no?

It's still a bug. The expectation of array(x, y, z) is that it's the sam=
e  =

as [x, y, z]. That is not true currently.

-Steve

"Dicebot" <public dicebot.lv> writes:

On Monday, 19 May 2014 at 13:44:45 UTC, monarch_dodra wrote:
 Recently, a new function in druntime was added: "_d_newarrayU".

 This void allocates a new array *with* appendable information. 
 We can hope it will be given a more formal and public 
 interface, and it would then be useable by array and/or 
 Appender, to produce slices that have appendable data.

Huh, will it also make possible to call `realloc` if capacity is 
exceeded? If it still resorts to GC in this case, utility of such 
addition sounds questionable.

"monarch_dodra" <monarchdodra gmail.com> writes:

On Monday, 19 May 2014 at 18:51:31 UTC, Dicebot wrote:
 If it still resorts to GC in this case, utility of such 
 addition sounds questionable.

It's not really an "addition" as much as it is a "necessary 
building block to make higher order GC functions work": For 
example, "dup" was recently made a druntime library implemented 
function, and as such, required that function to allocate, before 
building data onto it.

 Huh, will it also make possible to call `realloc` if capacity 
 is exceeded?

AFAIK, using the "GC.realloc" (or "GC.extent") function on it 
directly would not work. This may or may not be an issue with how 
"GC.realloc" is designed. The reason for this is because this 
functions are actually "extremelly" low level, and simply request 
GC memory, without knowing or caring about the APPENDABLE data. 
So while the calls could succeed, the result would not be useable.

Currently, you could just use "reserve" or simply allocate again, 
to achieve almost the desired result. reserve+assumeSafeAppend 
would basically be a "void-extend" (as opposed to "size", which 
would be an "initialized extend").

At the end of the day though, it can all be done, but it's really 
about what you want to expose in "object.d".

"Dicebot" <public dicebot.lv> writes:

On Monday, 19 May 2014 at 21:01:52 UTC, monarch_dodra wrote:
 Huh, will it also make possible to call `realloc` if capacity 
 is exceeded?

 AFAIK, using the "GC.realloc" (or "GC.extent") function on it 
 directly would not work. This may or may not be an issue with 
 how "GC.realloc" is designed. The reason for this is because 
 this functions are actually "extremelly" low level, and simply 
 request GC memory, without knowing or caring about the 
 APPENDABLE data. So while the calls could succeed, the result 
 would not be useable.

 Currently, you could just use "reserve" or simply allocate 
 again, to achieve almost the desired result. 
 reserve+assumeSafeAppend would basically be a "void-extend" (as 
 opposed to "size", which would be an "initialized extend").

 At the end of the day though, it can all be done, but it's 
 really about what you want to expose in "object.d".

I was actually thinking about plain C realloc as I didn't notice 
original example refers to GC malloc (oops!) Thus got an 
impression it can be actually possible to emulate appending to 
arbitrary allocated blocks. My bad, never mind :)