www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - New abstraction: Layout

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
I've been long bothered that the builtin .tupleof and our own 
abstractions Fields and RepresentationTypeTuple in std.traits - all omit 
the essential information of field offsets. That makes types that use 
align() to have the same .tupleof, Fields, and RepresentationTypeTuple 
even though they shouldn't.

The right answer is Layout a tuple of (offset, type) pairs describing 
entirely the memory layout of a type. We need such for memory 
allocation, garbage collection, serialization, and more.

The implementation turned out to be quite compact - 81 lines including a 
compile-time mergesort. Destroy!

https://github.com/dlang/phobos/pull/6192


Andrei
Feb 16 2018
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 17/02/2018 12:04 AM, Andrei Alexandrescu wrote:

 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - all omit 
 the essential information of field offsets. That makes types that use 
 align() to have the same .tupleof, Fields, and RepresentationTypeTuple 
 even though they shouldn't.
 
 The right answer is Layout a tuple of (offset, type) pairs describing 
 entirely the memory layout of a type. We need such for memory 
 allocation, garbage collection, serialization, and more.
 
 The implementation turned out to be quite compact - 81 lines including a 
 compile-time mergesort. Destroy!
 
 https://github.com/dlang/phobos/pull/6192
 
 
 Andrei


Could use the name for the field as well. At the minimum useful for 
debugging purposes.
Feb 16 2018
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 02/16/2018 10:10 PM, rikki cattermole wrote:

 On 17/02/2018 12:04 AM, Andrei Alexandrescu wrote:

 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - all 
 omit the essential information of field offsets. That makes types that 
 use align() to have the same .tupleof, Fields, and 
 RepresentationTypeTuple even though they shouldn't.

 The right answer is Layout a tuple of (offset, type) pairs describing 
 entirely the memory layout of a type. We need such for memory 
 allocation, garbage collection, serialization, and more.

 The implementation turned out to be quite compact - 81 lines including 
 a compile-time mergesort. Destroy!

 https://github.com/dlang/phobos/pull/6192


 Andrei


 
 Could use the name for the field as well. At the minimum useful for 
 debugging purposes.


That would be tricky because fields are decomposed down to primitive 
types. -- Andrei
Feb 17 2018
parent reply Nathan S. <no.public.email example.com> writes:
On Saturday, 17 February 2018 at 12:49:07 UTC, Andrei 
Alexandrescu wrote:

 On 02/16/2018 10:10 PM, rikki cattermole wrote:

 Could use the name for the field as well. At the minimum 
 useful for debugging purposes.


 That would be tricky because fields are decomposed down to 
 primitive types. -- Andrei


That's unfortunate. Not being able to get the names and types 
together is my main dissatisfaction with the current interface.
Feb 19 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/19/18 6:18 AM, Nathan S. wrote:

 On Saturday, 17 February 2018 at 12:49:07 UTC, Andrei Alexandrescu wrote:

 On 02/16/2018 10:10 PM, rikki cattermole wrote:

 Could use the name for the field as well. At the minimum useful for 
 debugging purposes.


 That would be tricky because fields are decomposed down to primitive 
 types. -- Andrei


 
 That's unfortunate. Not being able to get the names and types together 
 is my main dissatisfaction with the current interface.


.tupleof does all this:

struct S
{
   int x;
   double y;
}

static foreach(alias v; S.tupleof)
{
   pragma(msg, typeof(v));
   pragma(msg, v.offsetof);
   pragma(msg, __traits(identifier, v));
}

output while compiling:
int
0LU
x
double
8LU
y

It's admittedly an ugly interface...

-Steve
Feb 20 2018
prev sibling next sibling parent reply thedeemon <dlang thedeemon.com> writes:
On Saturday, 17 February 2018 at 00:04:16 UTC, Andrei 
Alexandrescu wrote:

 The implementation turned out to be quite compact - 81 lines 
 including a compile-time mergesort. Destroy!


Off-topic:
I've just realized Andrei puts "Destroy!" at the end of his 
messages because it's the end of scope and he wants to free any 
allocated resources. ;)
Feb 17 2018
parent psychoRabbit <meagain meagain.com> writes:
On Saturday, 17 February 2018 at 09:30:23 UTC, thedeemon wrote:

 On Saturday, 17 February 2018 at 00:04:16 UTC, Andrei 
 Alexandrescu wrote:

 The implementation turned out to be quite compact - 81 lines 
 including a compile-time mergesort. Destroy!


 Off-topic:
 I've just realized Andrei puts "Destroy!" at the end of his 
 messages because it's the end of scope and he wants to free any 
 allocated resources. ;)


We're talking about Andrei here, so it's far more likely 
'destroy' is an efficient, compact, generic term, that can take 
on many different meanings - depending on the context in which it 
used.

'D'estroy!
Feb 17 2018
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/16/18 7:04 PM, Andrei Alexandrescu wrote:

 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - all omit 
 the essential information of field offsets. That makes types that use 
 align() to have the same .tupleof, Fields, and RepresentationTypeTuple 
 even though they shouldn't.
 
 The right answer is Layout a tuple of (offset, type) pairs describing 
 entirely the memory layout of a type. We need such for memory 
 allocation, garbage collection, serialization, and more.
 
 The implementation turned out to be quite compact - 81 lines including a 
 compile-time mergesort. Destroy!
 
 https://github.com/dlang/phobos/pull/6192


Can't you just use offsetof?

struct S
{
    ubyte x;
    int y;
}

static foreach(i; 0 .. s.tupleof.length)
{
    writeln(s.tupleof[i].offsetof);
}

outputs:
0
4

-Steve
Feb 17 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/17/18 8:19 AM, Steven Schveighoffer wrote:

 On 2/16/18 7:04 PM, Andrei Alexandrescu wrote:

 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - all 
 omit the essential information of field offsets. That makes types that 
 use align() to have the same .tupleof, Fields, and 
 RepresentationTypeTuple even though they shouldn't.

 The right answer is Layout a tuple of (offset, type) pairs describing 
 entirely the memory layout of a type. We need such for memory 
 allocation, garbage collection, serialization, and more.

 The implementation turned out to be quite compact - 81 lines including 
 a compile-time mergesort. Destroy!

 https://github.com/dlang/phobos/pull/6192


 
 Can't you just use offsetof?
 
 struct S
 {
     ubyte x;
     int y;
 }
 
 static foreach(i; 0 .. s.tupleof.length)
 {
     writeln(s.tupleof[i].offsetof);
 }
 
 outputs:
 0
 4


I found this also works:

static foreach(alias x; S.tupleof)
{
    writeln(x.offsetof);
}

-Steve
Feb 17 2018
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 02/17/2018 09:03 AM, Steven Schveighoffer wrote:

 On 2/17/18 8:19 AM, Steven Schveighoffer wrote:

 On 2/16/18 7:04 PM, Andrei Alexandrescu wrote:

 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - all 
 omit the essential information of field offsets. That makes types 
 that use align() to have the same .tupleof, Fields, and 
 RepresentationTypeTuple even though they shouldn't.

 The right answer is Layout a tuple of (offset, type) pairs describing 
 entirely the memory layout of a type. We need such for memory 
 allocation, garbage collection, serialization, and more.

 The implementation turned out to be quite compact - 81 lines 
 including a compile-time mergesort. Destroy!

 https://github.com/dlang/phobos/pull/6192


 Can't you just use offsetof?

 struct S
 {
     ubyte x;
     int y;
 }

 static foreach(i; 0 .. s.tupleof.length)
 {
     writeln(s.tupleof[i].offsetof);
 }

 outputs:
 0
 4


 
 I found this also works:
 
 static foreach(alias x; S.tupleof)
 {
     writeln(x.offsetof);
 }


Yes, the implementation uses offsetof. -- Andrei
Feb 17 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/17/18 9:59 AM, Andrei Alexandrescu wrote:

 On 02/17/2018 09:03 AM, Steven Schveighoffer wrote:

 I found this also works:

 static foreach(alias x; S.tupleof)
 {
     writeln(x.offsetof);
 }


 
 Yes, the implementation uses offsetof.
 


I guess I'm just confused based on the statement "the builtin .tupleof 
... [omits] the essential information of field offsets."

What is this construct giving us that .tupleof doesn't?

-Steve
Feb 17 2018
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Saturday, 17 February 2018 at 19:37:12 UTC, Steven 
Schveighoffer wrote:

 On 2/17/18 9:59 AM, Andrei Alexandrescu wrote:

 On 02/17/2018 09:03 AM, Steven Schveighoffer wrote:

 I found this also works:

 static foreach(alias x; S.tupleof)
 {
     writeln(x.offsetof);
 }


 
 Yes, the implementation uses offsetof.
 


 I guess I'm just confused based on the statement "the builtin 
 .tupleof ... [omits] the essential information of field 
 offsets."

 What is this construct giving us that .tupleof doesn't?


I guess the construct captures offsets as part of type. This is 
useful for allocators + 2 things with same Layout can be 
bitblitted to each other.


 -Steve
Feb 18 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/18/18 4:52 AM, Dmitry Olshansky wrote:

 On Saturday, 17 February 2018 at 19:37:12 UTC, Steven Schveighoffer wrote:

 On 2/17/18 9:59 AM, Andrei Alexandrescu wrote:

 On 02/17/2018 09:03 AM, Steven Schveighoffer wrote:

 I found this also works:

 static foreach(alias x; S.tupleof)
 {
     writeln(x.offsetof);
 }


 Yes, the implementation uses offsetof.


 I guess I'm just confused based on the statement "the builtin .tupleof 
 ... [omits] the essential information of field offsets."

 What is this construct giving us that .tupleof doesn't?


 
 I guess the construct captures offsets as part of type. This is useful 
 for allocators + 2 things with same Layout can be bitblitted to each other.


I haven't looked at it in depth, so I didn't know the result of the 
abstraction (I thought it was a tuple, or a pair of tuples).

Note, you could do this without the need for a new abstraction, simply 
with .tupleof on the type itself.

And static foreach has made this much simpler. But definitely the 
interface to getting things from tupleof is not consistent. This reason 
alone may be cause to add such a construct.

-Steve
Feb 20 2018
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 02/20/2018 07:34 AM, Steven Schveighoffer wrote:

 On 2/18/18 4:52 AM, Dmitry Olshansky wrote:

 On Saturday, 17 February 2018 at 19:37:12 UTC, Steven Schveighoffer 
 wrote:

 On 2/17/18 9:59 AM, Andrei Alexandrescu wrote:

 On 02/17/2018 09:03 AM, Steven Schveighoffer wrote:

 I found this also works:

 static foreach(alias x; S.tupleof)
 {
     writeln(x.offsetof);
 }


 Yes, the implementation uses offsetof.


 I guess I'm just confused based on the statement "the builtin 
 .tupleof ... [omits] the essential information of field offsets."

 What is this construct giving us that .tupleof doesn't?


 I guess the construct captures offsets as part of type. This is useful 
 for allocators + 2 things with same Layout can be bitblitted to each 
 other.


 
 I haven't looked at it in depth, so I didn't know the result of the 
 abstraction (I thought it was a tuple, or a pair of tuples).
 
 Note, you could do this without the need for a new abstraction, simply 
 with .tupleof on the type itself.
 
 And static foreach has made this much simpler. But definitely the 
 interface to getting things from tupleof is not consistent. This reason 
 alone may be cause to add such a construct.


There's the difference that with inline static foreach you can express 
one processing of one layout, whereas with a structured result you can 
express the notion of any processing of any layout. The distinction 
between an inlined for loop and the map function comes to mind.

In particular, static foreach would be a good implementation device for 
Layout. I happened to express it a different way because it seemed 
simpler, but contributions welcome.

Next step is to manipulate layouts. I have in mind an algorithm 
"eraseNonPointerTypes". It does the following. Consider:

struct A
{
     int a, b;
     string x;
     double c;
     int[] y;
}

So the layout is Layout!(0, int, 4, int, 8, string, 24, double, 32, 
int[]). Next we want to not care about non-pointers - just merge all 
that nonsense together. So eraseNonPointerTypes applied to this layout 
would yield Layout!(0, ubyte[8], 8, string, 24, ubyte[8], 32, int[]).

At this point we figure that this partially erased layout is the same as 
for this other type:

struct B
{
     double a;
     string b;
     long c;
     int[] d;
}

And the ba dum tss of it all is that you can allocate an A, deallocate 
it, then allocate a B in the same memory - all in safe code. Even if you 
have a dangling pointer to A messing with a B object, the code will be 
incorrect but will not lose memory safety.

This is a major part of Alexandru Jercaianu's work on Blizzard - a 
memory-safe allocation framework. I am so excited about the impending 
release I can't stand myself.


Andrei
Feb 20 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/20/18 10:56 PM, Andrei Alexandrescu wrote:

 On 02/20/2018 07:34 AM, Steven Schveighoffer wrote:

 I haven't looked at it in depth, so I didn't know the result of the 
 abstraction (I thought it was a tuple, or a pair of tuples).

 Note, you could do this without the need for a new abstraction, simply 
 with .tupleof on the type itself.

 And static foreach has made this much simpler. But definitely the 
 interface to getting things from tupleof is not consistent. This 
 reason alone may be cause to add such a construct.


 
 There's the difference that with inline static foreach you can express 
 one processing of one layout, whereas with a structured result you can 
 express the notion of any processing of any layout. The distinction 
 between an inlined for loop and the map function comes to mind.
 


[snip]

Yep, this all makes sense, thanks! Sounds very similar to the Strawman 
structs thing, only probably better thought out ;)

-Steve
Feb 21 2018
prev sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Saturday, 17 February 2018 at 00:04:16 UTC, Andrei 
Alexandrescu wrote:

 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - 
 all omit the essential information of field offsets. That makes 
 types that use align() to have the same .tupleof, Fields, and 
 RepresentationTypeTuple even though they shouldn't.

 The right answer is Layout a tuple of (offset, type) pairs 
 describing entirely the memory layout of a type. We need such 
 for memory allocation, garbage collection, serialization, and 
 more.

 The implementation turned out to be quite compact - 81 lines 
 including a compile-time mergesort. Destroy!

 https://github.com/dlang/phobos/pull/6192


 Andrei


how does this deal with static members ?
Feb 20 2018
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, February 21, 2018 05:53:31 Basile B. via Digitalmars-d wrote:

 On Saturday, 17 February 2018 at 00:04:16 UTC, Andrei

 Alexandrescu wrote:

 I've been long bothered that the builtin .tupleof and our own
 abstractions Fields and RepresentationTypeTuple in std.traits -
 all omit the essential information of field offsets. That makes
 types that use align() to have the same .tupleof, Fields, and
 RepresentationTypeTuple even though they shouldn't.

 The right answer is Layout a tuple of (offset, type) pairs
 describing entirely the memory layout of a type. We need such
 for memory allocation, garbage collection, serialization, and
 more.

 The implementation turned out to be quite compact - 81 lines
 including a compile-time mergesort. Destroy!

 https://github.com/dlang/phobos/pull/6192


 Andrei


 how does this deal with static members ?


It's built on tupleof, so it doesn't do anything with static members. But
static members have nothing to do with the layout of a type, so it wouldn't
make sense for it to do anything with static members.

- Jonathan M Davis
Feb 20 2018
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 02/21/2018 12:53 AM, Basile B. wrote:

 On Saturday, 17 February 2018 at 00:04:16 UTC, Andrei Alexandrescu wrote:

 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - all 
 omit the essential information of field offsets. That makes types that 
 use align() to have the same .tupleof, Fields, and 
 RepresentationTypeTuple even though they shouldn't.

 The right answer is Layout a tuple of (offset, type) pairs describing 
 entirely the memory layout of a type. We need such for memory 
 allocation, garbage collection, serialization, and more.

 The implementation turned out to be quite compact - 81 lines including 
 a compile-time mergesort. Destroy!

 https://github.com/dlang/phobos/pull/6192


 Andrei


 
 how does this deal with static members ?


Static members are not part of the layout - aside from lexical scoping, 
they have the same regime as module-level static data. -- Andrei
Feb 21 2018