www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - New abstraction: Layout

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
I've been long bothered that the builtin .tupleof and our own 
abstractions Fields and RepresentationTypeTuple in std.traits - all omit 
the essential information of field offsets. That makes types that use 
align() to have the same .tupleof, Fields, and RepresentationTypeTuple 
even though they shouldn't.

The right answer is Layout a tuple of (offset, type) pairs describing 
entirely the memory layout of a type. We need such for memory 
allocation, garbage collection, serialization, and more.

The implementation turned out to be quite compact - 81 lines including a 
compile-time mergesort. Destroy!

https://github.com/dlang/phobos/pull/6192


Andrei
Feb 16 2018
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 17/02/2018 12:04 AM, Andrei Alexandrescu wrote:
 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - all omit 
 the essential information of field offsets. That makes types that use 
 align() to have the same .tupleof, Fields, and RepresentationTypeTuple 
 even though they shouldn't.
 
 The right answer is Layout a tuple of (offset, type) pairs describing 
 entirely the memory layout of a type. We need such for memory 
 allocation, garbage collection, serialization, and more.
 
 The implementation turned out to be quite compact - 81 lines including a 
 compile-time mergesort. Destroy!
 
 https://github.com/dlang/phobos/pull/6192
 
 
 Andrei
Could use the name for the field as well. At the minimum useful for debugging purposes.
Feb 16 2018
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 02/16/2018 10:10 PM, rikki cattermole wrote:
 On 17/02/2018 12:04 AM, Andrei Alexandrescu wrote:
 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - all 
 omit the essential information of field offsets. That makes types that 
 use align() to have the same .tupleof, Fields, and 
 RepresentationTypeTuple even though they shouldn't.

 The right answer is Layout a tuple of (offset, type) pairs describing 
 entirely the memory layout of a type. We need such for memory 
 allocation, garbage collection, serialization, and more.

 The implementation turned out to be quite compact - 81 lines including 
 a compile-time mergesort. Destroy!

 https://github.com/dlang/phobos/pull/6192


 Andrei
Could use the name for the field as well. At the minimum useful for debugging purposes.
That would be tricky because fields are decomposed down to primitive types. -- Andrei
Feb 17 2018
parent reply Nathan S. <no.public.email example.com> writes:
On Saturday, 17 February 2018 at 12:49:07 UTC, Andrei 
Alexandrescu wrote:
 On 02/16/2018 10:10 PM, rikki cattermole wrote:
 Could use the name for the field as well. At the minimum 
 useful for debugging purposes.
That would be tricky because fields are decomposed down to primitive types. -- Andrei
That's unfortunate. Not being able to get the names and types together is my main dissatisfaction with the current interface.
Feb 19 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/19/18 6:18 AM, Nathan S. wrote:
 On Saturday, 17 February 2018 at 12:49:07 UTC, Andrei Alexandrescu wrote:
 On 02/16/2018 10:10 PM, rikki cattermole wrote:
 Could use the name for the field as well. At the minimum useful for 
 debugging purposes.
That would be tricky because fields are decomposed down to primitive types. -- Andrei
That's unfortunate. Not being able to get the names and types together is my main dissatisfaction with the current interface.
.tupleof does all this: struct S { int x; double y; } static foreach(alias v; S.tupleof) { pragma(msg, typeof(v)); pragma(msg, v.offsetof); pragma(msg, __traits(identifier, v)); } output while compiling: int 0LU x double 8LU y It's admittedly an ugly interface... -Steve
Feb 20 2018
prev sibling next sibling parent reply thedeemon <dlang thedeemon.com> writes:
On Saturday, 17 February 2018 at 00:04:16 UTC, Andrei 
Alexandrescu wrote:
 The implementation turned out to be quite compact - 81 lines 
 including a compile-time mergesort. Destroy!
Off-topic: I've just realized Andrei puts "Destroy!" at the end of his messages because it's the end of scope and he wants to free any allocated resources. ;)
Feb 17 2018
parent psychoRabbit <meagain meagain.com> writes:
On Saturday, 17 February 2018 at 09:30:23 UTC, thedeemon wrote:
 On Saturday, 17 February 2018 at 00:04:16 UTC, Andrei 
 Alexandrescu wrote:
 The implementation turned out to be quite compact - 81 lines 
 including a compile-time mergesort. Destroy!
Off-topic: I've just realized Andrei puts "Destroy!" at the end of his messages because it's the end of scope and he wants to free any allocated resources. ;)
We're talking about Andrei here, so it's far more likely 'destroy' is an efficient, compact, generic term, that can take on many different meanings - depending on the context in which it used. 'D'estroy!
Feb 17 2018
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/16/18 7:04 PM, Andrei Alexandrescu wrote:
 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - all omit 
 the essential information of field offsets. That makes types that use 
 align() to have the same .tupleof, Fields, and RepresentationTypeTuple 
 even though they shouldn't.
 
 The right answer is Layout a tuple of (offset, type) pairs describing 
 entirely the memory layout of a type. We need such for memory 
 allocation, garbage collection, serialization, and more.
 
 The implementation turned out to be quite compact - 81 lines including a 
 compile-time mergesort. Destroy!
 
 https://github.com/dlang/phobos/pull/6192
Can't you just use offsetof? struct S { ubyte x; int y; } static foreach(i; 0 .. s.tupleof.length) { writeln(s.tupleof[i].offsetof); } outputs: 0 4 -Steve
Feb 17 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/17/18 8:19 AM, Steven Schveighoffer wrote:
 On 2/16/18 7:04 PM, Andrei Alexandrescu wrote:
 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - all 
 omit the essential information of field offsets. That makes types that 
 use align() to have the same .tupleof, Fields, and 
 RepresentationTypeTuple even though they shouldn't.

 The right answer is Layout a tuple of (offset, type) pairs describing 
 entirely the memory layout of a type. We need such for memory 
 allocation, garbage collection, serialization, and more.

 The implementation turned out to be quite compact - 81 lines including 
 a compile-time mergesort. Destroy!

 https://github.com/dlang/phobos/pull/6192
Can't you just use offsetof? struct S {    ubyte x;    int y; } static foreach(i; 0 .. s.tupleof.length) {    writeln(s.tupleof[i].offsetof); } outputs: 0 4
I found this also works: static foreach(alias x; S.tupleof) { writeln(x.offsetof); } -Steve
Feb 17 2018
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 02/17/2018 09:03 AM, Steven Schveighoffer wrote:
 On 2/17/18 8:19 AM, Steven Schveighoffer wrote:
 On 2/16/18 7:04 PM, Andrei Alexandrescu wrote:
 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - all 
 omit the essential information of field offsets. That makes types 
 that use align() to have the same .tupleof, Fields, and 
 RepresentationTypeTuple even though they shouldn't.

 The right answer is Layout a tuple of (offset, type) pairs describing 
 entirely the memory layout of a type. We need such for memory 
 allocation, garbage collection, serialization, and more.

 The implementation turned out to be quite compact - 81 lines 
 including a compile-time mergesort. Destroy!

 https://github.com/dlang/phobos/pull/6192
Can't you just use offsetof? struct S {     ubyte x;     int y; } static foreach(i; 0 .. s.tupleof.length) {     writeln(s.tupleof[i].offsetof); } outputs: 0 4
I found this also works: static foreach(alias x; S.tupleof) {    writeln(x.offsetof); }
Yes, the implementation uses offsetof. -- Andrei
Feb 17 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/17/18 9:59 AM, Andrei Alexandrescu wrote:
 On 02/17/2018 09:03 AM, Steven Schveighoffer wrote:
 I found this also works:

 static foreach(alias x; S.tupleof)
 {
     writeln(x.offsetof);
 }
Yes, the implementation uses offsetof.
I guess I'm just confused based on the statement "the builtin .tupleof ... [omits] the essential information of field offsets." What is this construct giving us that .tupleof doesn't? -Steve
Feb 17 2018
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Saturday, 17 February 2018 at 19:37:12 UTC, Steven 
Schveighoffer wrote:
 On 2/17/18 9:59 AM, Andrei Alexandrescu wrote:
 On 02/17/2018 09:03 AM, Steven Schveighoffer wrote:
 I found this also works:

 static foreach(alias x; S.tupleof)
 {
     writeln(x.offsetof);
 }
Yes, the implementation uses offsetof.
I guess I'm just confused based on the statement "the builtin .tupleof ... [omits] the essential information of field offsets." What is this construct giving us that .tupleof doesn't?
I guess the construct captures offsets as part of type. This is useful for allocators + 2 things with same Layout can be bitblitted to each other.
 -Steve
Feb 18 2018
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/18/18 4:52 AM, Dmitry Olshansky wrote:
 On Saturday, 17 February 2018 at 19:37:12 UTC, Steven Schveighoffer wrote:
 On 2/17/18 9:59 AM, Andrei Alexandrescu wrote:
 On 02/17/2018 09:03 AM, Steven Schveighoffer wrote:
 I found this also works:

 static foreach(alias x; S.tupleof)
 {
     writeln(x.offsetof);
 }
Yes, the implementation uses offsetof.
I guess I'm just confused based on the statement "the builtin .tupleof ... [omits] the essential information of field offsets." What is this construct giving us that .tupleof doesn't?
I guess the construct captures offsets as part of type. This is useful for allocators + 2 things with same Layout can be bitblitted to each other.
I haven't looked at it in depth, so I didn't know the result of the abstraction (I thought it was a tuple, or a pair of tuples). Note, you could do this without the need for a new abstraction, simply with .tupleof on the type itself. And static foreach has made this much simpler. But definitely the interface to getting things from tupleof is not consistent. This reason alone may be cause to add such a construct. -Steve
Feb 20 2018
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 02/20/2018 07:34 AM, Steven Schveighoffer wrote:
 On 2/18/18 4:52 AM, Dmitry Olshansky wrote:
 On Saturday, 17 February 2018 at 19:37:12 UTC, Steven Schveighoffer 
 wrote:
 On 2/17/18 9:59 AM, Andrei Alexandrescu wrote:
 On 02/17/2018 09:03 AM, Steven Schveighoffer wrote:
 I found this also works:

 static foreach(alias x; S.tupleof)
 {
     writeln(x.offsetof);
 }
Yes, the implementation uses offsetof.
I guess I'm just confused based on the statement "the builtin .tupleof ... [omits] the essential information of field offsets." What is this construct giving us that .tupleof doesn't?
I guess the construct captures offsets as part of type. This is useful for allocators + 2 things with same Layout can be bitblitted to each other.
I haven't looked at it in depth, so I didn't know the result of the abstraction (I thought it was a tuple, or a pair of tuples). Note, you could do this without the need for a new abstraction, simply with .tupleof on the type itself. And static foreach has made this much simpler. But definitely the interface to getting things from tupleof is not consistent. This reason alone may be cause to add such a construct.
There's the difference that with inline static foreach you can express one processing of one layout, whereas with a structured result you can express the notion of any processing of any layout. The distinction between an inlined for loop and the map function comes to mind. In particular, static foreach would be a good implementation device for Layout. I happened to express it a different way because it seemed simpler, but contributions welcome. Next step is to manipulate layouts. I have in mind an algorithm "eraseNonPointerTypes". It does the following. Consider: struct A { int a, b; string x; double c; int[] y; } So the layout is Layout!(0, int, 4, int, 8, string, 24, double, 32, int[]). Next we want to not care about non-pointers - just merge all that nonsense together. So eraseNonPointerTypes applied to this layout would yield Layout!(0, ubyte[8], 8, string, 24, ubyte[8], 32, int[]). At this point we figure that this partially erased layout is the same as for this other type: struct B { double a; string b; long c; int[] d; } And the ba dum tss of it all is that you can allocate an A, deallocate it, then allocate a B in the same memory - all in safe code. Even if you have a dangling pointer to A messing with a B object, the code will be incorrect but will not lose memory safety. This is a major part of Alexandru Jercaianu's work on Blizzard - a memory-safe allocation framework. I am so excited about the impending release I can't stand myself. Andrei
Feb 20 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/20/18 10:56 PM, Andrei Alexandrescu wrote:
 On 02/20/2018 07:34 AM, Steven Schveighoffer wrote:
 I haven't looked at it in depth, so I didn't know the result of the 
 abstraction (I thought it was a tuple, or a pair of tuples).

 Note, you could do this without the need for a new abstraction, simply 
 with .tupleof on the type itself.

 And static foreach has made this much simpler. But definitely the 
 interface to getting things from tupleof is not consistent. This 
 reason alone may be cause to add such a construct.
There's the difference that with inline static foreach you can express one processing of one layout, whereas with a structured result you can express the notion of any processing of any layout. The distinction between an inlined for loop and the map function comes to mind.
[snip] Yep, this all makes sense, thanks! Sounds very similar to the Strawman structs thing, only probably better thought out ;) -Steve
Feb 21 2018
prev sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Saturday, 17 February 2018 at 00:04:16 UTC, Andrei 
Alexandrescu wrote:
 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - 
 all omit the essential information of field offsets. That makes 
 types that use align() to have the same .tupleof, Fields, and 
 RepresentationTypeTuple even though they shouldn't.

 The right answer is Layout a tuple of (offset, type) pairs 
 describing entirely the memory layout of a type. We need such 
 for memory allocation, garbage collection, serialization, and 
 more.

 The implementation turned out to be quite compact - 81 lines 
 including a compile-time mergesort. Destroy!

 https://github.com/dlang/phobos/pull/6192


 Andrei
how does this deal with static members ?
Feb 20 2018
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, February 21, 2018 05:53:31 Basile B. via Digitalmars-d wrote:
 On Saturday, 17 February 2018 at 00:04:16 UTC, Andrei

 Alexandrescu wrote:
 I've been long bothered that the builtin .tupleof and our own
 abstractions Fields and RepresentationTypeTuple in std.traits -
 all omit the essential information of field offsets. That makes
 types that use align() to have the same .tupleof, Fields, and
 RepresentationTypeTuple even though they shouldn't.

 The right answer is Layout a tuple of (offset, type) pairs
 describing entirely the memory layout of a type. We need such
 for memory allocation, garbage collection, serialization, and
 more.

 The implementation turned out to be quite compact - 81 lines
 including a compile-time mergesort. Destroy!

 https://github.com/dlang/phobos/pull/6192


 Andrei
how does this deal with static members ?
It's built on tupleof, so it doesn't do anything with static members. But static members have nothing to do with the layout of a type, so it wouldn't make sense for it to do anything with static members. - Jonathan M Davis
Feb 20 2018
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 02/21/2018 12:53 AM, Basile B. wrote:
 On Saturday, 17 February 2018 at 00:04:16 UTC, Andrei Alexandrescu wrote:
 I've been long bothered that the builtin .tupleof and our own 
 abstractions Fields and RepresentationTypeTuple in std.traits - all 
 omit the essential information of field offsets. That makes types that 
 use align() to have the same .tupleof, Fields, and 
 RepresentationTypeTuple even though they shouldn't.

 The right answer is Layout a tuple of (offset, type) pairs describing 
 entirely the memory layout of a type. We need such for memory 
 allocation, garbage collection, serialization, and more.

 The implementation turned out to be quite compact - 81 lines including 
 a compile-time mergesort. Destroy!

 https://github.com/dlang/phobos/pull/6192


 Andrei
how does this deal with static members ?
Static members are not part of the layout - aside from lexical scoping, they have the same regime as module-level static data. -- Andrei
Feb 21 2018