www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Templates, D way

reply Void-995 <void995 gmail.com> writes:
Hi, everyone. I'm pretty new to D and trying my first steps in 
it. Currently I'm trying to port some code from C/C++ with pretty 
weird data structures and don't like idea of making boilerplate 
functions for accessing sub-lists in main binary structure (lets 
not talk about it's design, it's from legacy thing I want to deal 
with and I can't change the format itself or writing a lot of 
additional parsing code). With hour I spent on trying I've ended 
with what you may see below, but I wonder if there is more 
pleasant variant of re-writing that template as I haven't found 
anything better in either books or online documentation yet:

template DataList(const char[] listName, T, alias dataOffset, 
alias listLength)
{
	const char[] DataList = format(q{
		%s[] %s()
		{
			return (cast(%s *)(cast(byte *)(&this) + %s))[0 .. %s];
		}
	}, T.stringof, listName, T.stringof, dataOffset.stringof, 
listLength.stringof);
}

struct MyBinarySubStructAForA
{
	int someIntegerFieldA;
	float someFloatFieldA;
}

struct MyBinarySubStructBForA
{
	int someIntegerFieldB;
	float someFloatFieldB;
}

struct MyBinaryStructA
{
	int firstSublistMembersCount;
	int firstSublistMembersOffset;

	int secondSublistMembersCount;
	int secondSublistMembersOffset;
	
	 property mixin(DataList!("firstSublist", 
MyBinarySubStructAForA, firstSublistMembersCount, 
firstSublistMembersOffset));
	 property mixin(DataList!("secondSublist", 
MyBinarySubStructBForA, secondSublistMembersCount, 
secondSublistMembersOffset));
}

...

MyBinaryStructA *binaryData = cast(MyBinaryStructA *)fileData.ptr;
Sep 05 2017
next sibling parent John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 5 September 2017 at 11:08:57 UTC, Void-995 wrote:

 Hi, everyone. I'm pretty new to D and trying my first steps in 
 it. Currently I'm trying to port some code from C/C++ with 
 pretty weird data structures and don't like idea of making 
 boilerplate functions for accessing sub-lists in main binary 
 structure (lets not talk about it's design, it's from legacy 
 thing I want to deal with and I can't change the format itself 
 or writing a lot of additional parsing code). With hour I spent 
 on trying I've ended with what you may see below, but I wonder 
 if there is more pleasant variant of re-writing that template 
 as I haven't found anything better in either books or online 
 documentation yet:

 template DataList(const char[] listName, T, alias dataOffset, 
 alias listLength)
 {
 	const char[] DataList = format(q{
 		%s[] %s()
 		{
 			return (cast(%s *)(cast(byte *)(&this) + %s))[0 .. %s];
 		}
 	}, T.stringof, listName, T.stringof, dataOffset.stringof, 
 listLength.stringof);
 }

 struct MyBinarySubStructAForA
 {
 	int someIntegerFieldA;
 	float someFloatFieldA;
 }

 struct MyBinarySubStructBForA
 {
 	int someIntegerFieldB;
 	float someFloatFieldB;
 }

 struct MyBinaryStructA
 {
 	int firstSublistMembersCount;
 	int firstSublistMembersOffset;

 	int secondSublistMembersCount;
 	int secondSublistMembersOffset;
 	
 	 property mixin(DataList!("firstSublist", 
 MyBinarySubStructAForA, firstSublistMembersCount, 
 firstSublistMembersOffset));
 	 property mixin(DataList!("secondSublist", 
 MyBinarySubStructBForA, secondSublistMembersCount, 
 secondSublistMembersOffset));
 }

 ...

 MyBinaryStructA *binaryData = cast(MyBinaryStructA 
 *)fileData.ptr;


mixin template DataList(string listName, T, alias dataOffset, 
alias listLength)
{
	mixin(
		q{T[] } ~ listName ~ q{()  property
		{
			return (cast(T*)(cast(ubyte*)&this + dataOffset))[0 .. 
listLength];
		}}
	);
}

struct MyBinarySubStructAForA
{
	int someIntegerFieldA;
	float someFloatFieldA;
}

struct MyBinarySubStructBForA
{
	int someIntegerFieldB;
	float someFloatFieldB;
}

struct MyBinaryStructA
{
	int firstSublistMembersCount;
	int firstSublistMembersOffset;

	int secondSublistMembersCount;
	int secondSublistMembersOffset;
	
	mixin DataList!("firstSublist", MyBinarySubStructAForA, 
firstSublistMembersCount, firstSublistMembersOffset);
	mixin DataList!("secondSublist", MyBinarySubStructBForA, 
secondSublistMembersCount, secondSublistMembersOffset);
}
Sep 05 2017
prev sibling parent reply crimaniak <crimaniak gmail.com> writes:
On Tuesday, 5 September 2017 at 11:08:57 UTC, Void-995 wrote:

 	 property mixin(DataList!("firstSublist", 
 MyBinarySubStructAForA, firstSublistMembersCount, 
 firstSublistMembersOffset));


  I don't think string mixins are required here. It seems just 
template is more simple.

T[] getBytesAs(T, alias length, alias offset)()
{
     return (cast(T *)(cast(byte *)(&this) + offset))[0 .. length];
}

struct MyBinaryStructA
{
  ...
     alias firstList = getBytesAs!(MyBinarySubStructAForA, 
firstSublistMembersCount, firstSublistMembersOffset);
     alias secondList = getBytesAs!(MyBinarySubStructBForA, 
secondSublistMembersCount, secondSublistMembersOffset);
}

unittest
{
...
     MyBinaryStructA *binaryData = cast(MyBinaryStructA 
*)fileData.ptr;

     auto a = binaryData.firstList;
}
Sep 05 2017
next sibling parent reply Computermatronic <computermatronic gmail.com> writes:
On Tuesday, 5 September 2017 at 12:20:14 UTC, crimaniak wrote:

 On Tuesday, 5 September 2017 at 11:08:57 UTC, Void-995 wrote:

 	 property mixin(DataList!("firstSublist", 
 MyBinarySubStructAForA, firstSublistMembersCount, 
 firstSublistMembersOffset));


  I don't think string mixins are required here. It seems just 
 template is more simple.

 T[] getBytesAs(T, alias length, alias offset)()
 {
     return (cast(T *)(cast(byte *)(&this) + offset))[0 .. 
 length];
 }

 struct MyBinaryStructA
 {
  ...
     alias firstList = getBytesAs!(MyBinarySubStructAForA, 
 firstSublistMembersCount, firstSublistMembersOffset);
     alias secondList = getBytesAs!(MyBinarySubStructBForA, 
 secondSublistMembersCount, secondSublistMembersOffset);
 }

 unittest
 {
 ...
     MyBinaryStructA *binaryData = cast(MyBinaryStructA 
 *)fileData.ptr;

     auto a = binaryData.firstList;
 }


I find it very strange that this works, as a non-mixin template 
should not be able to capture the context of where it was 
instantiated. If you take the alias template parameters out it 
behaves how it should (that is an error message saying this is 
not accessible).
Sep 05 2017
next sibling parent crimaniak <crimaniak gmail.com> writes:
On Tuesday, 5 September 2017 at 12:41:45 UTC, Computermatronic 
wrote:

 I find it very strange that this works, ...


I'm too. :) In any case, it's not a big deal. It's possible to 
modify this template to transfer `this` as the first parameter 
and modify its usage accordingly.
Sep 05 2017
prev sibling next sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 5 September 2017 at 12:41:45 UTC, Computermatronic 
wrote:

 On Tuesday, 5 September 2017 at 12:20:14 UTC, crimaniak wrote:

 [...]


 I find it very strange that this works, as a non-mixin template 
 should not be able to capture the context of where it was 
 instantiated. If you take the alias template parameters out it 
 behaves how it should (that is an error message saying this is 
 not accessible).


https://issues.dlang.org/show_bug.cgi?id=17809
Sep 05 2017
parent crimaniak <crimaniak gmail.com> writes:
On Tuesday, 5 September 2017 at 14:43:19 UTC, John Colvin wrote:

 On Tuesday, 5 September 2017 at 12:41:45 UTC, Computermatronic 
 wrote:

 On Tuesday, 5 September 2017 at 12:20:14 UTC, crimaniak wrote:

 [...]


 I find it very strange that this works, as a non-mixin 
 template should not be able to capture the context of where it 
 was instantiated. If you take the alias template parameters 
 out it behaves how it should (that is an error message saying 
 this is not accessible).


 https://issues.dlang.org/show_bug.cgi?id=17809


What kind of harm can this be? Can this be considered a feature?
Sep 05 2017
prev sibling parent reply Corey Lubin <i-fear-the-spam coreylub.in> writes:
On Tuesday, 5 September 2017 at 12:41:45 UTC, Computermatronic 
wrote:

 I find it very strange that this works, as a non-mixin template 
 should not be able to capture the context of where it was 
 instantiated. If you take the alias template parameters out it 
 behaves how it should (that is an error message saying this is 
 not accessible).


Computermatronic (and crimaniak and John Colvin):

The behavior is as should be expected and is part of the 
definition of how templates' alias parameters are supposed to 
work.

The error that occurs when you stop using an alias parameter 
occurs because you're no longer forcing the capture of an 
enclosing scope by passing in an alias parameter that requires 
such a scope; not because the error should have occurred the 
first time as well.

Circumstances where some "context" is captured include:

  - template instantiations where an alias parameter requires a 
context in order to resolve the parameter correctly [[the 
instantiated function/struct/class will effectively become nested 
within whichever scope is determined to be needed in order to 
resolve the alias parameter -- whether the alias parameter is 
needed, used, or not]]

  - template instantiation of functions/structs/classes whose 
enclosing templates were declared within a function/struct/class 
[[the template instance's functions, structs, and classes will 
effectively become nested within the function/struct/class that 
the template was declared within -- whether the instantiated 
function/struct/class's original declaration strictly implied a 
need for an outer function/struct/class or not]]

  - and --hrmm... it's 6:20 AM; I haven't yet slept for the night; 
and I'm supposed to continue preparing to save myself from the 
strongest hurricane ever seen in the Atlantic Ocean in a couple 
hours, so I think that's where I'll end this partial list.

Some assorted details related to some miscellaneous capture 
situations:
  - https://dlang.org/spec/template.html#nested-templates
  - https://dlang.org/spec/template.html#nested_template_limitation
  - https://dlang.org/spec/template.html#aliasparameters
  - https://dlang.org/spec/template.html#limitations
  - https://dlang.org/spec/function.html#closures
Sep 08 2017
parent Corey Lubin <i-fear-the-spam coreylub.in> writes:
On Friday, 8 September 2017 at 10:27:57 UTC, Corey Lubin wrote:

 On Tuesday, 5 September 2017 at 12:41:45 UTC, Computermatronic 
 wrote:

 I find it very strange that this works, as a non-mixin 
 template should not be able to capture the context of where it 
 was instantiated. If you take the alias template parameters 
 out it behaves how it should (that is an error message saying 
 this is not accessible).


 Computermatronic (and crimaniak and John Colvin):

 The behavior is as should be expected and is part of the 
 definition of how templates' alias parameters are supposed to 
 work.

 The error that occurs when you stop using an alias parameter 
 occurs because you're no longer forcing the capture of an 
 enclosing scope by passing in an alias parameter that requires 
 such a scope; not because the error should have occurred the 
 first time as well.


I just actually payed attention to what the example code does 
just now. Looking at the specific example, I suppose I can see 
what the "strangeness" that's being referred to is. I suppose 
that even though the context is indeed captured in whole:

  1) the fact that the template's original declaration presents 
like an ordinary freestanding function makes it look weird (like 
a mixin template function might look weird referring to a context 
it's not in yet, except more weird because the hinting mixin 
keyword indicating it could be mixed in anywhere isn't there)

  2) the expectation is that the sneakily stolen context will/can 
only be used for and via the alias parameters it was created 
captured for, and that the allowing of explicit `this` usage 
within the definition is too much of a compiler implementation 
leak?


(though it makes it easier for implementations)

Gotta run.
Sep 08 2017
prev sibling parent reply Void-995 <void995 gmail.com> writes:
On Tuesday, 5 September 2017 at 12:20:14 UTC, crimaniak wrote:

 On Tuesday, 5 September 2017 at 11:08:57 UTC, Void-995 wrote:

 	 property mixin(DataList!("firstSublist", 
 MyBinarySubStructAForA, firstSublistMembersCount, 
 firstSublistMembersOffset));


  I don't think string mixins are required here. It seems just 
 template is more simple.

 T[] getBytesAs(T, alias length, alias offset)()
 {
     return (cast(T *)(cast(byte *)(&this) + offset))[0 .. 
 length];
 }

 struct MyBinaryStructA
 {
  ...
     alias firstList = getBytesAs!(MyBinarySubStructAForA, 
 firstSublistMembersCount, firstSublistMembersOffset);
     alias secondList = getBytesAs!(MyBinarySubStructBForA, 
 secondSublistMembersCount, secondSublistMembersOffset);
 }

 unittest
 {
 ...
     MyBinaryStructA *binaryData = cast(MyBinaryStructA 
 *)fileData.ptr;

     auto a = binaryData.firstList;
 }


Thanks, that definitely working and doesn't require mixin with 
strings. But while waiting for response I've tried another thing, 
and I doubt I would able do to that without string now:

template MyBinaryStructGenericDataList(string listName, alias 
Type)
{
	import std.string;

	const char[] MyBinaryStructGenericDataList = format(q{
		int %sCount;
		int %sOffset;

		 property %s[] %s() const
		{
			return (cast(%s *)(cast(ubyte *)(&this) + %sOffset))[0 .. 
%sCount];
		}
	}, listName, listName, Type.stringof, listName, Type.stringof, 
listName, listName);
}

struct MyBinarySubStructAForA
{
	int someIntegerFieldA;
	float someFloatFieldA;
}

struct MyBinarySubStructBForA
{
	int someIntegerFieldB;
	float someFloatFieldB;
}

struct MyBinaryStructA
{
	mixin(MyBinaryStructGenericDataList!("firstSublist", 
MyBinarySubStructAForA));
	mixin(MyBinaryStructGenericDataList!("secondSublist", 
MyBinarySubStructBForA));
}

...

MyBinaryStructA *binaryData = cast(MyBinaryStructA *)fileData.ptr;

...
Sep 05 2017
next sibling parent Computermatronic <computermatronic gmail.com> writes:
On Tuesday, 5 September 2017 at 12:54:20 UTC, Void-995 wrote:

 On Tuesday, 5 September 2017 at 12:20:14 UTC, crimaniak wrote:

 On Tuesday, 5 September 2017 at 11:08:57 UTC, Void-995 wrote:

 	 property mixin(DataList!("firstSublist", 
 MyBinarySubStructAForA, firstSublistMembersCount, 
 firstSublistMembersOffset));


  I don't think string mixins are required here. It seems just 
 template is more simple.

 T[] getBytesAs(T, alias length, alias offset)()
 {
     return (cast(T *)(cast(byte *)(&this) + offset))[0 .. 
 length];
 }

 struct MyBinaryStructA
 {
  ...
     alias firstList = getBytesAs!(MyBinarySubStructAForA, 
 firstSublistMembersCount, firstSublistMembersOffset);
     alias secondList = getBytesAs!(MyBinarySubStructBForA, 
 secondSublistMembersCount, secondSublistMembersOffset);
 }

 unittest
 {
 ...
     MyBinaryStructA *binaryData = cast(MyBinaryStructA 
 *)fileData.ptr;

     auto a = binaryData.firstList;
 }


 Thanks, that definitely working and doesn't require mixin with 
 strings. But while waiting for response I've tried another 
 thing, and I doubt I would able do to that without string now:

 template MyBinaryStructGenericDataList(string listName, alias 
 Type)
 {
 	import std.string;

 	const char[] MyBinaryStructGenericDataList = format(q{
 		int %sCount;
 		int %sOffset;

 		 property %s[] %s() const
 		{
 			return (cast(%s *)(cast(ubyte *)(&this) + %sOffset))[0 .. 
 %sCount];
 		}
 	}, listName, listName, Type.stringof, listName, Type.stringof, 
 listName, listName);
 }

 struct MyBinarySubStructAForA
 {
 	int someIntegerFieldA;
 	float someFloatFieldA;
 }

 struct MyBinarySubStructBForA
 {
 	int someIntegerFieldB;
 	float someFloatFieldB;
 }

 struct MyBinaryStructA
 {
 	mixin(MyBinaryStructGenericDataList!("firstSublist", 
 MyBinarySubStructAForA));
 	mixin(MyBinaryStructGenericDataList!("secondSublist", 
 MyBinarySubStructBForA));
 }

 ...

 MyBinaryStructA *binaryData = cast(MyBinaryStructA 
 *)fileData.ptr;

 ...


Yes, there is definately no other way but to use string mixins 
here, however if you switch to a mixin template, and put the 
string mixin inside of that, you don't have to do text 
replacement to use the template parameters.

Example:
mixin template GenericDataList(string name, T)
{
     mixin(`
         int `~name~`SublistCount;
         int `~name~`SublistOffset;

          property auto `~name~`Sublist()
         {
             return (cast(T*)(cast(byte*)&this + 
`~name~`SublistOffset))[0..`~name~`SublistCount];
         }`);
}

struct MyBinarySubStructAForA
{
     int someIntegerFieldA;
     float someFloatFieldA;
}

struct MyBinarySubStructBForA
{
     int someIntegerFieldB;
     float someFloatFieldB;
}

struct MyBinaryStructA
{
     mixin MyBinaryStructGenericDataList!("first", 
MyBinarySubStructAForA);
     mixin MyBinaryStructGenericDataList!("second", 
MyBinarySubStructBForA);
}
Sep 05 2017
prev sibling parent reply crimaniak <crimaniak gmail.com> writes:
On Tuesday, 5 September 2017 at 12:54:20 UTC, Void-995 wrote:

 Thanks, that definitely working and doesn't require mixin with 
 strings. But while waiting for response I've tried another 
 thing, and I doubt I would able do to that without string now:
...
 		int %sCount;
 		int %sOffset;


Yes, you can make such custom names only with string mixin, as I 
know. Computermatronic gives a good example how to do it simple 
way. I can give two additional advises: as I understand offsets 
as sizes are fixed so think about using unions to access sub 
structures instead of these auto-generated methods; I have a long 
history of using tricks like this starting from CHAIN MERGE usage 
in GW-BASIC to add some functions and for now such things are 
definitely code smell for me. There is always a way to do the 
same, but better.
Sep 05 2017
parent reply Void-995 <void995 gmail.com> writes:
On Tuesday, 5 September 2017 at 14:01:02 UTC, crimaniak wrote:

 On Tuesday, 5 September 2017 at 12:54:20 UTC, Void-995 wrote:

 Thanks, that definitely working and doesn't require mixin with 
 strings. But while waiting for response I've tried another 
 thing, and I doubt I would able do to that without string now:
...
 		int %sCount;
 		int %sOffset;


 Yes, you can make such custom names only with string mixin, as 
 I know. Computermatronic gives a good example how to do it 
 simple way. I can give two additional advises: as I understand 
 offsets as sizes are fixed so think about using unions to 
 access sub structures instead of these auto-generated methods; 
 I have a long history of using tricks like this starting from 
 CHAIN MERGE usage in GW-BASIC to add some functions and for now 
 such things are definitely code smell for me. There is always a 
 way to do the same, but better.


Using unions? <Type>Count and <Type>Offset are different 
depending on input data, so the address where they are is varying 
depending on which file I've loaded. Or I didn't get what you 
meant.
Sep 05 2017
parent reply crimaniak <crimaniak gmail.com> writes:
On Tuesday, 5 September 2017 at 14:55:21 UTC, Void-995 wrote:


 Using unions? <Type>Count and <Type>Offset are different 
 depending on input data, so the address where they are is 
 varying depending on which file I've loaded. Or I didn't get 
 what you meant.


  Yes, so you need separate union type for every type of input 
data. But these values are known in compile time, so you can do 
mixin, creating the union for each type like this:

union MyBinaryStructA
{
     ubyte[...] asBytes;
     struct asStruct
     {
         align(1):
         ubyte[...] dummy1; // padding before firstList
         MyBinarySubStructAForA[...] firstList;
         ubyte[...] dummy2; // padding between lists
         MyBinarySubStructBForA[...] secondList;
         ...
     }
}

unittest
{
     MyBinaryStructA a;

     a.asBytes = someBinarySourceForA; // fill from unstructured 
source

     auto item = a.asStruct.firstList[1]; // access to structure 
element
}
Sep 05 2017
parent reply Void-995 <void995 gmail.com> writes:
On Tuesday, 5 September 2017 at 21:18:11 UTC, crimaniak wrote:

 On Tuesday, 5 September 2017 at 14:55:21 UTC, Void-995 wrote:


 Using unions? <Type>Count and <Type>Offset are different 
 depending on input data, so the address where they are is 
 varying depending on which file I've loaded. Or I didn't get 
 what you meant.


  Yes, so you need separate union type for every type of input 
 data. But these values are known in compile time, so you can do 
 mixin, creating the union for each type like this:

 union MyBinaryStructA
 {
     ubyte[...] asBytes;
     struct asStruct
     {
         align(1):
         ubyte[...] dummy1; // padding before firstList
         MyBinarySubStructAForA[...] firstList;
         ubyte[...] dummy2; // padding between lists
         MyBinarySubStructBForA[...] secondList;
         ...
     }
 }

 unittest
 {
     MyBinaryStructA a;

     a.asBytes = someBinarySourceForA; // fill from unstructured 
 source

     auto item = a.asStruct.firstList[1]; // access to structure 
 element
 }


The whole thing is that I don't know either padding nor elements 
count. Those values are read from file. First you need to read 
whole file data and cast those bytes as pointer to file header 
structure. Offset of each list is bigger then size of the parent 
structure. Length of each list vary from 0 to whatever.
Sep 05 2017
parent crimaniak <crimaniak gmail.com> writes:
On Tuesday, 5 September 2017 at 22:51:45 UTC, Void-995 wrote:

 The whole thing is that I don't know either padding nor 
 elements count. Those values are read from file. ...


Sorry, I was not attentive enough. Yes, you can't use unions in 
this case.
Sep 06 2017