www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - turning an array of structs into a struct of arrays

reply "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
I have an array of structs eg

struct PriceBar
{
   DateTime date;
   double open;
   double high;
   double low;
   double close;
}

(which fields are present in this particular struct will depend 
on template arguments).

what is the best way to turn these at compile time into a struct 
of arrays? eg

struct PriceBars
{
   DateTime[] dates;
   double[] opens;
   double[] highs;
   double[] lows;
   double[] closes;
}

I can use FieldTypeTuple and FieldNameTuple, but I am a bit lost 
as to how without static foreach to loop through these in order 
to generate a mixin to declare the new type.  I can turn it into 
a string, but what is the better option?

Thanks.
Jul 03 2015
next sibling parent reply "Babu" <bb.temp gmx.com> writes:
On Friday, 3 July 2015 at 10:52:03 UTC, Laeeth Isharc wrote:
 I have an array of structs eg

 struct PriceBar
 {
   DateTime date;
   double open;
   double high;
   double low;
   double close;
 }

 (which fields are present in this particular struct will depend 
 on template arguments).

 what is the best way to turn these at compile time into a 
 struct of arrays? eg

 struct PriceBars
 {
   DateTime[] dates;
   double[] opens;
   double[] highs;
   double[] lows;
   double[] closes;
 }

 I can use FieldTypeTuple and FieldNameTuple, but I am a bit 
 lost as to how without static foreach to loop through these in 
 order to generate a mixin to declare the new type.  I can turn 
 it into a string, but what is the better option?

 Thanks.
see https://github.com/economicmodeling/soa
Jul 03 2015
parent "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
 see https://github.com/economicmodeling/soa
Thanks v much for this. Figured out in the meantime one solution here (probably quite ugly): https://gist.github.com/Laeeth/9174498f9b79dc90ac18
Jul 03 2015
prev sibling next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 3 July 2015 at 10:52:03 UTC, Laeeth Isharc wrote:
 I have an array of structs eg

 struct PriceBar
 {
   DateTime date;
   double open;
   double high;
   double low;
   double close;
 }

 (which fields are present in this particular struct will depend 
 on template arguments).

 what is the best way to turn these at compile time into a 
 struct of arrays? eg

 struct PriceBars
 {
   DateTime[] dates;
   double[] opens;
   double[] highs;
   double[] lows;
   double[] closes;
 }

 I can use FieldTypeTuple and FieldNameTuple, but I am a bit 
 lost as to how without static foreach to loop through these in 
 order to generate a mixin to declare the new type.  I can turn 
 it into a string, but what is the better option?

 Thanks.
Not necessarily the best solution to the problem, but I just wanted to show this because I think it's kinda cool: https://gist.github.com/John-Colvin/103d3c064ad6cb4cf435#file-transformmembers-d
Jul 03 2015
parent "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
 Not necessarily the best solution to the problem, but I just 
 wanted to show this because I think it's kinda cool:

 https://gist.github.com/John-Colvin/103d3c064ad6cb4cf435#file-transformmembers-d
Very impressive - now I have to try to understand how it works...!
Jul 03 2015
prev sibling parent reply Artur Skawina via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On 07/03/15 12:52, Laeeth Isharc via Digitalmars-d-learn wrote:
 I have an array of structs eg
 
 struct PriceBar
 {
   DateTime date;
   double open;
   double high;
   double low;
   double close;
 }
 
 (which fields are present in this particular struct will depend on template
arguments).
 
 what is the best way to turn these at compile time into a struct of arrays? eg
 
 struct PriceBars
 {
   DateTime[] dates;
   double[] opens;
   double[] highs;
   double[] lows;
   double[] closes;
 }
 
 I can use FieldTypeTuple and FieldNameTuple, but I am a bit lost as to how
without static foreach to loop through these in order to generate a mixin to
declare the new type.  I can turn it into a string, but what is the better
option?
The simplest solution is something like: template SOA(Struct, size_t LENGTH) { struct SOA { enum MEMBERNAME(size_t N) = __traits(identifier, Struct.tupleof[N]); static __gentypes() { string ret; foreach (I, TYPE; typeof(Struct.tupleof)) ret ~= "align(16) typeof(Struct.tupleof["~I.stringof~"])["~LENGTH.stringof~"] " ~ MEMBERNAME!I ~ ";"; return ret; } mixin(__gentypes()); } } alias PriceBars = SOA!(PriceBar, 8); which you'll have to adjust to your requirements (eg drop the '"~LENGTH.stringof~"' part to get a struct-of-slices, which is what your example above shows). artur
Jul 04 2015
next sibling parent "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
 I can use FieldTypeTuple and FieldNameTuple, but I am a bit 
 lost as to how without static foreach to loop through these in 
 order to generate a mixin to declare the new type.  I can turn 
 it into a string, but what is the better option?
The simplest solution is something like: template SOA(Struct, size_t LENGTH) { struct SOA { enum MEMBERNAME(size_t N) = __traits(identifier, Struct.tupleof[N]); static __gentypes() { string ret; foreach (I, TYPE; typeof(Struct.tupleof)) ret ~= "align(16) typeof(Struct.tupleof["~I.stringof~"])["~LENGTH.stringof~"] " ~ MEMBERNAME!I ~ ";"; return ret; } mixin(__gentypes()); } } alias PriceBars = SOA!(PriceBar, 8); which you'll have to adjust to your requirements (eg drop the '"~LENGTH.stringof~"' part to get a struct-of-slices, which is what your example above shows). artur
Thanks to Artur, John Colvin, and Babu for the very helpful responses. This should be up on the wiki, and ideally someone should write a little tutorial text talking the reader through how this works and why choices were made a certain way. I will stick these up on the wiki with attribution if there are no objections. Something is better than perfect in this situation. There is a big gap between understanding what language terms do and knowing in practice how to use them for metaprogramming if you don't previously come from a C++ background. I originally learnt C with parameters in K&R style, and the world has changed quite a lot since those days - both the relative slowness of memory, and CTFE/metaprogramming are quite new concepts to me (Forth doesn't count). It's very easy to be quickly productive in D for most things (I found it easier than Python, which I learnt previously) but there is a bigger leap in learning templates+CTFE - not because it's intrinsically hard, but it's a different way of thinking that is unfamiliar. Probably I should read more source code - the harder way is often easier in the longer term. In case it's of interest my own reason for wanting to do this transformation was not cache efficiency, but just to be able to easily get the data into a form I can access from PydMagic and PyD, since for some chart applications it's easier to have the data in this form, and I would rather do all transformations in D and stick to Python only for pure charting. I appreciate the thoughtful responses. Laeeth.
Jul 04 2015
prev sibling parent reply "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
Posted short write-up here.  Please make it better...

http://wiki.dlang.org/Transforming_slice_of_structs_into_struct_of_slices
Jul 04 2015
parent reply "Vlad Levenfeld" <vlevenfeld gmail.com> writes:
On Sunday, 5 July 2015 at 00:07:59 UTC, Laeeth Isharc wrote:
 Posted short write-up here.  Please make it better...

 http://wiki.dlang.org/Transforming_slice_of_structs_into_struct_of_slices
In John Colvin's solution, should alias TransformMembers(alias TypeTransform, alias NameTransform, T) = Tuple!( RoundRobin!(Pack!(staticMap!(TypeTransform, FieldTypeTuple!PriceBar)), Pack!(staticMap!(NameTransform, FieldNameTuple!PriceBar)))); read alias TransformMembers(alias TypeTransform, alias NameTransform, T) = Tuple!( RoundRobin!(Pack!(staticMap!(TypeTransform, FieldTypeTuple!T)), Pack!(staticMap!(NameTransform, FieldNameTuple!T)))); ???
Jul 06 2015
parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 6 July 2015 at 17:35:22 UTC, Vlad Levenfeld wrote:
 On Sunday, 5 July 2015 at 00:07:59 UTC, Laeeth Isharc wrote:
 Posted short write-up here.  Please make it better...

 http://wiki.dlang.org/Transforming_slice_of_structs_into_struct_of_slices
In John Colvin's solution, should alias TransformMembers(alias TypeTransform, alias NameTransform, T) = Tuple!( RoundRobin!(Pack!(staticMap!(TypeTransform, FieldTypeTuple!PriceBar)), Pack!(staticMap!(NameTransform, FieldNameTuple!PriceBar)))); read alias TransformMembers(alias TypeTransform, alias NameTransform, T) = Tuple!( RoundRobin!(Pack!(staticMap!(TypeTransform, FieldTypeTuple!T)), Pack!(staticMap!(NameTransform, FieldNameTuple!T)))); ???
Yikes, yes, of course!
Jul 06 2015