www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How do I use CTFE to generate an immutable associative array at

reply Chad Joan <chadjoan gmail.com> writes:
Hello all,

I'm trying to make this work:

---
pure string[string] parseTwoColumnCsv(string inputCsv)
{
	import std.csv;
	import std.typecons;
	
	string[string] result;
	
	foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) )
		result[record[0]] = record[1];
	
	return result;
}

immutable string[string] dataLookup = 
parseTwoColumnCsv(import("some_data.csv"));

void main()
{
	import std.stdio;
	writefln("dataLookup = %s", dataLookup);
}
---

But (with DMD 2.073.1) I am getting this error:
main.d(14): Error: non-constant expression [['a', 'b', 'c']:['x', 
'y', 'z'], ['1', '2', '3']:['4', '5', '6']]


The case with normal (non-associative) arrays seems to work fine, 
but is not what I needed:

---
pure string[][] parseTwoColumnCsv(string inputCsv)
{
	import std.csv;
	import std.typecons;
	
	string[][] result;
	
	foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) )
		result ~= [record[0],record[1]];
	
	return result;
}

immutable string[][] dataLookup = 
parseTwoColumnCsv(import("some_data.csv"));

void main()
{
	import std.stdio;
	writefln("dataLookup = %s", dataLookup);
}
---

Any idea how I can get this working?

I have tried a couple other things, like having the 
parseTwoColumnCsv function return an immutable string[string] and 
casting 'result' to that in the return statement, and I also 
tried just casting the value returned from parseTwoColumnCsv as 
it appears in the declaration of 'dataLookup'.  I tried 
std.exception's assumeUnique, but the function/template signature 
doesn't seem to support associative arrays.

Thanks.
Feb 21 2017
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 21 February 2017 at 21:53:23 UTC, Chad Joan wrote:
 Hello all,

 I'm trying to make this work:

 [...]
You cannot create AA's at ctfe and carry them over to runtime use. You'd have to use a costum dictionary-type. I think the vibe.d project has one you could use.
Feb 21 2017
parent reply Chad Joan <chadjoan gmail.com> writes:
On Tuesday, 21 February 2017 at 21:58:07 UTC, Stefan Koch wrote:
 On Tuesday, 21 February 2017 at 21:53:23 UTC, Chad Joan wrote:
 Hello all,

 I'm trying to make this work:

 [...]
You cannot create AA's at ctfe and carry them over to runtime use. You'd have to use a costum dictionary-type. I think the vibe.d project has one you could use.
I wondered if this might be the case. Well, I won't worry about it then. I'll have to find another way around. Thanks a bunch!
Feb 21 2017
parent reply Chad Joan <chadjoan gmail.com> writes:
On Tuesday, 21 February 2017 at 22:26:01 UTC, Chad Joan wrote:
 On Tuesday, 21 February 2017 at 21:58:07 UTC, Stefan Koch wrote:
 On Tuesday, 21 February 2017 at 21:53:23 UTC, Chad Joan wrote:
 Hello all,

 I'm trying to make this work:

 [...]
You cannot create AA's at ctfe and carry them over to runtime use. You'd have to use a costum dictionary-type. I think the vibe.d project has one you could use.
I wondered if this might be the case. Well, I won't worry about it then. I'll have to find another way around. Thanks a bunch!
For the sake of the rest of the internet, here is what I'm probably going to stick with: --- pure string[string] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[string] result; foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) ) result[record[0]] = record[1]; return result; } immutable string[string] dataLookup; static this() { dataLookup = parseTwoColumnCsv(import("some_data.csv")); } void main() { import std.stdio; writefln("dataLookup = %s", dataLookup); } --- In this case the AA isn't actually coded into the executable; but at least the configuration from some_data.csv will be in the executable as a string. The program will construct the AA at startup. It's not as "cool", but it should get the job done. HTH.
Feb 21 2017
next sibling parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Tue, Feb 21, 2017 at 10:34:57PM +0000, Chad Joan via Digitalmars-d-learn
wrote:
 On Tuesday, 21 February 2017 at 22:26:01 UTC, Chad Joan wrote:
 On Tuesday, 21 February 2017 at 21:58:07 UTC, Stefan Koch wrote:
 On Tuesday, 21 February 2017 at 21:53:23 UTC, Chad Joan wrote:
 Hello all,
 
 I'm trying to make this work:
 
 [...]
You cannot create AA's at ctfe and carry them over to runtime use. You'd have to use a costum dictionary-type. I think the vibe.d project has one you could use.
I wondered if this might be the case. Well, I won't worry about it then. I'll have to find another way around. Thanks a bunch!
For the sake of the rest of the internet, here is what I'm probably going to stick with: --- pure string[string] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[string] result; foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) ) result[record[0]] = record[1]; return result; } immutable string[string] dataLookup; static this() { dataLookup = parseTwoColumnCsv(import("some_data.csv")); } void main() { import std.stdio; writefln("dataLookup = %s", dataLookup); } --- In this case the AA isn't actually coded into the executable; but at least the configuration from some_data.csv will be in the executable as a string. The program will construct the AA at startup. It's not as "cool", but it should get the job done.
[...] Parsing strings at program startup is ugly and slow. What about parsing at compile-time with CTFE into an array literal and transforming that into an AA at startup, which should be a lot faster? // Warning: untested code pure string[2][] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[2][] result; foreach (record; csvReader!(Tuple!(string,string))(inputCsv)) { result ~= [record[0], record[1]]; } return result; } immutable string[string] dataLookup; static this() { enum halfCookedData = parseTwoColumnCsv(import("some_data.csv")); foreach (p; halfCookedData) { dataLookup[p[0]] = p[1]; } } T -- Those who've learned LaTeX swear by it. Those who are learning LaTeX swear at it. -- Pete Bleackley
Feb 21 2017
parent reply Chad Joan <chadjoan gmail.com> writes:
On Tuesday, 21 February 2017 at 22:43:15 UTC, H. S. Teoh wrote:
 Parsing strings at program startup is ugly and slow.  What 
 about parsing at compile-time with CTFE into an array literal 
 and transforming that into an AA at startup, which should be a 
 lot faster?

 	// Warning: untested code
 	pure string[2][] parseTwoColumnCsv(string inputCsv)
 	{
 		import std.csv;
 		import std.typecons;

 		string[2][] result;
 		foreach (record; csvReader!(Tuple!(string,string))(inputCsv)) 
 {
 			result ~= [record[0], record[1]];
 		}
 		return result;
 	}

 	immutable string[string] dataLookup;
 	static this()
 	{
 		enum halfCookedData = 
 parseTwoColumnCsv(import("some_data.csv"));
 		foreach (p; halfCookedData) {
 			dataLookup[p[0]] = p[1];
 		}
 	}


 T
Hi, That makes a lot of sense and it had crossed my mind. In my case the data sets are super small, so I'm probably going to be lazy/productive and leave it the way it is. Anyone else from the internet copying these examples: try to just do it H.S. Teoh's way from the start ;) Thanks for the suggestion. - Chad
Feb 21 2017
parent reply Chad Joan <chadjoan gmail.com> writes:
On Tuesday, 21 February 2017 at 23:30:52 UTC, Chad Joan wrote:
 On Tuesday, 21 February 2017 at 22:43:15 UTC, H. S. Teoh wrote:
 Parsing strings at program startup is ugly and slow.  What 
 about parsing at compile-time with CTFE into an array literal 
 and transforming that into an AA at startup, which should be a 
 lot faster?

 	// Warning: untested code
 	pure string[2][] parseTwoColumnCsv(string inputCsv)
 	{
 		import std.csv;
 		import std.typecons;

 		string[2][] result;
 		foreach (record; 
 csvReader!(Tuple!(string,string))(inputCsv)) {
 			result ~= [record[0], record[1]];
 		}
 		return result;
 	}

 	immutable string[string] dataLookup;
 	static this()
 	{
 		enum halfCookedData = 
 parseTwoColumnCsv(import("some_data.csv"));
 		foreach (p; halfCookedData) {
 			dataLookup[p[0]] = p[1];
 		}
 	}


 T
Hi, That makes a lot of sense and it had crossed my mind. In my case the data sets are super small, so I'm probably going to be lazy/productive and leave it the way it is. Anyone else from the internet copying these examples: try to just do it H.S. Teoh's way from the start ;) Thanks for the suggestion. - Chad
OK I couldn't help it: --- pure private string[][] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[][] result; foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) ) result ~= [record[0],record[1]]; return result; } pure private string[string] twoColumnArrayToAA(const string[][] arr) { string[string] result; foreach ( pair; arr ) result[pair[0]] = pair[1]; return result; } pure private string[string] importTwoColumnCsv(string csvFile)() { // Force the parse to happen at compile time. immutable string[][] tempArray = import(csvFile).parseTwoColumnCsv(); // Convert the parsed array into a runtime Associative Array and return it. return tempArray.twoColumnArrayToAA(); } immutable string[string] dataLookup; immutable string[string] dataLookup1; immutable string[string] dataLookup2; static this() { import std.range; dataLookup1 = importTwoColumnCsv!"some_data1.csv"; dataLookup2 = importTwoColumnCsv!"some_data2.csv"; foreach( pair; chain(dataLookup1.byKeyValue, dataLookup2.byKeyValue) ) dataLookup[pair.key] = pair.value; } void main() { import std.stdio; writefln("dataLookup = %s", dataLookup); } --- This example also shows joining associative arrays.
Feb 21 2017
parent "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Wed, Feb 22, 2017 at 12:38:47AM +0000, Chad Joan via Digitalmars-d-learn
wrote:
[...]

Hmm. It's actually not necessary to manually write a foreach loop to
convert an array to an AA. We could, instead, change parseTwoColumnCsv()
to return an array of std.typecons.Tuple instead (which, incidentally,
means you can elide that foreach loop too!), then there's this handy
function std.array.assocArray that will do the transcription for us,
as follows.

(In fact, you can merge AA's without ever needing to write `foreach` at
all! See below.)

---
pure private auto parseTwoColumnCsv(string inputCsv)
{
	import std.csv;
	import std.typecons;
	return csvReader!(Tuple!(string,string))(inputCsv);
}

immutable string[string] dataLookup;
immutable string[string] dataLookup1;
immutable string[string] dataLookup2;

static this()
{
	import std.array : assocArray;
	import std.range : chain;

	// Force CTFE
	immutable tuples1 = parseTwoColumnCsv("some_data1.csv");
	immutable tuples2 = parseTwoColumnCsv("some_data2.csv");

	dataLookup1 = tuples1.assocArray;	// Bam! :-P
	dataLookup2 = tuples2.assocArray;	// Bam! :-P

	// Bam, bam! - merge AA's in a single step!
	dataLookup3 = chain(tuples1, tuples2).assocArray;
}
---


T

-- 
You have to expect the unexpected. -- RL
Feb 21 2017
prev sibling parent Era Scarecrow <rtcvb32 yahoo.com> writes:
On Tuesday, 21 February 2017 at 22:34:57 UTC, Chad Joan wrote:
 In this case the AA isn't actually coded into the executable; 
 but at least the configuration from some_data.csv will be in 
 the executable as a string.  The program will construct the AA 
 at startup.  It's not as "cool", but it should get the job done.
I have a partial static AA implementation that seems like it works, I mentioned this in a different thread. https://github.com/rtcvb32/Side-Projects/blob/master/staticaa.d Try it out, etc. Usage: Create your AA as an enum (for simplicity) StaticAA!(KeyType, ValueType, getAALen(EnumAssosiativeArray), EnumAssosiativeArray.length)(EnumAssosiativeArray); Afterwards use it as you normally would for the same thing. Unittest example: enum AA = ["one":1, "two":2, "three":3, "four":4, "five":5, "six":6, "seven":7, "eight":8, "nine":9, "zero":0]; auto SAA = StaticAA!(string, int, getAALen(AA), AA.length)(AA); //just verifies the keys/values match. foreach(k, v; AA) { assert(SAA[k] == v); } Note: getAALen basically tests the array expanding it out until none of the hashes overlap or causes problems. Trying to compile these into a single call I've had issues, so if anyone has a better solution I'd go for it.
Feb 21 2017
prev sibling next sibling parent Daniel Kozak via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
I have similar issue and I beleive I was able to workaround that 
somehow, but it is so many years now :(. Have you tried enum dataLookup 
instead of immutable string[string] dataLookup


Dne 21.2.2017 v 22:53 Chad Joan via Digitalmars-d-learn napsal(a):
 Hello all,

 I'm trying to make this work:

 ---
 pure string[string] parseTwoColumnCsv(string inputCsv)
 {
     import std.csv;
     import std.typecons;

     string[string] result;

     foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) )
         result[record[0]] = record[1];

     return result;
 }

 immutable string[string] dataLookup = 
 parseTwoColumnCsv(import("some_data.csv"));

 void main()
 {
     import std.stdio;
     writefln("dataLookup = %s", dataLookup);
 }
 ---

 But (with DMD 2.073.1) I am getting this error:
 main.d(14): Error: non-constant expression [['a', 'b', 'c']:['x', 'y', 
 'z'], ['1', '2', '3']:['4', '5', '6']]


 The case with normal (non-associative) arrays seems to work fine, but 
 is not what I needed:

 ---
 pure string[][] parseTwoColumnCsv(string inputCsv)
 {
     import std.csv;
     import std.typecons;

     string[][] result;

     foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) )
         result ~= [record[0],record[1]];

     return result;
 }

 immutable string[][] dataLookup = 
 parseTwoColumnCsv(import("some_data.csv"));

 void main()
 {
     import std.stdio;
     writefln("dataLookup = %s", dataLookup);
 }
 ---

 Any idea how I can get this working?

 I have tried a couple other things, like having the parseTwoColumnCsv 
 function return an immutable string[string] and casting 'result' to 
 that in the return statement, and I also tried just casting the value 
 returned from parseTwoColumnCsv as it appears in the declaration of 
 'dataLookup'.  I tried std.exception's assumeUnique, but the 
 function/template signature doesn't seem to support associative arrays.

 Thanks.
Feb 21 2017
prev sibling parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Tue, Feb 21, 2017 at 11:50:02PM +0100, Daniel Kozak via Digitalmars-d-learn
wrote:
 I have similar issue and I beleive I was able to workaround that
 somehow, but it is so many years now :(. Have you tried enum
 dataLookup instead of immutable string[string] dataLookup
That may appear to work, but I would *strongly* recommend against it, because what happens when you use enum with an AA, is that the AA will be created *at runtime*, *every single time* it is referenced. (It is as if you copy-n-pasted the entire AA into the code each time you reference the enum.) Which will introduce ridiculous amounts of redundant work at runtime and cause a big performance penalty. T -- What did the alien say to Schubert? "Take me to your lieder."
Feb 21 2017
parent reply Jacob Carlborg <doob me.com> writes:
On 2017-02-21 23:49, H. S. Teoh via Digitalmars-d-learn wrote:

 That may appear to work, but I would *strongly* recommend against it,
 because what happens when you use enum with an AA, is that the AA will
 be created *at runtime*, *every single time* it is referenced.  (It is
 as if you copy-n-pasted the entire AA into the code each time you
 reference the enum.)  Which will introduce ridiculous amounts of
 redundant work at runtime and cause a big performance penalty.
You can use an enum to declare the AA and then assign it to an immutable variable using "static this". The you would only use to the immutable variable and never the enum. enum aa = [1 : 2]; immutable int[int] iaa; static this() { iaa = aa; } -- /Jacob Carlborg
Feb 22 2017
next sibling parent Daniel Kozak via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
Yes this is how I mean it.

Dne 22. 2. 2017 9:05 napsal u=C5=BEivatel "Jacob Carlborg via
Digitalmars-d-learn" <digitalmars-d-learn puremagic.com>:

 On 2017-02-21 23:49, H. S. Teoh via Digitalmars-d-learn wrote:

 That may appear to work, but I would *strongly* recommend against it,
 because what happens when you use enum with an AA, is that the AA will
 be created *at runtime*, *every single time* it is referenced.  (It is
 as if you copy-n-pasted the entire AA into the code each time you
 reference the enum.)  Which will introduce ridiculous amounts of
 redundant work at runtime and cause a big performance penalty.
You can use an enum to declare the AA and then assign it to an immutable variable using "static this". The you would only use to the immutable variable and never the enum. enum aa =3D [1 : 2]; immutable int[int] iaa; static this() { iaa =3D aa; } -- /Jacob Carlborg
Feb 22 2017
prev sibling parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Wed, Feb 22, 2017 at 09:00:36AM +0100, Jacob Carlborg via
Digitalmars-d-learn wrote:
[...]
 You can use an enum to declare the AA and then assign it to an
 immutable variable using "static this". The you would only use to the
 immutable variable and never the enum.
 
 enum aa = [1 : 2];
 
 immutable int[int] iaa;
 
 static this()
 {
     iaa = aa;
 }
[...] Wow, this is miles better than the hacks that I've come up with! Also makes the no-static-AA issue much less of a problem that I thought it was. (In fact, now I'm wondering if we could just hack dmd to emit the equivalent of this code as a lowering, whenever the user tries to declare a compile-time initialized AA.) T -- Computers are like a jungle: they have monitor lizards, rams, mice, c-moss, binary trees... and bugs.
Feb 22 2017
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 22 February 2017 at 15:27:22 UTC, H. S. Teoh wrote:
  (In fact, now I'm wondering if we could just
 hack dmd to emit the equivalent of this code as a lowering, 
 whenever the user tries to declare a compile-time initialized 
 AA.)
All the problems disappear if the AA's are compiler visible and CTFEable code. There is as far as I know a project by Martin Nowak that does that. And uses templates for AA's rather then whacky TypeInfos. So in the future this problem will disappear.
Feb 22 2017
parent reply "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Wed, Feb 22, 2017 at 04:08:45PM +0000, Stefan Koch via Digitalmars-d-learn
wrote:
 On Wednesday, 22 February 2017 at 15:27:22 UTC, H. S. Teoh wrote:
 (In fact, now I'm wondering if we could just hack dmd to emit the
 equivalent of this code as a lowering, whenever the user tries to
 declare a compile-time initialized AA.)
All the problems disappear if the AA's are compiler visible and CTFEable code.
I'm not sure it's that simple. Just because AA's become CTFEable doesn't mean they will automatically be convertible to object code in the executable that allows runtime AA lookups. For instance, a CTFE pointer will have a value that has no correspondence with the object code in the executable, and neither will memory allocated during CTFE have any correspondence with the emitted object code (because this memory is allocated in the compiler, not in the object code). So if the CTFE AA implementation allocates nodes for storing key/value pairs, they will only exist in the CTFE engine, and pointers to them will only make sense within the CTFE engine. In order to make them work in the executable (so that you can do AA lookups to these computed nodes at runtime), you will need to somehow map them to object code. Just because AA's become CTFEable will not automatically solve this for you.
 There is as far as I know a project by Martin Nowak that does that.
 And uses templates for AA's rather then whacky TypeInfos.
 
 So in the future this problem will disappear.
Oh I know, the problem is that this "future" has been taking a long time arriving. I was involved in an early (but unfortunately unsuccessful) effort to rewrite AA's as library code. But there was simply too much compiler magic involved with AA's that it couldn't work at the time. Much of this magic has been dispelled over the past few years, though, so we should be in better shape now for moving AA's completely into the library. I'm very much looking forward to Martin's implementation when it's ready. But in the meantime, lowering compile-time initialized AA's could have the above hack as a temporary workaround until we can get AA literals to be embeddable in object code. T -- GEEK = Gatherer of Extremely Enlightening Knowledge
Feb 22 2017
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 22 February 2017 at 17:05:17 UTC, H. S. Teoh wrote:
 On Wed, Feb 22, 2017 at 04:08:45PM +0000, Stefan Koch via 
 Digitalmars-d-learn wrote:
 [...]
I'm not sure it's that simple. Just because AA's become CTFEable doesn't mean they will automatically be convertible to object code in the executable that allows runtime AA lookups. For instance, a CTFE pointer will have a value that has no correspondence with the object code in the executable, and neither will memory allocated during CTFE have any correspondence with the emitted object code (because this memory is allocated in the compiler, not in the object code). So if the CTFE AA implementation allocates nodes for storing key/value pairs, they will only exist in the CTFE engine, and pointers to them will only make sense within the CTFE engine. In order to make them work in the executable (so that you can do AA lookups to these computed nodes at runtime), you will need to somehow map them to object code. Just because AA's become CTFEable will not automatically solve this for you. [...]
The CTFE allocated memory can be copied into the object file verbatim because the structure is the same. Pointers are easy to fix up as well, because by definition the have to point to static data that will be in the object file.
Feb 22 2017
parent "H. S. Teoh via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com> writes:
On Wed, Feb 22, 2017 at 06:17:14PM +0000, Stefan Koch via Digitalmars-d-learn
wrote:
 On Wednesday, 22 February 2017 at 17:05:17 UTC, H. S. Teoh wrote:
 On Wed, Feb 22, 2017 at 04:08:45PM +0000, Stefan Koch via
 Digitalmars-d-learn wrote:
 [...]
I'm not sure it's that simple. Just because AA's become CTFEable doesn't mean they will automatically be convertible to object code in the executable that allows runtime AA lookups. For instance, a CTFE pointer will have a value that has no correspondence with the object code in the executable, and neither will memory allocated during CTFE have any correspondence with the emitted object code (because this memory is allocated in the compiler, not in the object code). So if the CTFE AA implementation allocates nodes for storing key/value pairs, they will only exist in the CTFE engine, and pointers to them will only make sense within the CTFE engine. In order to make them work in the executable (so that you can do AA lookups to these computed nodes at runtime), you will need to somehow map them to object code. Just because AA's become CTFEable will not automatically solve this for you. [...]
The CTFE allocated memory can be copied into the object file verbatim because the structure is the same. Pointers are easy to fix up as well, because by definition the have to point to static data that will be in the object file.
This will fail for cross-compilation. Unless you mean that the new CTFE engine is emulating the target machine directly? T -- Computers shouldn't beep through the keyhole.
Feb 22 2017