www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - ctfe library

reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
Are there any good libraries for ctfe/code generation?

I don't know, things like parsing support for compile time strings, 
string formatting, type <-> string

My project seems to be growing ctfe, and it's all horribly hacky and 
ugly code.
Apr 19 2010
next sibling parent Ary Borenszweig <ary esperanto.org.ar> writes:
Ellery Newcomer wrote:

 Are there any good libraries for ctfe/code generation?
 
 I don't know, things like parsing support for compile time strings, 
 string formatting, type <-> string
 
 My project seems to be growing ctfe, and it's all horribly hacky and 
 ugly code.


I think all ctfe code is horrible hacky and ugly code. :-P
Apr 20 2010
prev sibling next sibling parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
Ellery Newcomer wrote:

 Are there any good libraries for ctfe/code generation?
 
 I don't know, things like parsing support for compile time strings, 
 string formatting, type <-> string
 
 My project seems to be growing ctfe, and it's all horribly hacky and 
 ugly code.


I think all ctfe code is horribly hacky and ugly code. :-P
Apr 20 2010
parent reply Don <nospam nospam.com> writes:
Ary Borenszweig wrote:

 Ellery Newcomer wrote:

 Are there any good libraries for ctfe/code generation?

 I don't know, things like parsing support for compile time strings, 
 string formatting, type <-> string

 My project seems to be growing ctfe, and it's all horribly hacky and 
 ugly code.


 
 I think all ctfe code is horribly hacky and ugly code. :-P


I think you're talking about string mixins, not CTFE. They are not the same.

I don't see anything particularly ugly or hacky about CTFE code, like 
the code below:

int greater(int a, int b) {
   if (a>b) return a;
   else return b;
}

int [ greater(short.max/3, 515) ] foo;
Apr 20 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 04/20/2010 10:17 AM, Don wrote:

 I don't see anything particularly ugly or hacky about CTFE code, like
 the code below:

 int greater(int a, int b) {
 if (a>b) return a;
 else return b;
 }

 int [ greater(short.max/3, 515) ] foo;


It isn't. Things don't get ugly until you hit the relatively low ceiling 
of what ctfe can handle. I don't pay much attention to where that 
ceiling is, but I am aware that the situation has been improving mostly 
due to your efforts, and I appreciate that.

So far, all my ctfe code is code generation for string mixins, and it's 
fairly modest stuff, to the point where a simple string formatting 
function could probably take care of a fair portion of the problem. It's 
just that I don't know of any. (I'm in D1/Tango land)

The other half of the problem is parsing input strings for simple dsls. 
Having to write out a lexer/parser from scratch just sucks regardless, 
and writing it in ctfe code, which isn't allowed composite data 
structures (last I checked) is really not worth doing. I think D fails 
on its promise here, where it says "with string mixins you can create 
your own dsls", but then provides no additional support.

As ctfe support matures, I dream of a full-fledged parser generator that 
can be evaluated at compile time, although that's way more heavy duty 
than what most people will need.

Another idea is something more on the lines of haskell's parsec, which 
provides building blocks for common tokens, etc. I don't know much about 
it, though.

Even compile time regular expressions would be better than nothing..

And whatever it is, it needs to be in the standard library.
Apr 20 2010
next sibling parent reply div0 <div0 users.sourceforge.net> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ellery Newcomer wrote:

 
 As ctfe support matures, I dream of a full-fledged parser generator that
 can be evaluated at compile time, although that's way more heavy duty
 than what most people will need.
 


My spirit port might be an option at some point soon:

http://www.sstk.co.uk/spiritd.php

Haven't tried it with current d2 compilers, but should still work w/ d1.

It uses lots of template code to compose parsers, so in theory it ought
to be usable at compile time. I doubt it can be used that way at the
moment, but hopefully dmd will get there and then you'll be able to do
some seriously funky stuff at compile time.

- --
My enormous talent is exceeded only by my outrageous laziness.
http://www.ssTk.co.uk
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFLzew3T9LetA9XoXwRAtpMAKCizK+fXjoF/d+FqAy0H2sQGpHGkACeMa9A
Buy6K2vMmvk75bNxlDsbhKE=
=gxsa
-----END PGP SIGNATURE-----
Apr 20 2010
parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 04/20/2010 01:02 PM, div0 wrote:

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1

 Ellery Newcomer wrote:

 As ctfe support matures, I dream of a full-fledged parser generator that
 can be evaluated at compile time, although that's way more heavy duty
 than what most people will need.


 My spirit port might be an option at some point soon:

 http://www.sstk.co.uk/spiritd.php


Yeah, I had spiritd in the back of my mind when I wrote that. I've never 
used spirit, but from a once-over of the docs, it looks like something I 
should make a point of getting familiar with this summer.


 Haven't tried it with current d2 compilers, but should still work w/ d1.

 It uses lots of template code to compose parsers, so in theory it ought
 to be usable at compile time. I doubt it can be used that way at the
 moment, but hopefully dmd will get there and then you'll be able to do
 some seriously funky stuff at compile time.


That would be awesome.
Apr 20 2010
prev sibling parent Don <nospam nospam.com> writes:
Ellery Newcomer wrote:

 On 04/20/2010 10:17 AM, Don wrote:

 I don't see anything particularly ugly or hacky about CTFE code, like
 the code below:

 int greater(int a, int b) {
 if (a>b) return a;
 else return b;
 }

 int [ greater(short.max/3, 515) ] foo;


 
 It isn't. Things don't get ugly until you hit the relatively low ceiling 
 of what ctfe can handle. I don't pay much attention to where that 
 ceiling is, but I am aware that the situation has been improving mostly 
 due to your efforts, and I appreciate that.
 
 So far, all my ctfe code is code generation for string mixins, and it's 
 fairly modest stuff, to the point where a simple string formatting 
 function could probably take care of a fair portion of the problem. It's 
 just that I don't know of any. (I'm in D1/Tango land)
 
 The other half of the problem is parsing input strings for simple dsls. 
 Having to write out a lexer/parser from scratch just sucks regardless, 
 and writing it in ctfe code, which isn't allowed composite data 
 structures (last I checked) is really not worth doing. 


Now that it can do struct member functions and delegates, it's possible 
to do fairly sophisticated things in CTFE. Unfortunately the two killer 
bugs (bug 1330 and 1382) still aren't fixed, so the frustration level is 
still pretty bad. We're still a few releases away from getting a fix for 
those ones.

I think D fails

 on its promise here, where it says "with string mixins you can create 
 your own dsls", but then provides no additional support.


Yes, the two CTFE bugs I mention are blockers. Once they're fixed, it 
will be fairly simple to add support for classes, exceptions, and nearly 
everything else.


 
 As ctfe support matures, I dream of a full-fledged parser generator that 
 can be evaluated at compile time, although that's way more heavy duty 
 than what most people will need.
 
 Another idea is something more on the lines of haskell's parsec, which 
 provides building blocks for common tokens, etc. I don't know much about 
 it, though.
 
 Even compile time regular expressions would be better than nothing..
 
 And whatever it is, it needs to be in the standard library.


Exactly.
Apr 20 2010
prev sibling next sibling parent reply Don <nospam nospam.com> writes:
Ellery Newcomer wrote:

 Are there any good libraries for ctfe/code generation?
 
 I don't know, things like parsing support for compile time strings, 
 string formatting, type <-> string
 
 My project seems to be growing ctfe, and it's all horribly hacky and 
 ugly code.


I've found that the function below improves things immensely. I might 
propose it for std.metastrings in the next Phobos release.

===============

/** Escape any quotes and backslashes inside the given string,
  * prefixing them with the given escape sequence. Use `\` to escape
  * once, `\\\` to escape twice.
  */
string enquote(string instr, string escape = `\`)
{
     // This function is critical for compilation speed.
     // Need to minimise the number of allocations.
     // It's worth using copy-on-write even for CTFE.

     for(int i = 0; i < instr.length; ++i)
     {
         if (instr[i] == '"' || instr[i] == '\\')
         {
             string str = instr[0..i] ~ escape;
             int m = i;
             foreach(int k, char c; instr[i+1..$])
             {
                 if (c=='"' || c=='\\')
                 {
                     str ~= instr[m..i+1+k] ~ escape;
                     m = i+k+1;
                 }
             }
             return str ~ instr[m..$];
         }
     }
     return instr;
}

unittest {
     assert(enquote(`"a\"`)==`\"a\\\"`);
     assert(enquote(`abc`)==`abc`);
}
Apr 20 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Don:

 I've found that the function below improves things immensely. I might 
 propose it for std.metastrings in the next Phobos release.


Can you tell me what this function can be useful for?

Thank you, bye,
bearophile
Apr 20 2010
parent Don <nospam nospam.com> writes:
bearophile wrote:

 Don:

 I've found that the function below improves things immensely. I might 
 propose it for std.metastrings in the next Phobos release.


 
 Can you tell me what this function can be useful for?
 
 Thank you, bye,
 bearophile


If you have recursive mixins (such as appears in many of the examples 
posted by downs), it lets you get rid of the horrible `\"` and 
backslashes which appear everywhere.
Apr 21 2010
prev sibling parent reply FeepingCreature <default_357-line yahoo.de> writes:
On 20.04.2010 01:49, Ellery Newcomer wrote:

 Are there any good libraries for ctfe/code generation?
 
 I don't know, things like parsing support for compile time strings,
 string formatting, type <-> string
 
 My project seems to be growing ctfe, and it's all horribly hacky and
 ugly code.


This might be useful if you're on D1:
http://dsource.org/projects/scrapple/browser/trunk/tools/tools/ctfe.d

Contents:

 * ctToString(int)
 * ctAtoi(string)
 * ctToLower(string)
 * ctFind(string, string)
 * ctFind(string, char)
 * ctRFind(string, string)
 * ctStripL(string)
 * ctStripR(string)
 * ctStrip(string)
 * ctSlice(ref string, string where, bool cutOff = true)
 * ctReplace(string, string, string)
 * ctBetween(string text, string from, string to, bool adhere_right = false)
 * ctReplace(string, string, string{, string, string})
 * Table parsing code (ctTableHeight, ctTableWidth, ctTableUnrollColMajor,
ctTableUnroll, ctTableUnrollColumn, ctTableLookup and related)
 * ctExpand (turns "foo, bar(0, 1)baz" into "|foo|bar0baz|bar1baz", used for
simpler enum generation)
Apr 21 2010
next sibling parent strtr <strtr spam.com> writes:
FeepingCreature Wrote:


 On 20.04.2010 01:49, Ellery Newcomer wrote:

 Are there any good libraries for ctfe/code generation?
 
 I don't know, things like parsing support for compile time strings,
 string formatting, type <-> string
 
 My project seems to be growing ctfe, and it's all horribly hacky and
 ugly code.


 
 This might be useful if you're on D1:
http://dsource.org/projects/scrapple/browser/trunk/tools/tools/ctfe.d
 
 Contents:
 
  * ctToString(int)
  * ctAtoi(string)
  * ctToLower(string)
  * ctFind(string, string)
  * ctFind(string, char)
  * ctRFind(string, string)
  * ctStripL(string)
  * ctStripR(string)
  * ctStrip(string)
  * ctSlice(ref string, string where, bool cutOff = true)
  * ctReplace(string, string, string)
  * ctBetween(string text, string from, string to, bool adhere_right = false)
  * ctReplace(string, string, string{, string, string})
  * Table parsing code (ctTableHeight, ctTableWidth, ctTableUnrollColMajor,
ctTableUnroll, ctTableUnrollColumn, ctTableLookup and related)
  * ctExpand (turns "foo, bar(0, 1)baz" into "|foo|bar0baz|bar1baz", used for
simpler enum generation)


When will Phobos D1 plus be released?
Apr 21 2010
prev sibling next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 04/21/2010 05:43 AM, FeepingCreature wrote:

 On 20.04.2010 01:49, Ellery Newcomer wrote:

 Are there any good libraries for ctfe/code generation?

 I don't know, things like parsing support for compile time strings,
 string formatting, type<->  string

 My project seems to be growing ctfe, and it's all horribly hacky and
 ugly code.


 This might be useful if you're on D1:
http://dsource.org/projects/scrapple/browser/trunk/tools/tools/ctfe.d


Looks like itś exactly what I want, thanks!

A couple notes:

Is there any particular license associated with it? (whatever tangoś is 
would be fine)

When I try to compile, I get forward referencing errors and had to take 
out the import to tools.compat
Apr 21 2010
parent reply FeepingCreature <default_357-line yahoo.de> writes:
On 21.04.2010 16:43, Ellery Newcomer wrote:

 On 04/21/2010 05:43 AM, FeepingCreature wrote:

 On 20.04.2010 01:49, Ellery Newcomer wrote:

 Are there any good libraries for ctfe/code generation?

 I don't know, things like parsing support for compile time strings,
 string formatting, type<->  string

 My project seems to be growing ctfe, and it's all horribly hacky and
 ugly code.


 This might be useful if you're on D1:
 http://dsource.org/projects/scrapple/browser/trunk/tools/tools/ctfe.d


 
 Looks like itś exactly what I want, thanks!
 
 A couple notes:
 
 Is there any particular license associated with it? (whatever tangoś is
 would be fine)
 


Feel free to use it as you wish.


 When I try to compile, I get forward referencing errors and had to take
 out the import to tools.compat
 


Yeah, nevermind that, it just defines string as char[] .. or used to, I think
that's in tools.text nowadays. tools.ctfe is pretty independent from the rest
of tools.


 Also, are there any examples for usage of the table parsing functions?


Here's one from my IF language, for converting dynamic into native types:

  union {
    bool b;
    int i;
    string s;
    float f;
    Scope sr;
  }
  T to(T)() {
    const string Table = `
                 | bool          | int         | string               | float  
| Scope
      -----------+---------------+-------------+----------------------+---------+----------
      Boolean    | b             | b           | b?q{true}p:q{false}p | ø     
 | ø
      Integer    | i != 0        | i           | Format(i)            | i      
| ø
      String     | s == q{true}p | atoi(s)     | s                    | atof(s)
| ø
      Float      | ø             | cast(int) f | Format(f)            | f     
 | ø
      ScopeRef   | !!sr          | ø           | (sr?sr.fqn:q{(null:r)}p) | ø
  | sr
      ScopeValue | sr.value().to!(T) | sr.value().to!(T) | sr.value().to!(T) |
sr.value().to!(T) | sr`;
    mixin(ctTableUnrollColMajor(Table,
      /* First parameter is the outer template, $COL substituting with the
column name, and $BODY substituting with the inner template*/
      `static if (is(T == $COL))
        switch (flatType) {
          $BODY
          default: throw new Exception(Format("Invalid type: ", flatType));
        }
      else `,
      /* Second parameter is the inner template, $ROW substituting with the row
name and $CELL with the cell's content.
      `case FlatType.$ROW:
        static if (q{$CELL}p == "ø")
          throw new Exception(q{Cannot convert $ROW to $COL: }p~to!(string)~q{!
}p);
        else return $CELL;
      `
    ).litstring_expand() /* expand q{}p into ""s with correct nesting */ ~
`static assert(false, "Unsupported type: "~T.stringof); `);
  }



 And to whom do I give attribution?


Attribution is not necessary. I'm glad someone finds it useful.
Apr 22 2010
parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
well, here you are:
http://www.dsource.org/projects/dexcelapi/browser/trunk/src/dxl/util/ctfe.d

and here is where I use it:
http://www.dsource.org/projects/dexcelapi/browser/trunk/src/dxl/biff/Enums.d

I don't think it's worth the effort of changing that stuff to work with 
your table parsing, but if you make your tables work with '\u2500' and 
friends, I might do it..


On 04/22/2010 12:36 PM, FeepingCreature wrote:

 On 21.04.2010 16:43, Ellery Newcomer wrote:

 On 04/21/2010 05:43 AM, FeepingCreature wrote:

 On 20.04.2010 01:49, Ellery Newcomer wrote:

 Are there any good libraries for ctfe/code generation?

 I don't know, things like parsing support for compile time strings,
 string formatting, type<->   string

 My project seems to be growing ctfe, and it's all horribly hacky and
 ugly code.


 This might be useful if you're on D1:
 http://dsource.org/projects/scrapple/browser/trunk/tools/tools/ctfe.d


 Looks like itś exactly what I want, thanks!

 A couple notes:

 Is there any particular license associated with it? (whatever tangoś is
 would be fine)


 Feel free to use it as you wish.


 When I try to compile, I get forward referencing errors and had to take
 out the import to tools.compat


 Yeah, nevermind that, it just defines string as char[] .. or used to, I think
that's in tools.text nowadays. tools.ctfe is pretty independent from the rest
of tools.


 Also, are there any examples for usage of the table parsing functions?


 Here's one from my IF language, for converting dynamic into native types:

    union {
      bool b;
      int i;
      string s;
      float f;
      Scope sr;
    }
    T to(T)() {
      const string Table = `
                   | bool          | int         | string               | float
  | Scope
        -----------+---------------+-------------+----------------------+---------+----------
        Boolean    | b             | b           | b?q{true}p:q{false}p | ø   
   | ø
        Integer    | i != 0        | i           | Format(i)            | i    
  | ø
        String     | s == q{true}p | atoi(s)     | s                    |
atof(s) | ø
        Float      | ø             | cast(int) f | Format(f)            | f   
   | ø
        ScopeRef   | !!sr          | ø           | (sr?sr.fqn:q{(null:r)}p) |
ø   | sr
        ScopeValue | sr.value().to!(T) | sr.value().to!(T) | sr.value().to!(T)
| sr.value().to!(T) | sr`;
      mixin(ctTableUnrollColMajor(Table,
        /* First parameter is the outer template, $COL substituting with the
column name, and $BODY substituting with the inner template*/
        `static if (is(T == $COL))
          switch (flatType) {
            $BODY
            default: throw new Exception(Format("Invalid type: ", flatType));
          }
        else `,
        /* Second parameter is the inner template, $ROW substituting with the
row name and $CELL with the cell's content.
        `case FlatType.$ROW:
          static if (q{$CELL}p == "ø")
            throw new Exception(q{Cannot convert $ROW to $COL:
}p~to!(string)~q{! }p);
          else return $CELL;
        `
      ).litstring_expand() /* expand q{}p into ""s with correct nesting */ ~
`static assert(false, "Unsupported type: "~T.stringof); `);
    }



 And to whom do I give attribution?


 Attribution is not necessary. I'm glad someone finds it useful.
Apr 22 2010
prev sibling parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
Also, are there any examples for usage of the table parsing functions?

And to whom do I give attribution?
Apr 21 2010