digitalmars.D.announce - Enki 1.1 Released
- pragma (19/19) Jun 13 2006 Enki 1.1 is now available.
- BCS (18/42) Jun 14 2006 Cool, tried it out. Didn't get to far (I haven't used a parser generator...
- pragma (13/33) Jun 14 2006 BCS, thanks for being patient with all this. I hope I haven't lost your
- BCS (46/69) Jun 14 2006 Nope, you have only wet it.
- pragma (13/59) Jun 14 2006 I'm inclined to agree, but its also at cross-purposes with the philosoph...
- BCS (27/51) Jun 14 2006 OK I'll grant that things might move, however in the general case those
- pragma (31/63) Jun 15 2006 That's not a bad idea. Something like a .supportlib() directive or some...
- BCS (15/56) Jun 15 2006 I'm already working on a lexer that so far does just about everything
- Georg Wrede (2/22) Jun 15 2006 That's what Defaults are for!
- Chris Nicholson-Sauls (6/7) Jun 14 2006 This subproject of yours couldn't have come at a better time... here I w...
- pragma (5/12) Jun 14 2006 You're quite welcome!
- BCS (19/19) Jun 20 2006 A few patches to make it compile under 0.161
- pragma (4/23) Jun 21 2006 Thanks! I was just going to get a patch underway for that - I'll take a...
- BCS (28/31) Jun 26 2006 [...]
Enki 1.1 is now available. Documentation (still in progress) http://www.dsource.org/projects/ddl/wiki/Enki * Download http://svn.dsource.org/projects/ddl/downloads/enki.1.1.win32.bin.zip http://svn.dsource.org/projects/ddl/downloads/enki.sdk.1.1.src.zip Enki is a frontend parser generator that uses an annotated EBNF syntax to produce .d sourcefiles. More details can be found at the link above. Enki 1.1 Changes: - enhancement: semantic analysis of binding types by search is now in declaration order - bug: repeated binding declarations in generated parser - bug: binding names were output as 'bind_x' in the comments for some expressions - enhancement: added function-call capability to ' ' operator - enhancement: added '&' (custom-terminal) operator - enhancement: added minimal sourcecode for custom parsers to binary distribution - EricAnderton at yahoo
Jun 13 2006
pragma wrote:Enki 1.1 is now available. Documentation (still in progress) http://www.dsource.org/projects/ddl/wiki/Enki * Download http://svn.dsource.org/projects/ddl/downloads/enki.1.1.win32.bin.zip http://svn.dsource.org/projects/ddl/downloads/enki.sdk.1.1.src.zip Enki is a frontend parser generator that uses an annotated EBNF syntax to produce .d sourcefiles. More details can be found at the link above. Enki 1.1 Changes: - enhancement: semantic analysis of binding types by search is now in declaration order - bug: repeated binding declarations in generated parser - bug: binding names were output as 'bind_x' in the comments for some expressions - enhancement: added function-call capability to ' ' operator - enhancement: added '&' (custom-terminal) operator - enhancement: added minimal sourcecode for custom parsers to binary distribution - EricAnderton at yahooCool, tried it out. Didn't get to far (I haven't used a parser generator before, so...). The generated parser still doesn't compile out of the box. I dug through the code and added a few imports to the render() function to get it to work. Here is a patch of what I did. *** old\enki\EnkiBackend.d Wed Jun 14 11:47:30 2006 --- new\sand\EnkiBackend.d Wed Jun 14 11:49:57 2006 *************** *** 184,189 **** --- 184,191 ---- emit("module " ~ moduleName ~ ";"); } emit("debug private import std.stdio;"); + emit("import enki.BaseParser;"); + emit("import enki.types;"); foreach(imp; imports){ emit("private import " ~ imp ~ ";"); }
Jun 14 2006
In article <e6plq9$1upb$1 digitaldaemon.com>, BCS says...pragma wrote:BCS, thanks for being patient with all this. I hope I haven't lost your interest. ;) What you've done here is hack the code-generator to emit the imports you need - I like the cut of your jib. As the documentation is incomplete, I can't blame you for trying this, but there is an easier way to do this. The .import() directive does this job nicely.Enki 1.1 is now available.Cool, tried it out. Didn't get to far (I haven't used a parser generator before, so...). The generated parser still doesn't compile out of the box. I dug through the code and added a few imports to the render() function to get it to work. Here is a patch of what I did. *** old\enki\EnkiBackend.d Wed Jun 14 11:47:30 2006 --- new\sand\EnkiBackend.d Wed Jun 14 11:49:57 2006 *************** *** 184,189 **** --- 184,191 ---- emit("module " ~ moduleName ~ ";"); } emit("debug private import std.stdio;"); + emit("import enki.BaseParser;"); + emit("import enki.types;"); foreach(imp; imports){ emit("private import " ~ imp ~ ";"); }.import("my.import.here");A solid example of this and other directives are in Enki's BNF definition: http://svn.dsource.org/projects/ddl/trunk/enki/enki.bnf You may find the .baseclass() .classname() and .code{{{ }}} directives helpful as well. The .start() directive is now depricated, and there's the .include() and .alias() directives that aren't used in that particular file. - EricAnderton at yahoo
Jun 14 2006
pragma wrote:In article <e6plq9$1upb$1 digitaldaemon.com>, BCS says...[...]Cool, tried it out. Didn't get to far (I haven't used a parser generator before, so...). The generated parser still doesn't compile out of the box. I dug through the code and added a few imports to the render() function to get it to work. Here is a patch of what I did.BCS, thanks for being patient with all this. I hope I haven't lost your interest. ;)Nope, you have only wet it.What you've done here is hack the code-generator to emit the imports you need - I like the cut of your jib. As the documentation is incomplete, I can't blame you for trying this, but there is an easier way to do this. The .import() directive does this job nicely.Are these imports not always needed? Won't any parser use things from these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list). On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed..import("my.import.here");A solid example of this and other directives are in Enki's BNF definition: http://svn.dsource.org/projects/ddl/trunk/enki/enki.bnfI'll take closer a look at that soon. ======== And on another topic: another hack It fixes a default case bug. Without it any rule of the form rule ::= (foo bar) always matches even if both foo and bar don't match. The case where the rule has annotations works as advertised. <patch> *** old\enki\Rule.d Wed Jun 14 13:50:06 2006 --- new\enki\Rule.d Wed Jun 14 13:50:59 2006 *************** *** 241,251 **** } public void renderPass(CodeGenerator generator){ ! generator.emit("return ResultT!(bool)(true);"); } public void renderFail(CodeGenerator generator){ ! generator.emit("return ResultT!(bool)(false);"); } public void semanticPass(Rule thisRule,BaseEnkiParser root){ --- 242,252 ---- } public void renderPass(CodeGenerator generator){ ! generator.emit("return ResultT!(bool)(true,true);"); } public void renderFail(CodeGenerator generator){ ! generator.emit("return ResultT!(bool)(false,false);"); } public void semanticPass(Rule thisRule,BaseEnkiParser root){ </patch>
Jun 14 2006
In article <e6ptut$2c61$1 digitaldaemon.com>, BCS says...I'm inclined to agree, but its also at cross-purposes with the philosophy I had in mind with the tool. I really think that it should try to impose as little as possible, wherever possible. In this case, what happens if the user moves or renames 'types.d' to something else, or merged the templates in with their base parser? I'd rather make imports explicit so the developer has this flexibility.Are these imports not always needed? Won't any parser use things from these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list). On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed..import("my.import.here");And on another topic: another hack It fixes a default case bug. Without it any rule of the form rule ::= (foo bar) always matches even if both foo and bar don't match. The case where the rule has annotations works as advertised. <patch> *** old\enki\Rule.d Wed Jun 14 13:50:06 2006 --- new\enki\Rule.d Wed Jun 14 13:50:59 2006 *************** *** 241,251 **** } public void renderPass(CodeGenerator generator){ ! generator.emit("return ResultT!(bool)(true);"); } public void renderFail(CodeGenerator generator){ ! generator.emit("return ResultT!(bool)(false);"); } public void semanticPass(Rule thisRule,BaseEnkiParser root){ --- 242,252 ---- } public void renderPass(CodeGenerator generator){ ! generator.emit("return ResultT!(bool)(true,true);"); } public void renderFail(CodeGenerator generator){ ! generator.emit("return ResultT!(bool)(false,false);"); } public void semanticPass(Rule thisRule,BaseEnkiParser root){ </patch>Ah, now that's a serious bug. Thanks for tracking it down. I'm glad that things are written such that you can find this stuff so easily. Thanks for the patch! Another way to tackle this would be tochange the failure rule to this:public void renderFail(CodeGenerator generator){ ! generator.emit("return ResultT!(bool)();"); }Which is equivalent. When I get around to expanding the docs to detail how to write custom rules, this will make more sense. - EricAnderton at yahoo
Jun 14 2006
pragma wrote:In article <e6ptut$2c61$1 digitaldaemon.com>, BCS says...OK I'll grant that things might move, however in the general case those imports will be needed by most people. Good defaults make for good programs. How about make those imports the default and put in some sort of Override directive.I'm inclined to agree, but its also at cross-purposes with the philosophy I had in mind with the tool. I really think that it should try to impose as little as possible, wherever possible. In this case, what happens if the user moves or renames 'types.d' to something else, or merged the templates in with their base parser? I'd rather make imports explicit so the developer has this flexibility.Are these imports not always needed? Won't any parser use things from these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list). On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed..import("my.import.here");Ah, now that's a serious bug. Thanks for tracking it down. I'm glad that things are written such that you can find this stuff so easily. Thanks for the patch!Find-in-files, grep and kin are wonderful aren't they. <g> Another thought; as it is written, Enki is purely an ASCII text program. It would be nice if the parsed data had its own type that, is only used for the input data. This would allow for the parsing of an array on any type (enums or structs come to mind). Of course this would requirer the introduction of a new terminal matching syntax. Just a sketch of what might be done: <code name="somewher.d"> ... enum bar { i1, i2, i3} struct baz{ bar thisType; } ... </code> <code name="syn.bnf"> ... .parsetype(baz) ... foo ::= .thisType == bar.i1; ... </code>
Jun 14 2006
In article <e6q2pu$2cqo$1 digitaldaemon.com>, BCS says...pragma wrote:That's not a bad idea. Something like a .supportlib() directive or something. I'll consider it. :)In article <e6ptut$2c61$1 digitaldaemon.com>, BCS says...OK I'll grant that things might move, however in the general case those imports will be needed by most people. Good defaults make for good programs. How about make those imports the default and put in some sort of Override directive.I'm inclined to agree, but its also at cross-purposes with the philosophy I had in mind with the tool. I really think that it should try to impose as little as possible, wherever possible. In this case, what happens if the user moves or renames 'types.d' to something else, or merged the templates in with their base parser? I'd rather make imports explicit so the developer has this flexibility.Are these imports not always needed? Won't any parser use things from these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list). On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed..import("my.import.here");Another thought; as it is written, Enki is purely an ASCII text program. It would be nice if the parsed data had its own type that, is only used for the input data. This would allow for the parsing of an array on any type (enums or structs come to mind). Of course this would requirer the introduction of a new terminal matching syntax.I'm well ahead of you on this front. I'm presently composing a lexer for D, which will then support a secondary parser for the lexed tokens. Both of these will be Enki based. So far I'm starting to see the limitations of the String/char[] setup, and yearn for multi-byte char support. So I'll be working on a way to provide this. As far as non-ascii parsing is concerned, Enki can already support this to an extent. code{{{ enum MyType: uint{ Foo, Bar, Baz } }}} MyRule ::= &Foo:x &Bar:y &Baz:z; Under the hood, Enki will generate terminal expressions for each & term, which are then generated as "terminal(Foo)". As long as your base parser has a terminal(MyType) declared, you can handle these expressions. It then becomes the responsility of that terminal to determine if the token matches or not. As far as the internal stream of data is concerned, the IParser position() and sliceData() methods abstract the data away to the same extent. When working with a non-text input, sliceData() can be redefined as returning something other than string. The only caveat here is that you must be careful to avoid expressions cases where Enki will evaluate the default type of a binding to String: Foobar ::= ("hello world"):x;.parsetype(baz)A wonderful idea. This way Enki can use this as the default binding type when it encounters expressions like the one above. - EricAnderton at yahoo
Jun 15 2006
BCS <BCS pathlink.com> pragma wrote:In article <e6q2pu$2cqo$1 digitaldaemon.com>, BCS says...I'm already working on a lexer that so far does just about everything but numeric literals, lexing it down to an array of structs. It is totally native D and is done from the start with the intention that it be easy to change. Adding a static token type (say "~~") involves about three lines of code in the lexer. Adding something like a new type of string only involves adding two of those lines and a function to find the rest. Its still a bit buggy but it works for the most part.Another thought; as it is written, Enki is purely an ASCII text program. It would be nice if the parsed data had its own type that, is only used for the input data. This would allow for the parsing of an array on any type (enums or structs come to mind). Of course this would requirer the introduction of a new terminal matching syntax.I'm well ahead of you on this front. I'm presently composing a lexer for D, which will then support a secondary parser for the lexed tokens. Both of these will be Enki based.So far I'm starting to see the limitations of the String/char[] setup, and yearn for multi-byte char support. So I'll be working on a way to provide this. As far as non-ascii parsing is concerned, Enki can already support this to an extent.What I did for mine is make a wrapper Stream struct that presents the whole stream as dchars. For convenience it also catches look-ahead and such. I'm not sure what goals your project is perusing, mine is easy modification targeted at alpha testing of features. However I intend to drop my code into the wild at some point so...code{{{ enum MyType: uint{ Foo, Bar, Baz } }}} MyRule ::= &Foo:x &Bar:y &Baz:z; Under the hood, Enki will generate terminal expressions for each & term, which are then generated as "terminal(Foo)". As long as your base parser has a terminal(MyType) declared, you can handle these expressions. It then becomes the responsility of that terminal to determine if the token matches or not. As far as the internal stream of data is concerned, the IParser position() and sliceData() methods abstract the data away to the same extent. When working with a non-text input, sliceData() can be redefined as returning something other than string. The only caveat here is that you must be careful to avoid expressions cases where Enki will evaluate the default type of a binding to String: Foobar ::= ("hello world"):x;I'll check that out.
Jun 15 2006
pragma wrote:In article <e6ptut$2c61$1 digitaldaemon.com>, BCS says...That's what Defaults are for!I'm inclined to agree, but its also at cross-purposes with the philosophy I had in mind with the tool. I really think that it should try to impose as little as possible, wherever possible. In this case, what happens if the user moves or renames 'types.d' to something else, or merged the templates in with their base parser? I'd rather make imports explicit so the developer has this flexibility.Are these imports not always needed? Won't any parser use things from these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list). On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed..import("my.import.here");
Jun 15 2006
pragma wrote:Enki 1.1 is now available.This subproject of yours couldn't have come at a better time... here I was writing a byte-compiler/vm for a simple scripted object-database language in another project of mine (codename "Lyra"), and you drop this lovely golden egg right in my lap! :) Goodbye messy amateur object hierarchy, hello much easier to work with grammar file and backend library! -- Chris Nicholson-Sauls
Jun 14 2006
In article <e6pqne$2826$1 digitaldaemon.com>, Chris Nicholson-Sauls says...pragma wrote:You're quite welcome! If you find any bugs, or have any suggestions on how to improve things, don't be afraid to check in and make some noise. ;) - EricAnderton at yahooEnki 1.1 is now available.This subproject of yours couldn't have come at a better time... here I was writing a byte-compiler/vm for a simple scripted object-database language in another project of mine (codename "Lyra"), and you drop this lovely golden egg right in my lap! :) Goodbye messy amateur object hierarchy, hello much easier to work with grammar file and backend library! -- Chris Nicholson-Sauls
Jun 14 2006
A few patches to make it compile under 0.161
--two do nothing function now cause errors, so I put assert(0)s in them.
--the variable "start" is used in nested scopes a lot, so now it is
suffixed with its indent level.
e.g.
{
uint start;
{
uint start;
}
}
becomes
{
uint start1;
{
uint start2;
}
}
www.uidaho.edu/~shro8822/enki_patch_161.zip
Jun 20 2006
In article <e79r5q$1ds7$9 digitaldaemon.com>, BCS says...
A few patches to make it compile under 0.161
--two do nothing function now cause errors, so I put assert(0)s in them.
--the variable "start" is used in nested scopes a lot, so now it is
suffixed with its indent level.
e.g.
{
uint start;
{
uint start;
}
}
becomes
{
uint start1;
{
uint start2;
}
}
www.uidaho.edu/~shro8822/enki_patch_161.zip
Thanks! I was just going to get a patch underway for that - I'll take a look at
your submission.
- EricAnderton at yahoo
Jun 21 2006
In article <e6ntg4$270o$1 digitaldaemon.com>, pragma says...Enki 1.1 is now available. Documentation (still in progress) http://www.dsource.org/projects/ddl/wiki/Enki[...] Yet another patch!!!!! This time to the grammar. Enki won't allow a rule of the form Rule = char[][] var ::= ["foo":var Rule:~var] This would logically generate an array of arrays of char. Adding it doesn't seem to brake anything. *** enki.bnf Mon Jun 26 21:24:51 2006 --- enki.bnf Mon Jun 26 18:56:07 2006 *************** *** 94,100 **** ExplicitParam = new Param(bool isArray,String type,String name) ! ::= Identifier:type ws [ "[]":isArray ws] Identifier:name ; --- 94,103 ---- ExplicitParam = new Param(bool isArray,String type,String name) ! ::= Identifier:type ws [ "[]":isArray Brackets ws] Identifier:name ; ! ! Brackets ::= ["[]" Brackets]; !
Jun 26 2006
In article <e7qdln$juq$1 digitaldaemon.com>, BCS says...In article <e6ntg4$270o$1 digitaldaemon.com>, pragma says... [...] Yet another patch!!!!![...]Adding it doesn't seem to brake anything.How to change a program: 1: think 2: try 3: TEST 4: tell but the generated code doesn't compile . :op It's making some assumptions that I voided, I'll try to get a working version sooner or later.
Jun 27 2006
In article <e7qngu$10lt$1 digitaldaemon.com>, BCS says...In article <e7qdln$juq$1 digitaldaemon.com>, BCS says...Exactly!In article <e6ntg4$270o$1 digitaldaemon.com>, pragma says... [...] Yet another patch!!!!![...]Adding it doesn't seem to brake anything.How to change a program: 1: think 2: try 3: TEST 4: tellbut the generated code doesn't compile . :op It's making some assumptions that I voided, I'll try to get a working version sooner or later.No worries. I'm *way* behind schedule on everything these days (being off your ass with a bad cold will do that). I'll get to folding your contributions in as soon as I can get to it. :) Thank you for helping out ! - EricAnderton at yahoo
Jun 27 2006
pragma wrote: [...]No worries. I'm *way* behind schedule on everything these days (being off your ass with a bad cold will do that). I'll get to folding your contributions in as soon as I can get to it. :) Thank you for helping out ! - EricAnderton at yahooSome cases of negation will result in illegal code { ... mismatch53: // nothing to label } this patch should fix the problem. (don't count on it to apply, I had to hand edit it to remove my other changes.) Another solution would be to make an emptyExpression render to "{}" rather than nothing. *** enki/Expression.d Mon Jul 17 18:15:04 2006 --- enki/Expression.d Mon Jul 17 22:55:01 2006 *************** *** 776,792 **** emit("{//Negate"); indent(); render(expr,gotoMatch,gotoMismatch); render(matchLabel); render(fail); render(mismatchLabel); + if(pass.isEmpty) emit("{}"); render(pass); unindent(); emit("}"); } } public String toBNF(){ return "!" ~ expr.toBNF(); --- 776,791 ----
Jul 18 2006