www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - The Language Machine - a metalanguage in D with D-to-D translator

reply Peri Hankey <mpah thegreen.co.uk> writes:
Hello

I have just uploaded my language machine, toolkit for language and 
grammar to sourceforge. The  url is http://languagemachine.sourceforge.net.

Here's the summary as posted to freshmeat:

The Language Machine is a free software toolkit for language and 
grammar: a library, main program, and self-hosted rule compilers with 
one frontend. The system is easy to use. It directly implements 
unrestricted rule-based grammars with actions and external interfaces. 
In its present form the language machine consists of a shared library, a 
minimal main program, and a metalanguage (LMN - language machine/meta 
notation). The shared library is written in the Digital Mars D language 
using gdc, the D language front end for the GNU gcc compiler collection. 
The LMN compilers share a common frontend and translate into C and D as 
well as producing C, D, and shell script wrappers for an internal 
format. There is a proof-of-concept interface to Fabrice Bellard's tiny 
C compiler, with a working example. LMN source is compatible with the 
wikipedia text format - the project's web pages are an example of how 
the language machine can be used. Diagnostics include the lm-diagram, a 
diagram which show what happens when unrestricted grammar rules are applied.

The D-to-D translator example is on the web pages and is included in the 
distribution - without much annotation. It has been used to translate 
the whole of the language machine, and the result used to repeat the 
whole process. It has also been used to translate DMDscript. The 
translator is a ruleset with a frontend and a backend, and it does a 
fairly thorough analysis, but it may well be inaccurate in its idea of D 
grammar.

C is used only for external interfaces - the whole system is written 
entirely in D with a small metalanguage bootstrap.

Your comments and ideas are welcome.

Peri
Jul 11 2005
next sibling parent reply Nod <Nod_member pathlink.com> writes:
In article <dau3bc$1hj0$1 digitaldaemon.com>, Peri Hankey says...

Hello

I have just uploaded my language machine, toolkit for language and 
grammar to sourceforge. The  url is http://languagemachine.sourceforge.net.

Here's the summary as posted to freshmeat:

The Language Machine is a free software toolkit for language and 
grammar: a library, main program, and self-hosted rule compilers with 
one frontend. The system is easy to use. It directly implements 
unrestricted rule-based grammars with actions and external interfaces. 
In its present form the language machine consists of a shared library, a 
minimal main program, and a metalanguage (LMN - language machine/meta 
notation). The shared library is written in the Digital Mars D language 
using gdc, the D language front end for the GNU gcc compiler collection. 
The LMN compilers share a common frontend and translate into C and D as 
well as producing C, D, and shell script wrappers for an internal 
format. There is a proof-of-concept interface to Fabrice Bellard's tiny 
C compiler, with a working example. LMN source is compatible with the 
wikipedia text format - the project's web pages are an example of how 
the language machine can be used. Diagnostics include the lm-diagram, a 
diagram which show what happens when unrestricted grammar rules are applied.

The D-to-D translator example is on the web pages and is included in the 
distribution - without much annotation. It has been used to translate 
the whole of the language machine, and the result used to repeat the 
whole process. It has also been used to translate DMDscript. The 
translator is a ruleset with a frontend and a backend, and it does a 
fairly thorough analysis, but it may well be inaccurate in its idea of D 
grammar.

C is used only for external interfaces - the whole system is written 
entirely in D with a small metalanguage bootstrap.

Your comments and ideas are welcome.

Peri


On first glance, the source looks really clean, and the meta-language well
thought out. I'm not too fond of the syntax, but I guess one could change that
quite easily, no? The information on the webpage is really good too; I was glued
to the guide for quite a while. Me likes meta-languages :)

A few questions though:
* What exactly does the D-to-D translator do? Have I understood correctly if I
say that it lexes D source, verifying syntax correctness, and outputs D source
in a standard format?

* And what is the work division between the frontend/backend?

* Is the language machine meant primarily for interpreted languages, or could it
be used as, say, a frontend for GCC?

I'll post more thorough impressions when I have had time to play with it a bit.
As of currently, I really need to sleep. 

Good work!

-Nod-
Jul 12 2005
parent reply Peri Hankey <mpah thegreen.co.uk> writes:
Nod wrote:

 In article <dau3bc$1hj0$1 digitaldaemon.com>, Peri Hankey says...
 

Hello

I have just uploaded my language machine, toolkit for language and 
grammar to sourceforge. The  url is http://languagemachine.sourceforge.net.

Here's the summary as posted to freshmeat:

The Language Machine is a free software toolkit for language and 
grammar: a library, main program, and self-hosted rule compilers with 
one frontend. The system is easy to use. It directly implements 
unrestricted rule-based grammars with actions and external interfaces. 
In its present form the language machine consists of a shared library, a 
minimal main program, and a metalanguage (LMN - language machine/meta 
notation). The shared library is written in the Digital Mars D language 
using gdc, the D language front end for the GNU gcc compiler collection. 
The LMN compilers share a common frontend and translate into C and D as 
well as producing C, D, and shell script wrappers for an internal 
format. There is a proof-of-concept interface to Fabrice Bellard's tiny 
C compiler, with a working example. LMN source is compatible with the 
wikipedia text format - the project's web pages are an example of how 
the language machine can be used. Diagnostics include the lm-diagram, a 
diagram which show what happens when unrestricted grammar rules are applied.

The D-to-D translator example is on the web pages and is included in the 
distribution - without much annotation. It has been used to translate 
the whole of the language machine, and the result used to repeat the 
whole process. It has also been used to translate DMDscript. The 
translator is a ruleset with a frontend and a backend, and it does a 
fairly thorough analysis, but it may well be inaccurate in its idea of D 
grammar.

C is used only for external interfaces - the whole system is written 
entirely in D with a small metalanguage bootstrap.

Your comments and ideas are welcome.

Peri


 
 
 On first glance, the source looks really clean, and the meta-language well
 thought out. I'm not too fond of the syntax, but I guess one could change that
 quite easily, no? The information on the webpage is really good too; I was
glued
 to the guide for quite a while. Me likes meta-languages :)


That's terrific - most of the people I have explained it to fall asleep 
instantly - a good cure for insomnia.

It's easy to change the syntax, but hard to find the right one. I was 
looking for a minimal syntax to get the job done.


 
 A few questions though:
 * What exactly does the D-to-D translator do? Have I understood correctly if I
 say that it lexes D source, verifying syntax correctness, and outputs D source
 in a standard format?


The D-to-D translator started because I was looking for ways to 
integrate grammar rules directly as an extension of D (this would 
probably have been a mistake). The best way to prove to myself that my 
rules were understanding D source was to produce something could be 
compiled and tested. Also there was a lot of talk here about 
experimenting with things that could be done with D, so I thought there 
might be some interest


 * And what is the work division between the frontend/backend?


Both frontend and backend are just rules: the LMN notation allows you to 
produce prepackaged analyses such as

     '?{' whatnot :F expression :X '}' <- thing :{ whatnotkindofthing :{ 
operator :F :X }};

This means: '?{' then a  "whatnot" (with a value we call F) followed by 
an "expression" (with a value we call X) and by '}' can be treated as a 
"thing" which produces a bag containing a "whatnotkindofthing" which 
produces a bag containing an "operator" with the F and the X as its values.

A backend rule to handle this would receive a "whatnotkindofthing" in 
the first place. It can strip off some of the packaging:

     whatnotkindofthing :Q  <- code - 'hoho it\'s a ' Q eoc;

Or it can look inside the bag:

     whatnotkindofthing : thiskindofthing :T :F :X  <- code - .....

Here it looks in the bag and forces the "operator" to be treated as a 
"thiskindofthing".

So the frontend rules produce prepackaged stuff that is recognised and 
translated into some external representation by the backend.

There's a hypersimple example of how this is done in the rules for 
converting mediawiki text to html: 
http://languagemachine.sourceforge.net/mediawiki.html.

The prepackaged stuff is in a sense the symbolic representation of a 
parse tree that can be treated as input by backend rules. But the same 
kind of rule is used at the lexical level, for syntax, and in the back end.


 * Is the language machine meant primarily for interpreted languages, or could
it
 be used as, say, a frontend for GCC?


It could be used in either way: in particular, you could use the 
interface with tcc to compile via C into machine code im memory for 
immediate execution.


 I'll post more thorough impressions when I have had time to play with it a bit.
 As of currently, I really need to sleep. 
 
 Good work!
 
 -Nod-
Jul 12 2005
parent reply Nod <Nod_member pathlink.com> writes:
In article <davvoc$8h9$1 digitaldaemon.com>, Peri Hankey says...

 <snip>


Whoa! The Language Machine is way cool, indeed!

I have a ton of questions, but I feel it is rather OT for this newsgroup...
Would it be possible for you to set up a dedicated communications forum for TLM?

A mailing-list on sourceforge would suffice, but might I suggest requesting an
account on www.dsource.org? In addition to providing a nice forum, it also gives
access to an SVN repository, rather than the ancient CVS used on sf.net.

And of course, www.dsource.org is a lot more responsive to feature requests than
sf.net :)

-Nod-
Jul 14 2005
parent reply Peri Hankey <mpah thegreen.co.uk> writes:
Nod wrote:

 In article <davvoc$8h9$1 digitaldaemon.com>, Peri Hankey says...
 

<snip>


 
 Whoa! The Language Machine is way cool, indeed!
 
 I have a ton of questions, but I feel it is rather OT for this newsgroup...
 Would it be possible for you to set up a dedicated communications forum for
TLM?
 
 A mailing-list on sourceforge would suffice, but might I suggest requesting an
 account on www.dsource.org? In addition to providing a nice forum, it also
gives
 access to an SVN repository, rather than the ancient CVS used on sf.net.
 
 And of course, www.dsource.org is a lot more responsive to feature requests
than
 sf.net :)
 
 -Nod-
 
 


Nod - thanks - I agree that this isn't the place. I have emailed 
dsource.org, and we'll see what happens.

In the meantime there are 2 forums at the sourceforge project pages 
https://sourceforge.net/forum/?group_id=143185 - we can later distill 
and extract useful stuff to a faq or to wherever the forum finally goes.

Peri
Jul 14 2005
parent Brad Anderson <brad dsource.dot.org> writes:
Peri Hankey wrote:

 Nod wrote:
 

 In article <davvoc$8h9$1 digitaldaemon.com>, Peri Hankey says...


 <snip>


 Whoa! The Language Machine is way cool, indeed!

 I have a ton of questions, but I feel it is rather OT for this
 newsgroup...
 Would it be possible for you to set up a dedicated communications
 forum for TLM?

 A mailing-list on sourceforge would suffice, but might I suggest
 requesting an
 account on www.dsource.org? In addition to providing a nice forum, it
 also gives
 access to an SVN repository, rather than the ancient CVS used on sf.net.

 And of course, www.dsource.org is a lot more responsive to feature
 requests than
 sf.net :)

 -Nod-


 
 Nod - thanks - I agree that this isn't the place. I have emailed
 dsource.org, and we'll see what happens.


Been on vacation but I'm back now.  I've sent Peri an email...

BA


 
 In the meantime there are 2 forums at the sourceforge project pages
 https://sourceforge.net/forum/?group_id=143185 - we can later distill
 and extract useful stuff to a faq or to wherever the forum finally goes.
 
 Peri
Jul 18 2005
prev sibling parent reply B Shropshire <B_member pathlink.com> writes:
In article <dau3bc$1hj0$1 digitaldaemon.com>, Peri Hankey says...


The D-to-D translator example is on the web pages and is included in the 
distribution - without much annotation. It has been used to translate 
the whole of the language machine, and the result used to repeat the 
whole process.


How would this thing do for a d-to-HTML converter. Maybe generate a syntax
highlighted HTML file from a d file, maybe even parse out the comments into
something nicer to look at.
Jul 12 2005
parent Peri Hankey <mpah thegreen.co.uk> writes:
B Shropshire wrote:

 In article <dau3bc$1hj0$1 digitaldaemon.com>, Peri Hankey says...
 
 

The D-to-D translator example is on the web pages and is included in the 
distribution - without much annotation. It has been used to translate 
the whole of the language machine, and the result used to repeat the 
whole process.


 
 
 How would this thing do for a d-to-HTML converter. Maybe generate a syntax
 highlighted HTML file from a d file, maybe even parse out the comments into
 something nicer to look at.
 
 


Yes - it would be a matter of making an adapted backend ruleset. At 
present the translator consists of

d2xfe.lmn - the frontend ruleset
d2dbe.lmn - the d2d backend ruleset

But if you look at the metalanguage compilers, these are constructed 
with (frontend, backend) pairs:

lmn2xfe.lmn, lmn2mbe.lmn - compile to own loader format
lmn2xfe.lmn, lmn2dbe.lmn - wrap loader format as D
lmn2xfe.lmn, lmn2cbe.lmn - wrap loader format as C
lmn2xfe.lmn, lmn2Dbe.lmn - compile to D
lmn2xfe.lmn, lmn2Cbe.lmn - compile to C

The ruleset pairs can either be combined as source or concatenated at 
runtime.

Peri
Jul 12 2005