www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - DMD's lexer available on code.dlang.org

reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
I've created a dub package for the D version of DMD's lexer, generated 
automatically from the C++ source.

github: https://github.com/yebblies/ddmd
dub: http://code.dlang.org/packages/ddmd

There are a few annoying limitations, such that it uses dmd's error printing 
and allocation functions, and requires configuration through 'global'.

Here is an example program that uses the lexer:

======================

import std.stdio;
import std.file;

import ddmd.tokens;
import ddmd.lexer;

/////////////////////////

void main()
{
    Lexer.initLexer();

    string data = "void blah() {} // stuff";
    auto l = new Lexer("myfile", data.ptr, 0, data.length, 0, 0);
    l.nextToken();
    do
    {
        printf("token: %s\n", l.token.toChars());
    }
    while (l.nextToken() != TOKeof);
}

======================

Prints:

token: void
token: blah
token: (
token: )
token: {
token: }
Jan 04 2015
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 5/01/2015 2:07 a.m., Daniel Murphy wrote:

 I've created a dub package for the D version of DMD's lexer, generated
 automatically from the C++ source.

 github: https://github.com/yebblies/ddmd
 dub: http://code.dlang.org/packages/ddmd

 There are a few annoying limitations, such that it uses dmd's error
 printing and allocation functions, and requires configuration through
 'global'.

 Here is an example program that uses the lexer:

 ======================

 import std.stdio;
 import std.file;

 import ddmd.tokens;
 import ddmd.lexer;

 /////////////////////////

 void main()
 {
     Lexer.initLexer();

     string data = "void blah() {} // stuff";
     auto l = new Lexer("myfile", data.ptr, 0, data.length, 0, 0);
     l.nextToken();
     do
     {
         printf("token: %s\n", l.token.toChars());
     }
     while (l.nextToken() != TOKeof);
 }

 ======================

 Prints:

 token: void
 token: blah
 token: (
 token: )
 token: {
 token: }


I saw that. I'm really looking forward to getting my teeth into it and 
doing some good old refactoring. Although that will be a while because 
of the auto generated thing.
Jan 04 2015
next sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Rikki Cattermole"  wrote in message news:m8be2m$1dlp$1 digitalmars.com... 


 I saw that. I'm really looking forward to getting my teeth into it and 
 doing some good old refactoring. Although that will be a while because 
 of the auto generated thing.


There's plenty of refactoring to be done on the C++ side too.
Jan 04 2015
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 5/01/2015 2:39 a.m., Daniel Murphy wrote:

 "Rikki Cattermole"  wrote in message news:m8be2m$1dlp$1 digitalmars.com...

 I saw that. I'm really looking forward to getting my teeth into it and
 doing some good old refactoring. Although that will be a while because
 of the auto generated thing.


 There's plenty of refactoring to be done on the C++ side too.


What I want to do is D side unfortunately. I just hate to see large 
lists in e.g. globals with constructors. When they can be dealt with a 
simple array and a bit of CTFE magic.
Jan 04 2015
prev sibling parent reply "Vadim Lopatin" <coolreader.org gmail.com> writes:
On Sunday, 4 January 2015 at 13:09:42 UTC, Rikki Cattermole wrote:

 On 5/01/2015 2:07 a.m., Daniel Murphy wrote:

 I've created a dub package for the D version of DMD's lexer, 
 generated
 automatically from the C++ source.

 github: https://github.com/yebblies/ddmd
 dub: http://code.dlang.org/packages/ddmd

 There are a few annoying limitations, such that it uses dmd's 
 error
 printing and allocation functions, and requires configuration 
 through
 'global'.

 Here is an example program that uses the lexer:

 ======================

 import std.stdio;
 import std.file;

 import ddmd.tokens;
 import ddmd.lexer;

 /////////////////////////

 void main()
 {
    Lexer.initLexer();

    string data = "void blah() {} // stuff";
    auto l = new Lexer("myfile", data.ptr, 0, data.length, 0, 
 0);
    l.nextToken();
    do
    {
        printf("token: %s\n", l.token.toChars());
    }
    while (l.nextToken() != TOKeof);
 }

 ======================

 Prints:

 token: void
 token: blah
 token: (
 token: )
 token: {
 token: }


 I saw that. I'm really looking forward to getting my teeth into 
 it and doing some good old refactoring. Although that will be a 
 while because of the auto generated thing.


I have a bit similar project - lexer for D written in D.
Written just based on "Lexical" documentation page.

https://github.com/buggins/ddc

Trying make it fast and to do as few memory allocations as 
possible.
Jan 06 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/6/2015 1:37 AM, Vadim Lopatin wrote:

 I have a bit similar project - lexer for D written in D.
 Written just based on "Lexical" documentation page.

 https://github.com/buggins/ddc

 Trying make it fast and to do as few memory allocations as possible.


Should also make it available on code.dlang.org
Jan 06 2015
prev sibling next sibling parent reply "Kiith-Sa" <kiithsacmp gmail.com> writes:
On Sunday, 4 January 2015 at 13:07:34 UTC, Daniel Murphy wrote:

 I've created a dub package for the D version of DMD's lexer, 
 generated automatically from the C++ source.

 github: https://github.com/yebblies/ddmd
 dub: http://code.dlang.org/packages/ddmd

 There are a few annoying limitations, such that it uses dmd's 
 error printing and allocation functions, and requires 
 configuration through 'global'.

 Here is an example program that uses the lexer:

 ======================

 import std.stdio;
 import std.file;

 import ddmd.tokens;
 import ddmd.lexer;

 /////////////////////////

 void main()
 {
    Lexer.initLexer();

    string data = "void blah() {} // stuff";
    auto l = new Lexer("myfile", data.ptr, 0, data.length, 0, 0);
    l.nextToken();
    do
    {
        printf("token: %s\n", l.token.toChars());
    }
    while (l.nextToken() != TOKeof);
 }

 ======================

 Prints:

 token: void
 token: blah
 token: (
 token: )
 token: {
 token: }


(sorry if you get this question too often)

How is DDMD as a whole going? Is it getting closer or are ongoing 
DMD changes slowing it down too much?

Anyway, great to have the lexer on DUB, not going to use if for 
now because I expect there to be changes, but I guess it may 
eventually be useful for writing tools.
Jan 04 2015
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Kiith-Sa"  wrote in message news:nffxogzwpmayydyomaju forum.dlang.org...


 (sorry if you get this question too often)

 How is DDMD as a whole going? Is it getting closer or are ongoing DMD 
 changes slowing it down too much?


It's been sitting still for 8 nearly months because of 
https://github.com/braddr/d-tester/issues/39 and 
https://github.com/braddr/d-tester/issues/40

I don't mind getting asked, I just wish the answer would change.


 Anyway, great to have the lexer on DUB, not going to use if for now 
 because I expect there to be changes, but I guess it may eventually be 
 useful for writing tools.


Hopefully lots of changes.
Jan 04 2015
parent "Kiith-Sa" <kiithsacmp gmail.com> writes:
On Sunday, 4 January 2015 at 17:27:57 UTC, Daniel Murphy wrote:

 "Kiith-Sa"  wrote in message 
 news:nffxogzwpmayydyomaju forum.dlang.org...


 (sorry if you get this question too often)

 How is DDMD as a whole going? Is it getting closer or are 
 ongoing DMD changes slowing it down too much?


 It's been sitting still for 8 nearly months because of 
 https://github.com/braddr/d-tester/issues/39 and 
 https://github.com/braddr/d-tester/issues/40

 I don't mind getting asked, I just wish the answer would change.


 Anyway, great to have the lexer on DUB, not going to use if 
 for now because I expect there to be changes, but I guess it 
 may eventually be useful for writing tools.


 Hopefully lots of changes.


I can't really help with those, but FWIW I bumped the issues.
Jan 04 2015
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-01-04 14:07, Daniel Murphy wrote:

 I've created a dub package for the D version of DMD's lexer, generated
 automatically from the C++ source.


This is very cool :), keep up to good work.

-- 
/Jacob Carlborg
Jan 04 2015
prev sibling next sibling parent reply "Laeeth Isharc" <laeethnospam spammenot_laeeth.com> writes:
 I've created a dub package for the D version of DMD's lexer, 
 generated automatically from the C++ source.


very cool.

on a related note, have you considered sharing your translation 
tool (c++ -> D)?  I completely understand if you would rather not 
of course.



Laeeth.
Jan 04 2015
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Laeeth Isharc"  wrote in message 
news:yzmwemaevaltcmkywmgk forum.dlang.org...


 on a related note, have you considered sharing your translation tool 
 (c++ -> D)?  I completely understand if you would rather not of course.


The translation tool is available on github and is boost licensed.

This pull request contains the latest version in src/magicport - 
https://github.com/D-Programming-Language/dmd/pull/3410

Please keep in mind that the tool makes a lot of assumptions about the C++ 
source that may not be valid for projects other than dmd.  It's fairly easy 
to adapt to other projects, but it won't work on them out of the box.
Jan 04 2015
parent reply "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
On Monday, 5 January 2015 at 02:51:17 UTC, Daniel Murphy wrote:

 "Laeeth Isharc"  wrote in message 
 news:yzmwemaevaltcmkywmgk forum.dlang.org...


 on a related note, have you considered sharing your 
 translation tool (c++ -> D)?  I completely understand if you 
 would rather not of course.


 The translation tool is available on github and is boost 
 licensed.

 This pull request contains the latest version in src/magicport 
 - https://github.com/D-Programming-Language/dmd/pull/3410

 Please keep in mind that the tool makes a lot of assumptions 
 about the C++ source that may not be valid for projects other 
 than dmd.  It's fairly easy to adapt to other projects, but it 
 won't work on them out of the box.


Thanks v much - this will be very helpful indeed.
Jan 04 2015
parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Laeeth Isharc"  wrote in message 
news:vtgirvyjsalkzjvlzdws forum.dlang.org...


 Thanks v much - this will be very helpful indeed.


Let me know if you have any questions about it.
Jan 04 2015
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/4/2015 5:07 AM, Daniel Murphy wrote:

 I've created a dub package for the D version of DMD's lexer, generated
 automatically from the C++ source.

 github: https://github.com/yebblies/ddmd
 dub: http://code.dlang.org/packages/ddmd


Great! Thank you!
Jan 04 2015
prev sibling next sibling parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Daniel Murphy"  wrote in message news:m8bdul$1dke$1 digitalmars.com... 


 I've created a dub package for the D version of DMD's lexer, generated 
 automatically from the C++ source.
 
 github: https://github.com/yebblies/ddmd
 dub: http://code.dlang.org/packages/ddmd


I've pushed a new version which should fix the 64-bit build errors.
Jan 04 2015
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
It will be really cool when same package will be reused by DMD 
itself :P
Jan 06 2015
parent "Joakim" <dlang joakim.fea.st> writes:
On Tuesday, 6 January 2015 at 14:38:21 UTC, Dicebot wrote:

 It will be really cool when same package will be reused by DMD 
 itself :P


I believe ddmd has passed all tests on most platforms for a long 
time now, so there is nothing stopping those building from source 
from using ddmd now. :)
Jan 06 2015