www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - So... let's document dmd

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
https://github.com/D-Programming-Language/dlang.org/pull/1196
https://github.com/D-Programming-Language/dmd/pull/5352

Destroy!!

Andrei
Jan 14 2016
next sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Friday, 15 January 2016 at 04:53:16 UTC, Andrei Alexandrescu 
wrote:

 https://github.com/D-Programming-Language/dlang.org/pull/1196
 https://github.com/D-Programming-Language/dmd/pull/5352

 Destroy!!

 Andrei


Since dmd is now in D, is there any reason to not allow people to 
import from it while using dmd? Wouldn't it make anyone's life 
who wanted to parse D code easier by just doing

import ddmd.parser; (or what ever it is)

and use the actual parser?
Jan 15 2016
parent reply Joakim <dlang joakim.fea.st> writes:
On Friday, 15 January 2016 at 23:58:32 UTC, Jack Stouffer wrote:

 On Friday, 15 January 2016 at 04:53:16 UTC, Andrei Alexandrescu 
 wrote:

 https://github.com/D-Programming-Language/dlang.org/pull/1196
 https://github.com/D-Programming-Language/dmd/pull/5352

 Destroy!!

 Andrei


 Since dmd is now in D, is there any reason to not allow people 
 to import from it while using dmd? Wouldn't it make anyone's 
 life who wanted to parse D code easier by just doing

 import ddmd.parser; (or what ever it is)

 and use the actual parser?


The ddmd lexer is already on dub, just have to add the parser 
next:

http://code.dlang.org/packages/ddmd
Jan 15 2016
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Saturday, 16 January 2016 at 05:11:57 UTC, Joakim wrote:

 The ddmd lexer is already on dub, just have to add the parser 
 next:

 http://code.dlang.org/packages/ddmd


You missed the main point of my question.
Jan 15 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Saturday, 16 January 2016 at 05:29:03 UTC, Jack Stouffer wrote:

 On Saturday, 16 January 2016 at 05:11:57 UTC, Joakim wrote:

 The ddmd lexer is already on dub, just have to add the parser 
 next:

 http://code.dlang.org/packages/ddmd


 You missed the main point of my question.


I'm confused as to your point as well. Perhaps you could rephrase?
Jan 15 2016
next sibling parent Joakim <dlang joakim.fea.st> writes:
On Saturday, 16 January 2016 at 05:36:23 UTC, tsbockman wrote:

 On Saturday, 16 January 2016 at 05:29:03 UTC, Jack Stouffer 
 wrote:

 On Saturday, 16 January 2016 at 05:11:57 UTC, Joakim wrote:

 The ddmd lexer is already on dub, just have to add the parser 
 next:

 http://code.dlang.org/packages/ddmd


 You missed the main point of my question.


 I'm confused as to your point as well. Perhaps you could 
 rephrase?


I think he's trying to emphasize that he'd like it to come with 
dmd.  There was talk of putting the lexer/parser into phobos, but 
I believe dub was chosen as a staging ground for people to play 
with it first.  So if you want to use the lexer/parser, use it 
from dub for now, then push for its inclusion into phobos later.
Jan 15 2016
prev sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Saturday, 16 January 2016 at 05:36:23 UTC, tsbockman wrote:

 On Saturday, 16 January 2016 at 05:29:03 UTC, Jack Stouffer 
 wrote:

 On Saturday, 16 January 2016 at 05:11:57 UTC, Joakim wrote:

 The ddmd lexer is already on dub, just have to add the parser 
 next:

 http://code.dlang.org/packages/ddmd


 You missed the main point of my question.


 I'm confused as to your point as well. Perhaps you could 
 rephrase?


dmd is now in D; theoretically that should allow for other 
projects to import from it like a normal D project. So why not 
make all of the ddmd modules available from any code that is 
complied with it, i.e. just like Phobos?

The parser was just an example, there are any number of things 
that one could use from the code base.
Jan 15 2016
parent reply tsbockman <thomas.bockman gmail.com> writes:
On Saturday, 16 January 2016 at 07:52:11 UTC, Jack Stouffer wrote:

 dmd is now in D; theoretically that should allow for other 
 projects to import from it like a normal D project. So why not 
 make all of the ddmd modules available from any code that is 
 complied with it, i.e. just like Phobos?

 The parser was just an example, there are any number of things 
 that one could use from the code base.


That would be cool in theory, but it would likely become a major 
drag on development of the compiler itself.

Once you start promoting the compiler's internals as part of the 
standard library, you can expect sooner or later that pressure 
will mount to bring them up to the same high standards of API 
design and stability as Phobos.

While this process would be beneficial to the overall quality of 
the DMD code being exposed, it would also turn internal 
refactorings into public breaking changes. I really doubt that's 
a good trade-off at this point in DMD's development.
Jan 16 2016
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, Jan 16, 2016 at 08:35:40AM +0000, tsbockman via Digitalmars-d wrote:

 On Saturday, 16 January 2016 at 07:52:11 UTC, Jack Stouffer wrote:

dmd is now in D; theoretically that should allow for other projects
to import from it like a normal D project. So why not make all of the
ddmd modules available from any code that is complied with it, i.e.
just like Phobos?

The parser was just an example, there are any number of things that
one could use from the code base.


 
 That would be cool in theory, but it would likely become a major drag
 on development of the compiler itself.
 
 Once you start promoting the compiler's internals as part of the
 standard library, you can expect sooner or later that pressure will
 mount to bring them up to the same high standards of API design and
 stability as Phobos.
 
 While this process would be beneficial to the overall quality of the
 DMD code being exposed, it would also turn internal refactorings into
 public breaking changes. I really doubt that's a good trade-off at
 this point in DMD's development.


I disagree. I think having the dmd itself (lexer, parser, etc.) as a
library (with the dmd executable merely being the default frontend) will
do D a lot of good.

For one thing, IDE's will no longer need to reinvent a D parser for the
purposes of syntax highlighting; they can reuse the actual parser the
compiler itself uses, and thus actually be correct (whereas their own
parser may actually only parse a subset of D correctly).  They will also
be able to support instant feedback as the user types in D code
(highlight syntax errors on the fly, etc.).

This will also help with writing various tools that operate on D code,
e.g., pretty-printers like dfmt won't need to roll its own lexer/parser
anymore. Many other tools also become possible (automatic refactorings,
function call tree analysis, dependency analysis, etc.).

You could even use libdmd in your own D code, thus being able to have
the equivalent of the `eval` available in many scripting languages
(except that it will have native performance(!)).

The problem of dmd refactorings becoming public breaking changes is not
as big a problem if the public API is properly designed. If the compiler
code itself is refactored to interface between its sub-components via
cleanly-designed APIs with proper encapsulations, I don't see that it
will become prohibitively difficult to minimize public API changes
without hampering compiler development efforts. This is the way to go
for better compiler code quality anyway, so even if libdmd never
happens, the code should *still* be refactored in this way.

Besides, I don't think anyone is sanely suggesting that we allow dmd
source files to be free-for-all externally-importable modules. The more
likely scenario is that there will be a clearly-defined external
interface that is unlikely to change much, whereas the implementation
underneath is still free to be changed, refactored, reorganized,
rewritten, etc..


T

-- 
BREAKFAST.COM halted...Cereal Port Not Responding. -- YHL
Jan 16 2016
next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Saturday, 16 January 2016 at 15:13:28 UTC, H. S. Teoh wrote:

 On Sat, Jan 16, 2016 at 08:35:40AM +0000, tsbockman via 
 Digitalmars-d wrote:

 On Saturday, 16 January 2016 at 07:52:11 UTC, Jack Stouffer 
 wrote:

dmd is now in D; theoretically that should allow for other 
projects to import from it like a normal D project. So why 
not make all of the ddmd modules available from any code that 
is complied with it, i.e. just like Phobos?

The parser was just an example, there are any number of 
things that one could use from the code base.


 
 That would be cool in theory, but it would likely become a 
 major drag on development of the compiler itself.
 
 Once you start promoting the compiler's internals as part of 
 the standard library, you can expect sooner or later that 
 pressure will mount to bring them up to the same high 
 standards of API design and stability as Phobos.
 
 While this process would be beneficial to the overall quality 
 of the DMD code being exposed, it would also turn internal 
 refactorings into public breaking changes. I really doubt 
 that's a good trade-off at this point in DMD's development.


 I disagree. I think having the dmd itself (lexer, parser, etc.) 
 as a
 library (with the dmd executable merely being the default 
 frontend) will
 do D a lot of good.

 For one thing, IDE's will no longer need to reinvent a D parser 
 for the purposes of syntax highlighting; they can reuse the 
 actual parser the compiler itself uses, and thus actually be 
 correct (whereas their own parser may actually only parse a 
 subset of D correctly).  They will also be able to support 
 instant feedback as the user types in D code (highlight syntax 
 errors on the fly, etc.).

 This will also help with writing various tools that operate on 
 D code, e.g., pretty-printers like dfmt won't need to roll its 
 own lexer/parser anymore. Many other tools also become possible 
 (automatic refactorings, function call tree analysis, 
 dependency analysis, etc.).

 You could even use libdmd in your own D code, thus being able 
 to have
 the equivalent of the `eval` available in many scripting 
 languages
 (except that it will have native performance(!)).

 The problem of dmd refactorings becoming public breaking 
 changes is not as big a problem if the public API is properly 
 designed. If the compiler code itself is refactored to 
 interface between its sub-components via cleanly-designed APIs 
 with proper encapsulations, I don't see that it will become 
 prohibitively difficult to minimize public API changes without 
 hampering compiler development efforts. This is the way to go 
 for better compiler code quality anyway, so even if libdmd 
 never happens, the code should *still* be refactored in this 
 way.

 Besides, I don't think anyone is sanely suggesting that we 
 allow dmd source files to be free-for-all externally-importable 
 modules. The more likely scenario is that there will be a 
 clearly-defined external interface that is unlikely to change 
 much, whereas the implementation underneath is still free to be 
 changed, refactored, reorganized, rewritten, etc..


While this would all be nice in principle, the reality is that 
dmd is mostly worked on by two people these days 
(https://github.com/D-Programming-Language/dmd/graphs/contributors) and they're
unlikely to refactor ddmd to put forth a libdmd.  It will depend on someone
caring enough to take it up, as Daniel did with the port of the frontend to D. 
As tsbockman said, I don't think we're at the stage where anybody will put in
that much effort into this.
Jan 16 2016
parent Daniel Murphy <yebbliesnospam gmail.com> writes:
On 17/01/2016 2:29 AM, Joakim wrote:

 While this would all be nice in principle, the reality is that dmd is
 mostly worked on by two people these days
 (https://github.com/D-Programming-Language/dmd/graphs/contributors) and
 they're unlikely to refactor ddmd to put forth a libdmd.  It will depend
 on someone caring enough to take it up, as Daniel did with the port of
 the frontend to D.  As tsbockman said, I don't think we're at the stage
 where anybody will put in that much effort into this.


I've already spent a huge amount of time on refactoring the frontend, 
there's just so much more to do.  More important than allowing new 
third-party uses of the frontend, is that well defined interfaces will 
make maintaining GDC and LDC less tricky.  Having to maintain a stable 
API would probably hurt more than it helps at this point.
Jan 16 2016
prev sibling next sibling parent tsbockman <thomas.bockman gmail.com> writes:
On Saturday, 16 January 2016 at 15:13:28 UTC, H. S. Teoh wrote:

 I disagree. I think having the dmd itself (lexer, parser, etc.) 
 as a
 library (with the dmd executable merely being the default 
 frontend) will
 do D a lot of good.

 For one thing, IDE's will no longer need to reinvent a D parser 
 for the purposes of syntax highlighting; they can reuse the 
 actual parser the compiler itself uses, and thus actually be 
 correct (whereas their own parser may actually only parse a 
 subset of D correctly).  They will also be able to support 
 instant feedback as the user types in D code (highlight syntax 
 errors on the fly, etc.).

 This will also help with writing various tools that operate on 
 D code, e.g., pretty-printers like dfmt won't need to roll its 
 own lexer/parser anymore. Many other tools also become possible 
 (automatic refactorings, function call tree analysis, 
 dependency analysis, etc.).

 You could even use libdmd in your own D code, thus being able 
 to have
 the equivalent of the `eval` available in many scripting 
 languages
 (except that it will have native performance(!)).

 The problem of dmd refactorings becoming public breaking 
 changes is not as big a problem if the public API is properly 
 designed. If the compiler code itself is refactored to 
 interface between its sub-components via cleanly-designed APIs 
 with proper encapsulations, I don't see that it will become 
 prohibitively difficult to minimize public API changes without 
 hampering compiler development efforts. This is the way to go 
 for better compiler code quality anyway, so even if libdmd 
 never happens, the code should *still* be refactored in this 
 way.

 Besides, I don't think anyone is sanely suggesting that we 
 allow dmd source files to be free-for-all externally-importable 
 modules. The more likely scenario is that there will be a 
 clearly-defined external interface that is unlikely to change 
 much, whereas the implementation underneath is still free to be 
 changed, refactored, reorganized, rewritten, etc..


 T


You're sort of making my point for me.

I fully agree about all the benefits of making the lexer, parser, 
etc. available as a library.

HOWEVER, as you say, this would first require a large refactoring 
of the code, to make it suitable for use as a library. Time spent 
on that refactoring is time that could have been spent improving 
the compiler in other ways.

D is short of development resources as it is; I think fixing the 
many bugs in the compiler and fleshing out important complex 
features like C++ inter-op is of more value to the language in 
the long run.

Also worth mentioning, is that the SDC project was actually 
designed from the beginning for the use you're advocating. I'm 
sure deadalnix would love to have more help with that...
Jan 16 2016
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/16/2016 7:13 AM, H. S. Teoh via Digitalmars-d wrote:

 I disagree. I think having the dmd itself (lexer, parser, etc.) as a
 library (with the dmd executable merely being the default frontend) will
 do D a lot of good.

 For one thing, IDE's will no longer need to reinvent a D parser for the
 purposes of syntax highlighting;


On the other hand, using lexer.d and parse.d as a guide to build your own is a 
trivial undertaking. The Boost license is designed so this can be done without 
worrying about making a derived work.

I looked into doing syntax highlighting for my editor, MicroEmacs. It turns out 
it is not so easy to just use a compiler lexer/parser for it. For one thing,
the 
one used in the compiler is optimized for speed in a forward pass through the
text.

But a syntax highlighter in a text editor is different. Suppose I change a 
character in the middle of a line. All the highlighting from that point forward 
may change. And to figure out what that change is, the parser/lexer has to
start 
over from the beginning of the file! (Think string literals, nested comments, 
quoted string literals, etc.) This would make editing slow.

I surmised that a solution is to have each line in the editor be tagged with a 
state to show the lexing state at the beginning of that line. Then, when a 
character in a line changes, the lexer can be restarted from that state, and it 
continues forward until the next line state matches the new computed state.
This 
would make it enormously faster.

But the compiler's lexer is not designed to be restartable in the middle.

Similarly, a source code formatter would be different in a different way. It 
would need, for example, extra information about comments, token start/end 
positions, etc. Such data collection would be irrelevant to a compiler, and 
would slow it down and consume unneeded memory.
Apr 05 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Tuesday, 5 April 2016 at 08:46:30 UTC, Walter Bright wrote:

 On 1/16/2016 7:13 AM, H. S. Teoh via Digitalmars-d wrote:

 I disagree. I think having the dmd itself (lexer, parser, 
 etc.) as a
 library (with the dmd executable merely being the default 
 frontend) will
 do D a lot of good.

 For one thing, IDE's will no longer need to reinvent a D 
 parser for the
 purposes of syntax highlighting;


 On the other hand, using lexer.d and parse.d as a guide to 
 build your own is a trivial undertaking. The Boost license is 
 designed so this can be done without worrying about making a 
 derived work.

 I looked into doing syntax highlighting for my editor, 
 MicroEmacs. It turns out it is not so easy to just use a 
 compiler lexer/parser for it. For one thing, the one used in 
 the compiler is optimized for speed in a forward pass through 
 the text.

 But a syntax highlighter in a text editor is different. Suppose 
 I change a character in the middle of a line. All the 
 highlighting from that point forward may change. And to figure 
 out what that change is, the parser/lexer has to start over 
 from the beginning of the file! (Think string literals, nested 
 comments, quoted string literals, etc.) This would make editing 
 slow.


This is how CE highlither works. The lexer used to highlight 
processes line by line. For each line infos about the previous 
line are available (nested comments count, region kind like 
quoted string, raw quoted string, asm, etc).

Also keyword detection might use different dictionaries, up to 3 
(one for the keywords, another for special keywords like 
__FILE__, a third for asm opcodes). Operators doesn't need to 
have a special token (tokxorequ, toplusplus, etc) there's just 
one. Also lexing number doesn't need to be as accurate as the 
front-end of the compiler (especially if the HL doesnt have a 
token type for the illegal "lexem".

Another big difference is that the lexer used by an highlighter 
doesn't store the identifier associated to a token.

Using the front-end (or even libdparse) would require a 
modularisation. But lexing is not hard so I don't think it's 
worse. At last it would only be used by 4 or 5 softwares.
Apr 05 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/5/2016 6:47 AM, Basile B. wrote:

 Also lexing number doesn't need to be as accurate as the
 front-end of the compiler (especially if the HL doesnt have a token type for
the
 illegal "lexem".


That is an interesting design point. If I was doing a highlighter, I'd
highlight 
in red tokens that the compiler would reject, meaning I'd do the accurate
number 
lexing.

Lexing numbers correctly is not trivial, but since the compiler lexer's 
implementation can be cut/pasted, it is trivial in practice.
Apr 05 2016
parent reply Basile B. <b2.temp gmx.com> writes:
On Tuesday, 5 April 2016 at 21:37:09 UTC, Walter Bright wrote:

 On 4/5/2016 6:47 AM, Basile B. wrote:

 Also lexing number doesn't need to be as accurate as the
 front-end of the compiler (especially if the HL doesnt have a 
 token type for the
 illegal "lexem".


 That is an interesting design point. If I was doing a 
 highlighter, I'd highlight in red tokens that the compiler 
 would reject, meaning I'd do the accurate number lexing.

 Lexing numbers correctly is not trivial, but since the compiler 
 lexer's implementation can be cut/pasted, it is trivial in 
 practice.


Even if when the most naive lexer see a number and consumes until 
a blank, a symbol or an operator, it's clear that this can be 
done:

http://i.imgur.com/ehjps04.png

Actually numbers is the only part of the D lexer where errors can 
be detected.
There's no possible syntax errors otherwise.

But one thing I forget to say in my previous post is that lexing 
can be "multi-pass". The D front-end does everything in a single 
pass, for example it direclty detects tokPlusPlus or tokXorEqu, 
but actually a multi pass lexer can work in 3 sub phases:
1/ split words
2/ detects token families in the words; identifier, keyword, 
operator, etc.
3/ specialize tokens: tokOp.data == "++" -> tokPlusPlus
Apr 05 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/5/2016 7:59 PM, Basile B. wrote:

 Actually numbers is the only part of the D lexer where errors can be detected.
 There's no possible syntax errors otherwise.


D's lexer.d detects many errors. Search lexer.d for "error".
Apr 05 2016
prev sibling parent Basile B. <b2.temp gmx.com> writes:
On Tuesday, 5 April 2016 at 13:47:55 UTC, Basile B. wrote:

 [...]
 by 4 or 5 softwares.


Believe or not, in 2012 I've registered a company with the same 
error (<stuff>.softwares).
I don't know why...it hasnot been a commercial success.
Apr 05 2016
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 05.04.2016 10:46, Walter Bright wrote:

 On 1/16/2016 7:13 AM, H. S. Teoh via Digitalmars-d wrote:

 I disagree. I think having the dmd itself (lexer, parser, etc.) as a
 library (with the dmd executable merely being the default frontend) will
 do D a lot of good.

 For one thing, IDE's will no longer need to reinvent a D parser for the
 purposes of syntax highlighting;


 On the other hand, using lexer.d and parse.d as a guide to build your
 own is a trivial undertaking. The Boost license is designed so this can
 be done without worrying about making a derived work.

 I looked into doing syntax highlighting for my editor, MicroEmacs. It
 turns out it is not so easy to just use a compiler lexer/parser for it.
 For one thing, the one used in the compiler is optimized for speed in a
 forward pass through the text.

 But a syntax highlighter in a text editor is different. Suppose I change
 a character in the middle of a line. All the highlighting from that
 point forward may change. And to figure out what that change is, the
 parser/lexer has to start over from the beginning of the file! (Think
 string literals, nested comments, quoted string literals, etc.) This
 would make editing slow.
 ...


Also, tools might want to parse a more intuitive grammar in order to be 
able to give suggestions on how to adapt the code such that DMDs parser 
will accept it.


 I surmised that a solution is to have each line in the editor be tagged
 with a state to show the lexing state at the beginning of that line.
 Then, when a character in a line changes, the lexer can be restarted
 from that state, and it continues forward until the next line state
 matches the new computed state. This would make it enormously faster.
 ...


Some additional care will need to be taken, e.g.

/+ <backspace> <backspace>

or even simply

/+ comment +/


 But the compiler's lexer is not designed to be restartable in the middle.

 Similarly, a source code formatter would be different in a different
 way. It would need, for example, extra information about comments, token
 start/end positions, etc. Such data collection would be irrelevant to a
 compiler, and would slow it down and consume unneeded memory.
Apr 05 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/5/2016 4:08 PM, Timon Gehr wrote:

 Some additional care will need to be taken, e.g.

 /+ <backspace> <backspace>

 or even simply

 /+ comment +/


that should not be a problem.
Apr 05 2016
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 06.04.2016 06:16, Walter Bright wrote:

 On 4/5/2016 4:08 PM, Timon Gehr wrote:

 Some additional care will need to be taken, e.g.

 /+ <backspace> <backspace>

 or even simply

 /+ comment +/


 that should not be a problem.


I think it can be for large files if the highlighter implements the 
described strategy in a literal fashion. /+ changes the lexer state at 
each position until the end of the file, and all the soon-to-be reusable 
information is replaced by irrelevant garbage by the reprocessing step. 
I think it is better to sometimes keep multiple lexer states per 
location, and maybe not necessarily run the lexer to completion each time.

Maybe this is one reason why some editors choose to close delimiters for 
you automatically by default.


One idea would be to handle all states simultaneously by keeping an 
efficient summary of the effect certain existing code snippets have on 
the lexer state. Summaries should be efficiently composable to compute 
effects (and hence the lexer state) quickly for any prefix of the code, 
while allowing code updates. I think it should be possible to fill in 
the details.
Apr 06 2016
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-01-16 16:13, H. S. Teoh via Digitalmars-d wrote:


 I disagree. I think having the dmd itself (lexer, parser, etc.) as a
 library (with the dmd executable merely being the default frontend) will
 do D a lot of good.

 For one thing, IDE's will no longer need to reinvent a D parser for the
 purposes of syntax highlighting; they can reuse the actual parser the
 compiler itself uses, and thus actually be correct (whereas their own
 parser may actually only parse a subset of D correctly).  They will also
 be able to support instant feedback as the user types in D code
 (highlight syntax errors on the fly, etc.).


I agree very much with this. Unfortunately many (most?) editors use 
their own weird syntax to describe grammars for syntax highlighting and 
cannot use a proper lexer/parser.

-- 
/Jacob Carlborg
Apr 05 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/5/2016 4:34 AM, Jacob Carlborg wrote:

 I agree very much with this. Unfortunately many (most?) editors use their own
 weird syntax to describe grammars for syntax highlighting and cannot use a
 proper lexer/parser.



Haha, MicroEmacs is written in D, so just use D code :-)

I don't really understand why IDE makers don't use an actual programming 
language for plugins - I'd use javascript as there's a D implementation of a 
javascript engine you can use. (Using D as a plugin language would require 
making DLLs which is a pain or making a D interpreter, which would be a major 
project (CTFE in theory should work for that, but it consumes too much memory).)
Apr 05 2016
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-04-05 23:40, Walter Bright wrote:


 Haha, MicroEmacs is written in D, so just use D code :-)

 I don't really understand why IDE makers don't use an actual programming
 language for plugins - I'd use javascript as there's a D implementation
 of a javascript engine you can use. (Using D as a plugin language would
 require making DLLs which is a pain or making a D interpreter, which
 would be a major project (CTFE in theory should work for that, but it
 consumes too much memory).)


TextMate, Sublime and Atom all use the same syntax to describe the 
grammar for a language. All of them supports plugins (to various 
degrees), but none of them uses plugins to syntax highlight code, as far 
as I know.

I guess it's easier for most of us, I know you write a lexer in two days 
:), to use the custom syntax to describe the grammar than to create a 
proper lexer and parser. Note that a lexer is not enough, these grammars 
can describe how a function (and other constructs) look like.

-- 
/Jacob Carlborg
Apr 06 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/6/2016 1:09 AM, Jacob Carlborg wrote:

 TextMate, Sublime and Atom all use the same syntax to describe the grammar for
a
 language. All of them supports plugins (to various degrees), but none of them
 uses plugins to syntax highlight code, as far as I know.

 I guess it's easier for most of us, I know you write a lexer in two days :), to
 use the custom syntax to describe the grammar than to create a proper lexer and
 parser.


I doubt any of them could lex C/C++ correctly (trigraphs, macros, backslash
line 
splicing, wysiwyg string literals).

Even D has some issues - token string literals - that would defeat most grammar 
attempts.

A straightforward lexer (i.e. one not heavily optimized for speed) written in a 
reasonable high level language does not look all that different from BNF, and 
has the advantage of not being borked if something is not quite expressible in 
the grammar language.



 Note that a lexer is not enough, these grammars can describe how a
 function (and other constructs) look like.


Are you talking about an AST?

(Good luck doing an AST for C++ without a real C++ compiler front end! Not that 
hard for D, though.)
Apr 06 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-04-07 02:53, Walter Bright wrote:


 I doubt any of them could lex C/C++ correctly (trigraphs, macros,
 backslash line splicing, wysiwyg string literals).


No. But it doesn't need to be perfect. There's also nothing that tells 
what's wrong and what's right. For example, in TextMate the identifier 
in a function declaration is syntax highlighted, but in Eclipse it's not 
syntax highlighted.


 Are you talking about an AST?

 (Good luck doing an AST for C++ without a real C++ compiler front end!
 Not that hard for D, though.)


Yes. In TextMate one describes a grammar and assigns different scopes to 
the different rules. These scopes are then used by the theme engine to 
syntax highlight the code. The scopes are also used to enable/disable 
different bundle commands. For example, pressing Ctrl+B inside a comment 
could make the selected text bold (replace the text with an HTML or 
Markdown tag) while outside comments something else happens.

TextMate also adds top level declarations like classes, structs and 
functions to a jump-list, for easy navigation.

Note that I'm not defending the editors, I would love if one could use a 
proper front end to do the lexing and parsing. I'm just saying it's not 
as easy as it sounded in the post I originally replied to.

-- 
/Jacob Carlborg
Apr 06 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/6/2016 11:36 PM, Jacob Carlborg wrote:

 On 2016-04-07 02:53, Walter Bright wrote:


 I doubt any of them could lex C/C++ correctly (trigraphs, macros,
 backslash line splicing, wysiwyg string literals).


 No. But it doesn't need to be perfect.


I suppose that being a compiler guy I wouldn't care to build a half-assed one. 
It's not *that* hard to get it perfect, but you will need to use a programming 
language to do it.
Apr 07 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-04-07 09:28, Walter Bright wrote:


 I suppose that being a compiler guy I wouldn't care to build a
 half-assed one. It's not *that* hard to get it perfect, but you will
 need to use a programming language to do it.


I usually feel the same, but that would require me modify the editor. 
I've many times thought about create a plugin which uses a proper lexer 
and parser but it's a lot of work for something that doesn't have a 
plugin API.

Also, GitHub is using the TextMate grammar to do syntax highlighting for 
D. If TextMate used a plugin for this instead, there would most likely 
be no one maintaining the the grammar GitHub is using.

-- 
/Jacob Carlborg
Apr 07 2016
prev sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 04/05/2016 05:40 PM, Walter Bright wrote:

 I don't really understand why IDE makers don't use an actual programming
 language for plugins


Programmer's Notepad 2 uses Python for plugins. Although the syntax 
highlighting is done through Scintilla which has it's own separate 
(non-programming-language IIRC) system for lex/parse.
Apr 07 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 01/14/2016 11:53 PM, Andrei Alexandrescu wrote:

 https://github.com/D-Programming-Language/dlang.org/pull/1196
 https://github.com/D-Programming-Language/dmd/pull/5352

 Destroy!!

 Andrei


Finally got around to updating this. To make it work, please review and 
pull https://github.com/D-Programming-Language/dmd/pull/5617 and then 
https://github.com/D-Programming-Language/dlang.org/pull/1196.

BTW I'd forgotten we set up undefined macros to emit <!--UNDEFINED 
MACRO: "Xyz"--> in the generated html. That has been of great help.


Andrei
Apr 01 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Alright, we're on! Entry point ain't sexy and the pages need all work, 
but it's a start. http://dlang.org/dmd-prerelease/ -- Andrei
Apr 01 2016