www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [gsoc] D Language Server protocol based on the DMD frontend

reply Seb <seb wilzba.ch> writes:
Another thread on a project from the GSoC wiki to summarize all 
information, spark discussions and inspire students.

The excerpt of this project from the wiki:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A Language Server is meant to provide the language-specific 
smarts and communicate with development tools over a protocol 
that enables inter-process communication.
The Language Server Protocol [1] (LSP) defines the protocol used 
between an editor or IDE and a language server that provides 
language features like auto complete, go to definition, find all 
references etc.

Adding features like auto complete, go to definition, or 
documentation on hover for a programming language takes 
significant effort. Traditionally this work had to be repeated 
for each development tool, as each tool provides different APIs 
for implementing the same feature (see the list of Editors [2] 
and IDEs [3]). Additionally as the existing tools use `libdparse` 
(used in DLS [4] and DCD [5]), they lack the ability to perform 
semantic analysis on the AST and can't evaluate `mixin`'s nor D's 
very powerful Compile-Time Function Evaluation (CTFE).

The idea behind the Language Server Protocol (LSP) is to 
standardize the protocol for how such servers and development 
tools communicate. This way, a single D Language Server can be 
re-used in multiple development tools, which in turn can support 
multiple languages with minimal effort.

Recent development in the DMD compiler made it available as a DUB 
library [7]. This mean that the compiler itself can be re-used 
and third-party tools don't have to re-implement e.g. D's complex 
CTFE. This project is about using the DMD compiler as a library 
to implement a Language Server for D.

How to start:

* Simple example of using DMD as a library [8]
* More complex use cases [9]

[1] 
https://wiki.dlang.org/GSOC_2019_Ideas#Language_Server_Protocol_for_D
[2] https://wiki.dlang.org/Editors
[3] https://wiki.dlang.org/IDEs
[4] https://microsoft.github.io/language-server-protocol
[5] https://github.com/d-language-server/dls
[6] https://github.com/dlang-community/DCD
[7] 
https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library
[8] 
https://github.com/dlang/dmd/blob/master/test/dub_package/frontend.d
[9] https://github.com/jacob-carlborg/dlp

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I think the project itself contains of two parts:

1) Improving DMD as a Library

We are still actively working on this, so expect that you might 
need to dive into DMD and refactor things.
Here's a recent example:

https://github.com/dlang/dmd/pull/9350

2) Exposing the DMD as a language server

You probably can get a many inspirations here from the existing 
tools:

- https://github.com/d-language-server/dls
- https://github.com/dlang-community/DCD


How to start
------------

I think the most crucial step is to familiarize yourself with the 
DMD codebase. It's very non-trivial and I think the best way is 
to start fixing easier DMD bugs.
Often ICEs are good for beginners as they already point you to 
the rough location which most likely needs fixing.

A more concrete suggestion from Jacob
-------------------------------------

When it comes to improving DMD as a library, I think one of the 
key things would be to reduce global state. When the compiler is 
used as a library it needs to be able to invoke the library 
multiple times without changing the outcome. Not be dependent of 
previous state. We have a few workarounds , like [1], to reset 
the state, but it would be much better if there wasn’t any global 
state to begin with.

A related thing is that all diagnostic handling (after the 
parsing phase) goes through a global function which always 
outputs to stderr. This makes it basically impossible to 
configure the diagnostic handling which is useful when using the 
compiler as a library.

Using the `pure` function attribute helps to identify global 
state. My tool, DLP [2], allows to identify leaf functions 
(functions that don’t call other functions), which would be a 
good start to add attributes on. My tool can also serve as an 
example of using the DMD as a library.

Perhaps adding to DLP, or writing a new tool, that can suggest 
attributes that can be added to functions could be a good start. 
It would both help by learning how to use DMD as a library and 
the tool can then be used to identify which functions can be 
`pure` and which can’t.


[1] 
https://github.com/dlang/dmd/blob/6e2ee54aad00ff0f22dd1f9139ecea230aa8426a/src/dmd/frontend.d#L81-L108
[2] http://github.com/jacob-carlborg/dlp

Community
---------

 community: what language smarts in an editor are the most 
important to you? What features do you miss in DLS/DCD?

 students: please don't be shy. This is on open discussion and 
your questions are welcome!
Mar 13 2019
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
These two bugs can be considered blockers (Win64):

https://issues.dlang.org/show_bug.cgi?id=18810
https://issues.dlang.org/show_bug.cgi?id=18811

The first shouldn't be too much work for a hacky solution. Port file to 
D, version as required and move to root/.
The second I haven't retested with the precise GC. It may solve it. 
Otherwise I have got no idea where to go from here.
Mar 13 2019
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2019-03-13 12:13, Seb wrote:


  community: what language smarts in an editor are the most important to 
 you? What features do you miss in DLS/DCD?


* Stability. DCD crashes quite often for me. Then there's this problem 
[1]. I have a locally patched version, which is required for me to even 
use DCD.

* I would really like to see the accuracy being improved. For example, 
autocompletion with support for UFCS.

* Not sure if this is supported by LSP but incremental compilation and 
reporting compile errors would be really cool.

* I'm not sure how feasible this would be but using Eclipse and Java 
autocompletion works even without any imports. When an autocompletion 
item is selected the IDE will automatically add the correct include.

For a bunch of more ideas, here's a list of supported features of an old 
Eclipse plugin, Descent, for D1 [2]:

* Syntax highlighting.
* Awesome, flexible formatter.
* Folding.
* Automatic closing of strings, brackets, chars and comments.
* Code templates.
* Syntax errors and some semantic errors are shown with a reg squiggle.
* Autocompletion which writes the imports for you.
* Source on hover (shift+hover).
* Go to definition (ctrl+hover).
* Some compile-time function evaluation (ctrl+shift+hover).
* Greying of inactive code.
* Project Explorer.
* Outline view.
* Type Hierarchy view.
* Automatic generation of documentation comments (Params, Author, etc.).
* Spell checking.
* TODO tasks.
* Ddoc view.
* Declaration (source) view.
* Highlighting of all occurrences of the current token under the cursor.
* trace.log viewer
* Compile-time debugging
* Runs and debugs programs within the IDE.

[1] https://github.com/dlang-community/DCD/issues/345
[2] http://www.dsource.org/projects/descent

-- 
/Jacob Carlborg
Mar 13 2019
prev sibling next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 13 March 2019 at 11:13:13 UTC, Seb wrote:

It's very non-trivial and I think the best
 way is to start fixing easier DMD bugs.
 Often ICEs are good for beginners as they already point you to


No. Common Seb. Most of the ICEs are in the backend or in CTFE.
Mar 13 2019
parent Mike Franklin <slavo5150 yahoo.com> writes:
On Wednesday, 13 March 2019 at 21:03:21 UTC, Basile B. wrote:

 On Wednesday, 13 March 2019 at 11:13:13 UTC, Seb wrote:

It's very non-trivial and I think the best
 way is to start fixing easier DMD bugs.
 Often ICEs are good for beginners as they already point you to


 No. Common Seb. Most of the ICEs are in the backend or in CTFE.


I find ICEs to be the easiest to fix because they are often (but 
not always) simply dereferencing a null, and sometimes (though 
not always) the fix is just adding a check for null or something 
of the like. Tracking them down is also easier because the stack 
trace tells you exactly where the problem occurs.  Other bugs are 
MUCH harder to track down.

So, I encourage those who want to get started hacking DMD to go 
for the ICEs.  Not all fit the description above, but I think 
they're good for beginners.

Mike
Mar 13 2019
prev sibling next sibling parent reply Rubn <where is.this> writes:
On Wednesday, 13 March 2019 at 11:13:13 UTC, Seb wrote:

 Another thread on a project from the GSoC wiki to summarize all 
 information, spark discussions and inspire students.

 The excerpt of this project from the wiki:

 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 A Language Server is meant to provide the language-specific 
 smarts and communicate with development tools over a protocol 
 that enables inter-process communication.
 The Language Server Protocol [1] (LSP) defines the protocol 
 used between an editor or IDE and a language server that 
 provides language features like auto complete, go to 
 definition, find all references etc.

 Adding features like auto complete, go to definition, or 
 documentation on hover for a programming language takes 
 significant effort. Traditionally this work had to be repeated 
 for each development tool, as each tool provides different APIs 
 for implementing the same feature (see the list of Editors [2] 
 and IDEs [3]). Additionally as the existing tools use 
 `libdparse` (used in DLS [4] and DCD [5]), they lack the 
 ability to perform semantic analysis on the AST and can't 
 evaluate `mixin`'s nor D's very powerful Compile-Time Function 
 Evaluation (CTFE).

 The idea behind the Language Server Protocol (LSP) is to 
 standardize the protocol for how such servers and development 
 tools communicate. This way, a single D Language Server can be 
 re-used in multiple development tools, which in turn can 
 support multiple languages with minimal effort.

 Recent development in the DMD compiler made it available as a 
 DUB library [7]. This mean that the compiler itself can be 
 re-used and third-party tools don't have to re-implement e.g. 
 D's complex CTFE. This project is about using the DMD compiler 
 as a library to implement a Language Server for D.

 How to start:

 * Simple example of using DMD as a library [8]
 * More complex use cases [9]

 [1] 
 https://wiki.dlang.org/GSOC_2019_Ideas#Language_Server_Protocol_for_D
 [2] https://wiki.dlang.org/Editors
 [3] https://wiki.dlang.org/IDEs
 [4] https://microsoft.github.io/language-server-protocol
 [5] https://github.com/d-language-server/dls
 [6] https://github.com/dlang-community/DCD
 [7] 
 https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library
 [8] 
 https://github.com/dlang/dmd/blob/master/test/dub_package/frontend.d
 [9] https://github.com/jacob-carlborg/dlp

 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 I think the project itself contains of two parts:

 1) Improving DMD as a Library

 We are still actively working on this, so expect that you might 
 need to dive into DMD and refactor things.
 Here's a recent example:

 https://github.com/dlang/dmd/pull/9350

 2) Exposing the DMD as a language server

 You probably can get a many inspirations here from the existing 
 tools:

 - https://github.com/d-language-server/dls
 - https://github.com/dlang-community/DCD


 How to start
 ------------

 I think the most crucial step is to familiarize yourself with 
 the DMD codebase. It's very non-trivial and I think the best 
 way is to start fixing easier DMD bugs.
 Often ICEs are good for beginners as they already point you to 
 the rough location which most likely needs fixing.

 A more concrete suggestion from Jacob
 -------------------------------------

 When it comes to improving DMD as a library, I think one of the 
 key things would be to reduce global state. When the compiler 
 is used as a library it needs to be able to invoke the library 
 multiple times without changing the outcome. Not be dependent 
 of previous state. We have a few workarounds , like [1], to 
 reset the state, but it would be much better if there wasn’t 
 any global state to begin with.

 A related thing is that all diagnostic handling (after the 
 parsing phase) goes through a global function which always 
 outputs to stderr. This makes it basically impossible to 
 configure the diagnostic handling which is useful when using 
 the compiler as a library.

 Using the `pure` function attribute helps to identify global 
 state. My tool, DLP [2], allows to identify leaf functions 
 (functions that don’t call other functions), which would be a 
 good start to add attributes on. My tool can also serve as an 
 example of using the DMD as a library.

 Perhaps adding to DLP, or writing a new tool, that can suggest 
 attributes that can be added to functions could be a good 
 start. It would both help by learning how to use DMD as a 
 library and the tool can then be used to identify which 
 functions can be `pure` and which can’t.


 [1] 
 https://github.com/dlang/dmd/blob/6e2ee54aad00ff0f22dd1f9139ecea230aa8426a/src/dmd/frontend.d#L81-L108
 [2] http://github.com/jacob-carlborg/dlp

 Community
 ---------

  community: what language smarts in an editor are the most 
 important to you? What features do you miss in DLS/DCD?

  students: please don't be shy. This is on open discussion and 
 your questions are welcome!


Using the actual DMD compiler may not be that great as a language 
server.

https://code.visualstudio.com/api/language-extensions/language-server-extension-guide#error-tolerant-parser-for-language-server

https://github.com/Microsoft/tolerant-php-parser/blob/master/docs/HowItWorks.md

It might be good to provide auto complete that is aware of things 
like mixins, but if you want language features when you are 
working on an incomplete incorrect source file. Not sure how good 
DMD will be able to handle that.

Whatever happened to the funding for code-d, did nothing ever 
come out of that?
Mar 13 2019
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-03-14 00:32, Rubn wrote:


 It might be good to provide auto complete that is aware of things like 
 mixins, but if you want language features when you are working on an 
 incomplete incorrect source file. Not sure how good DMD will be able to 
 handle that.


If that's required, then that should be part of the GSoC project.

-- 
/Jacob Carlborg
Mar 14 2019
parent reply Rubn <where is.this> writes:
On Thursday, 14 March 2019 at 20:03:29 UTC, Jacob Carlborg wrote:

 On 2019-03-14 00:32, Rubn wrote:


 It might be good to provide auto complete that is aware of 
 things like mixins, but if you want language features when you 
 are working on an incomplete incorrect source file. Not sure 
 how good DMD will be able to handle that.


 If that's required, then that should be part of the GSoC 
 project.


Which part? Point is using DMD as a language server isn't going 
to be intuitive, compilers are designed to fail on invalid input 
but editors need to work for incomplete source files. Either way 
I feel this is out of scope for a single student to complete in a 
single summer. Thinking that most of the work is done because 
they can use DMD is just backwards and haven't ever written 
tooling tools.
Mar 14 2019
parent reply Jacob Carlborg <doob me.com> writes:
On 2019-03-14 23:34, Rubn wrote:


 Which part? Point is using DMD as a language server isn't going to be 
 intuitive, compilers are designed to fail on invalid input but editors 
 need to work for incomplete source files.


Then DMD needs to be changed to handle incomplete source files. I'm no 
expert on this topic but wouldn't it be enough if the lexer and parser 
can handle incomplete source files? That might be feasible to complete 
during a summer.


 Either way I feel this is out 
 of scope for a single student to complete in a single summer.


I'm thinking it's very ambitious too.


 Thinking that most of the work is done because they can use DMD is just
backwards 
 and haven't ever written tooling tools.


Yes. There's no magic to use DMD as a library. Just compile all the 
necessary files and create a library out of the object files. It's been 
possible to do this for as long as DMD has existed. We just created a 
Dub package and some utility functions to package it nicely.

-- 
/Jacob Carlborg
Mar 16 2019
next sibling parent Rubn <where is.this> writes:
On Saturday, 16 March 2019 at 15:24:00 UTC, Jacob Carlborg wrote:

 On 2019-03-14 23:34, Rubn wrote:


 Which part? Point is using DMD as a language server isn't 
 going to be intuitive, compilers are designed to fail on 
 invalid input but editors need to work for incomplete source 
 files.


 Then DMD needs to be changed to handle incomplete source files. 
 I'm no expert on this topic but wouldn't it be enough if the 
 lexer and parser can handle incomplete source files? That might 
 be feasible to complete during a summer.


No it shouldn't, it's a compiler it should compile. Trying to 
shoehorn in features so it makes for a better library to be used 
as a language server will just make it a worse compiler.


 Thinking that most of the work is done because they can use 
 DMD is just backwards and haven't ever written tooling tools.


 Yes. There's no magic to use DMD as a library. Just compile all 
 the necessary files and create a library out of the object 
 files. It's been possible to do this for as long as DMD has 
 existed. We just created a Dub package and some utility 
 functions to package it nicely.


I wasn't talking about how difficult it is to compile DMD..
Mar 16 2019
prev sibling parent Rajat Soni <rajat.tcp gmail.com> writes:
On Saturday, 16 March 2019 at 15:24:00 UTC, Jacob Carlborg wrote:

 On 2019-03-14 23:34, Rubn wrote:


 Which part? Point is using DMD as a language server isn't 
 going to be intuitive, compilers are designed to fail on 
 invalid input but editors need to work for incomplete source 
 files.


 Then DMD needs to be changed to handle incomplete source files. 
 I'm no expert on this topic but wouldn't it be enough if the 
 lexer and parser can handle incomplete source files? That might 
 be feasible to complete during a summer.


 Either way I feel this is out of scope for a single student to 
 complete in a single summer.


 I'm thinking it's very ambitious too.


 Thinking that most of the work is done because they can use 
 DMD is just backwards and haven't ever written tooling tools.


 Yes. There's no magic to use DMD as a library. Just compile all 
 the necessary files and create a library out of the object 
 files. It's been possible to do this for as long as DMD has 
 existed. We just created a Dub package and some utility 
 functions to package it nicely.


This can be helpful ->

Source :- 
https://stackoverflow.com/questions/3061654/what-is-the-difference-between-javac-and-the-eclipse-compiler


Eclipse has implemented its own compiler called as Eclipse 
Compiler for Java (ECJ).

It is different from the javac, the compiler that is shipped with 
Sun JDK. One notable difference is that the Eclipse compiler lets 
you run code that didn't actually properly compile. If the block 
of code with the error is never ran, your program will run fine. 
Otherwise, it will throw an exception indicating that you tried 
to run code that doesn't compile.

Another difference is that the Eclipse compiler allows for 
incremental builds from within the Eclipse IDE, that is, all code 
is compiled as soon as you finish typing.

The fact that Eclipse comes with its own compiler is also 
apparent because you can write, compile, and run Java code in 
Eclipse without even installing the Java SDK.
Mar 24 2019
prev sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Wednesday, 13 March 2019 at 23:32:13 UTC, Rubn wrote:

 Using the actual DMD compiler may not be that great as a 
 language server.

 https://code.visualstudio.com/api/language-extensions/language-server-extension-guide#error-tolerant-parser-for-language-server

 https://github.com/Microsoft/tolerant-php-parser/blob/master/docs/HowItWorks.md


Similar concept, used in Atom's last release for C++ and JS 
parsing:

https://tree-sitter.github.io/tree-sitter/
Mar 16 2019
prev sibling parent Andrea Fontana <nospam example.org> writes:
On Wednesday, 13 March 2019 at 11:13:13 UTC, Seb wrote:

  community: what language smarts in an editor are the most 
 important to you? What features do you miss in DLS/DCD?


I agree with Jacob. I would add another feature I miss: methods / 
properties / etc. should be sorted by relevance. For example 
overriden methods (or new ones) should come before inherited 
ones. Also frequency should be a sort key. You can see something 
like this in android studio.

Andrea
Mar 14 2019