www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.internals - DMD with GC

reply RazvanN <razvan.nitu1305 gmail.com> writes:
Hi all,

Currently the compiler is not garbage collected due to the fact 
that the GC slows it down. This means that at some point you 
could enable the GC, while at the moment if you do that you get 
ugly segfaults (see [1]). I am interested in this because the 
compiler library should have the possibility of being garbage 
collected and I am curios if somebody knows what was the change 
which made the compiler incompatible with the GC.

Thanks,
RazvanN

[1] 
http://forum.dlang.org/thread/dywncjvlezvrbjkfkbsz forum.dlang.org
Nov 07 2017
next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 07.11.2017 10:04, RazvanN wrote:

 Hi all,
 
 Currently the compiler is not garbage collected due to the fact that the 
 GC slows it down. This means that at some point you could enable the GC, 
 while at the moment if you do that you get ugly segfaults (see [1]). I 
 am interested in this because the compiler library should have the 
 possibility of being garbage collected and I am curios if somebody knows 
 what was the change which made the compiler incompatible with the GC.
 
 Thanks,
 RazvanN
 
 [1] http://forum.dlang.org/thread/dywncjvlezvrbjkfkbsz forum.dlang.org


I recently built a semantic server for Visual D from the dmd front end 
with the GC enabled, and it didn't crash AFAICT. I don't think I had to 
make any patch but the one to make it compile 
(https://github.com/dlang/dmd/pull/7156), but you have to stop the GC 
before calling the backend and actually generate code.

It has some other issues though prohibiting it to be a viable solution 
for an IDE ATM:

- the allocation through the rmem module has the disadvantage that type 
information is lost, so that every block has to be scanned 
conservatively, even if it never contains pointers. This makes it more 
likely to have false pointers.

- there are a number of (private or function static) global variables 
that are populated during compilation, and it's not simple (though 
possible) to reset them to restart the analysis

- it eats memory like crazy. One problem I identified is that there are 
a number of pools that allocate memory in large chunks that continue to 
contain old references.

- the semantic analysis does quite a bit of transformations that make it 
difficult to match the AST to the source. Worst of all: the error 
propagation prunes complete sub trees of the AST, so there is no 
information to lookup while the code doesn't compile. This prohibits any 
help while editing, e.g. code completion or tool tips with inferred type 
information.

So it worked for displaying semantic errors during editing (Visual D 
currently only displays parsing errors), but it's not really fast.
It would be nice to have some reasonable dependency tracking so that a 
complete analysis doesn't have to be redone with every edit.
Nov 07 2017
parent reply RazvanN <razvan.nitu1305 gmail.com> writes:
On Tuesday, 7 November 2017 at 21:14:07 UTC, Rainer Schuetze 
wrote:

 I recently built a semantic server for Visual D from the dmd 
 front end with the GC enabled, and it didn't crash AFAICT. I 
 don't think I had to make any patch but the one to make it 
 compile (https://github.com/dlang/dmd/pull/7156), but you have 
 to stop the GC before calling the backend and actually generate 
 code.


I tried to compile the parser as a library with the GC disabled,
but running the example in dmd/src/examples/test_parser.d results
in a segmentation fault. Running with the GC disabled completes
successfully, so I guess there still are some corner cases, though
I am noob when it comes to GCed dmd. The patch you mentioned does 
not
fix the case presented in [1].

[1] 
http://forum.dlang.org/thread/dywncjvlezvrbjkfkbsz forum.dlang.org
Nov 08 2017
parent reply Jacob Carlborg <doob me.com> writes:
On 2017-11-09 07:54, RazvanN wrote:


 I tried to compile the parser as a library with the GC disabled,
 but running the example in dmd/src/examples/test_parser.d results
 in a segmentation fault. Running with the GC disabled completes
 successfully, so I guess there still are some corner cases, though
 I am noob when it comes to GCed dmd. The patch you mentioned does not
 fix the case presented in [1].
 
 [1] http://forum.dlang.org/thread/dywncjvlezvrbjkfkbsz forum.dlang.org


Looks like the problem occurs when importing the ddmd.root.rmem [1] 
module, which is basically always done one way or other. This module 
will override some of the D runtime functions [2], which somehow causes 
a conflict with GC. When the module defines these functions [2] they 
will end up in the object file for the module. When that happens the 
linker won't search for these symbols in any external libraries, i.e. 
druntime (statically linked into Phobos), which it normally does otherwise.

The solution is available in that module, compile with the "GC" version 
enabled, i.e. "dmd -version=GC".

[1] https://github.com/dlang/dmd/blob/master/src/ddmd/root/rmem.d
[2] https://github.com/dlang/dmd/blob/master/src/ddmd/root/rmem.d#L195

-- 
/Jacob Carlborg
Nov 09 2017
parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 09.11.2017 09:20, Jacob Carlborg wrote:

 On 2017-11-09 07:54, RazvanN wrote:
 

 I tried to compile the parser as a library with the GC disabled,
 but running the example in dmd/src/examples/test_parser.d results
 in a segmentation fault. Running with the GC disabled completes
 successfully, so I guess there still are some corner cases, though
 I am noob when it comes to GCed dmd. The patch you mentioned does not
 fix the case presented in [1].

 [1] http://forum.dlang.org/thread/dywncjvlezvrbjkfkbsz forum.dlang.org


 
 Looks like the problem occurs when importing the ddmd.root.rmem [1] 
 module, which is basically always done one way or other. This module 
 will override some of the D runtime functions [2], which somehow causes 
 a conflict with GC. When the module defines these functions [2] they 
 will end up in the object file for the module. When that happens the 
 linker won't search for these symbols in any external libraries, i.e. 
 druntime (statically linked into Phobos), which it normally does otherwise.
 
 The solution is available in that module, compile with the "GC" version 
 enabled, i.e. "dmd -version=GC".
 
 [1] https://github.com/dlang/dmd/blob/master/src/ddmd/root/rmem.d
 [2] https://github.com/dlang/dmd/blob/master/src/ddmd/root/rmem.d#L195
 


Yes, that's what I did, too: compile the dmd frontend with -version=GC.

IIRC I could also compile phobos unittests with the GC version (it was 
just 10-20% slower), but building them all with a single dmd invocation 
crashed during code generation (which has to run with the GC disabled).
AFAICT it got further than the regular version but probably I didn't 
have enough swap space when the process needed more than 16 GB).
Nov 09 2017
prev sibling parent Joakim <dlang joakim.fea.st> writes:
On Tuesday, 7 November 2017 at 09:04:36 UTC, RazvanN wrote:

 Hi all,

 Currently the compiler is not garbage collected due to the fact 
 that the GC slows it down. This means that at some point you 
 could enable the GC, while at the moment if you do that you get 
 ugly segfaults (see [1]). I am interested in this because the 
 compiler library should have the possibility of being garbage 
 collected and I am curios if somebody knows what was the change 
 which made the compiler incompatible with the GC.


Seeing this late, but the dmd compiler itself was never a GC'ed 
program.  The frontend was written in C++ then mostly 
automatically translated to D, and the old manual allocation 
scheme was kept. To make it actually work with GC for the first 
time would probably require some work, same for your library.

The good news is your library is  nogc for free! ;)
Dec 19 2017