www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [gsoc] DMD - Ideas

reply Seb <seb wilzba.ch> writes:
So this is the second thread in my series to explore potential 
GSoC projects for students and provide more detailed insights, 
ideas, impression and advise for them.

The first thread can be found here [1] and the list of all 
potential GSoC ideas is still at [2].

This thread I want to devote to DMD as it's obviously one (if not 
the) most important piece in D's infrastructure.

The wiki already suggests a few ideas for DMD:

- header generation for C/C++ [3]
- language server protocol for D based on DMD as a library [4]

And there's also DIP1014 [5] which could be implemented as part 
of a GSoC project.

Anyhow, I make a start and list my personal favorite shortcomings 
of DMD:

- better error messages (see e.g. Adam's recent thread [6])
- multiple alias this (see [7] for the last attempt)
- string interpolation (see [8] for the last attempt)

There are a few more DMD goals defined in last year's survey (see 
e.g. [9]), though some of them are too much work for the three 
months of GSoC.

 community: what other features do you miss the most from DMD 
that could be done in the span of a GSoC?

[1] 
https://forum.dlang.org/post/eftttpylxanvxjhoigqu forum.dlang.org
[2] https://wiki.dlang.org/GSOC_2019_Ideas
[3] 
https://wiki.dlang.org/GSOC_2019_Ideas#Header_generation_for_C.2FC.2B.2B
[4] 
https://wiki.dlang.org/GSOC_2019_Ideas#Language_Server_Protocol_for_D
[5] 
https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1014.md
[6] 
https://forum.dlang.org/post/amiprwpqkkmbdbhaggua forum.dlang.org
[7] https://github.com/dlang/dmd/pull/8378
[8] https://github.com/dlang/dmd/pull/7988
[9] 
https://rawgit.com/wilzbach/state-of-d/master/report.html#180373345
Mar 07 2019
next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Thursday, 7 March 2019 at 09:53:13 UTC, Seb wrote:

  community: what other features do you miss the most from DMD 
 that could be done in the span of a GSoC?


I would like to have an additional coverage report format. 
Instead of having multiple
lst files, I would like to have 1 file. Either in XML or JSON 
format. Something
like Cobertura format would be great:

<coverage lines-valid="463" lines-covered="299" 
line-rate="0.6458">
<packages>
<package name="runtime.shell" line-rate="0.01666">
<class name="runtime.shell.app" 
filename="source\runtime\shell\app.d" line-rate="1">
<lines>
<line number="9" hits="1"/>
</lines>
</class>
<class name="runtime.shell.command" 
filename="source\runtime\shell\command.d" line-rate="0">
<lines>
<line number="17" hits="0"/>
<line number="19" hits="0"/>
<line number="29" hits="0"/>
<line number="30" hits="0"/>
<line number="37" hits="0"/>

The luxury solution of course would be, if I could hook into the 
coverage report logic of DMD and could write my own coverage 
reporter.

Kind regards
André
Mar 07 2019
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 7 March 2019 at 09:53:13 UTC, Seb wrote:

  community: what other features do you miss the most from DMD 
 that could be done in the span of a GSoC?


Convert runtime hooks [1] to templates

Example
--------
void main()
{
     auto bArray = new byte[5];
     bArray.length = 8;  // gets lowered to `_d_arraysetlengthT`
}

`bArray.length = 8` gets lowered by the compiler to the runtime 
hook `_d_arraysetlengthT` [2].  If you look at the signature for 
`_d_arraysetlengthT`, you'll see that it takes a runtime 
`TypeInfo` object. That is unfortunate because everything the 
compiler needs for the implementation is known at compile-time. 
Therefore, `_d_arraysetlengthT` should be converted to a 
template, and the compiler should be changed to lower 
`bArray.length = 8` to the template implementation.

Lucia Cojocaru led the charge on this in 2017.  She gave a great 
talk about it at DConf 2017 [3].  Her motivations for doing this 
work came accross much different than mine, but still valid and 
beneficial to D.  For me, one of the problems with `TypeInfo` is 
that it is a class, and clesses require a great deal of runtime 
support in D.  If the runtime hooks have a dependency on 
`TypeInfo` classes, then that means every D language feature that 
gets lowered to a runtime hook also depends on runtime support 
for classes.  Runtime support for classes includes the GC, and 
that doesn't scale well to bare-metal and resource-constrained 
platforms that neither want, nor need, that runtime overhead.  
Implementing the runtime hooks as templates leverages D's 
all-star features such as design by introspection, 
metaprogramming, and CTFE which can yield performance benefits as 
demonstrated by Lucia.

Although it was before my time, my understanding for why things 
are currently implemented with runtime `TypeInfo` when everything 
is known at compile-time is because the runtime hooks were 
written before D had templates. Now that D has templates, it 
seems, at least to me, that templates is the obvious choice.

There are challenges with this effort, however. In the compiler, 
the lowerings are performed after the semantic phase in e2ir.d 
[4]. This means that currently, for example, setting the length 
of a dynamic array can be done in a ` safe`, `pure`, and 
`nothrow` context [5], but `_d_arraysetlengthT` is neither 
` safe`, `pure`, nor `nothrow` [2]. The compiler is lying to us. 
So, if you convert these runtime hooks to templates, the lowering 
will be done in the semantic phase, and suddenly, you'll be 
forced to be honest. After a conversation with Walter, I was told 
to make use of `pureMalloc` and friends [6] to work around the 
`pure` constraint. `trusted` can be used to work around the 
` safe` constraint, and instead of throwing exceptions in 
`nothrow` code, `assert`ions can be used.

See [7] for an example implementation.

Mike

[1] https://wiki.dlang.org/Runtime_Hooks
[2] 
https://github.com/dlang/druntime/blob/c86e4a0e541dc63f9e9f2ee09c7efd325c0be2d5/src/rt/lifetime.d#L1443
[3] https://www.youtube.com/watch?v=endKC3fDxqs
[4] 
https://github.com/dlang/dmd/blob/f4f29c3e47ee30b2646cede35ed0f9bd823f0add/src/dmd/e2ir.d#L2611
[5] https://run.dlang.io/is/2aefNu
[6] 
https://github.com/dlang/druntime/blob/29f495b5b3571484bcf4d0af2fe211d6b7d86830/src/core/memory.d#L862
[7] https://github.com/dlang/dmd/pull/8531
Mar 07 2019
prev sibling next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Thursday, 7 March 2019 at 09:53:13 UTC, Seb wrote:

 So this is the second thread in my series to explore potential 
 GSoC projects for students and provide more detailed insights, 
 ideas, impression and advise for them.

 [...]


Compile time float to string conversion would also be great. See 
thread here

https://forum.dlang.org/thread/uytbknagemrzwlhawxud forum.dlang.org

Kind regards
Andre
Mar 07 2019
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 7 March 2019 at 09:53:13 UTC, Seb wrote:

  community: what other features do you miss the most from DMD 
 that could be done in the span of a GSoC?


`alloca` cannot be used without linking in druntime [1], and 
`scope` arrays should allocate on the stack [2]

In DMD `alloca` is implemented in druntime, so to do dynamic 
stack allocation, one must link in druntime.  -betterC does not 
link in the druntime, so dynamic stack allocation cannot be done 
in -betterC compiled code.  This is only a limitation of DMD, 
however; GDC and LDC can both do dynamic stack allocation without 
linking in druntime.

Also, it is odd that `scope` classes are allocated on the stack, 
but `scope` arrays are not.  My understanding is that allocating 
`scope` classes on the stack is done only as an optimization, but 
I don't see why that optimization wouldn't apply equally well to 
`scope` arrays.

Mike

[1] https://issues.dlang.org/show_bug.cgi?id=19159
[2] https://issues.dlang.org/show_bug.cgi?id=18788
Mar 07 2019
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 07, 2019 at 09:53:13AM +0000, Seb via Digitalmars-d wrote:
[...]

  community: what other features do you miss the most from DMD that
 could be done in the span of a GSoC?


[...]

Enable the GC in DMD and have it not crash and/or otherwise produce
wrong results / behaviour.

This is critical for making dmd (and D in general) *not* look like
laughing stock on low-memory machines.


T

-- 
What are you when you run out of Monet? Baroque.
Mar 07 2019
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2019-03-07 10:53, Seb wrote:


 The wiki already suggests a few ideas for DMD:


Library related:

* Source code transformations

* Incremental compilation. I'm not talking about recompiling a whole 
module and relink. I'm talking about lexing and parsing exactly only the 
text that has changed in a file since last time. Then run the semantic 
phases and code generation on those changes

New features to the language:

* Struct initialization that works everywhere [1]
* Better support for tuples [2]

[1] https://github.com/dlang/DIPs/pull/71
[2] https://github.com/tgehr/DIPs/blob/tuple-syntax/DIPs/DIP1xxx-tg.md

-- 
/Jacob Carlborg
Mar 08 2019
prev sibling next sibling parent Andrea Fontana <nospam example.org> writes:
On Thursday, 7 March 2019 at 09:53:13 UTC, Seb wrote:

 So this is the second thread in my series to explore potential 
 GSoC projects for students and provide more detailed insights, 
 ideas, impression and advise for them.


This [1] is a good starting point to make wishes come true.

[1] https://dlang.typeform.com/report/H1GTak/PY9NhHkcBFG0t6ig
Mar 08 2019
prev sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 7 March 2019 at 09:53:13 UTC, Seb wrote:


  community: what other features do you miss the most from DMD 
 that could be done in the span of a GSoC?


There is currently an unfinished project in the DMD repository 
that is intended to replace all makefiles with a script written 
in D [1].  It was started by Seb [2] (Thanks! Seb) and I tried to 
move it along [3].

There are a number of reasons for doing this.

1. The current build system relies on Posix-y and Nix-y tooling 
to work. Much of that tooling is not available by default on 
Windows, so a number of 3rd party dependencies need to be 
installed first to get a build on Windows.  This increases the 
barrier to entry for those wishing to contribute to D or build 
one's own compiler.

2. The build system is difficult to understand, further 
heightening the barrier to entry.  The one thing all contributors 
to D have in common is the D language itself.  It will be much 
easier to understand and debug a build system written in D than 
one written with makefiles.

3. Why in the world do we work so hard building extremely 
powerful programming languages like D to automate our lives, and 
then resort to inferior tools like makefiles to build our 
software? rdmd makes scripting in D easy an convenient.

build.d currently works for Linux, but it needs more work for 
Windows.  It's almost there; it just needs a little more work.

It currently mimics the makefiles, as it was ported from them, 
but I believe that once build.d is working, it can be refactored 
to something more idiomatic-D that all proficient D users can 
easily understand and debug.

Mike

[1] https://github.com/dlang/dmd/blob/master/src/build.d
[2] https://github.com/dlang/dmd/pull/8293
[3] 
https://github.com/dlang/dmd/pulls?q=is%3Apr+build.d+author%3AJinShil
Mar 13 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 14, 2019 at 12:16:26AM +0000, Mike Franklin via Digitalmars-d wrote:
[...]

 There is currently an unfinished project in the DMD repository that is
 intended to replace all makefiles with a script written in D [1].  It
 was started by Seb [2] (Thanks! Seb) and I tried to move it along [3].


Finally!



 There are a number of reasons for doing this.
 
 1. The current build system relies on Posix-y and Nix-y tooling to
 work.  Much of that tooling is not available by default on Windows, so
 a number of 3rd party dependencies need to be installed first to get a
 build on Windows.  This increases the barrier to entry for those
 wishing to contribute to D or build one's own compiler.


Yes, that is rather annoying. (Not that I'd know, though, since I don't
use Windows.)



 2. The build system is difficult to understand, further heightening
 the barrier to entry.  The one thing all contributors to D have in
 common is the D language itself.  It will be much easier to understand
 and debug a build system written in D than one written with makefiles.


That depends on how well-written the build system is. :-P  As somebody
once said, "you can write assembly code in *any* language..."



 3. Why in the world do we work so hard building extremely powerful
 programming languages like D to automate our lives, and then resort to
 inferior tools like makefiles to build our software? rdmd makes
 scripting in D easy an convenient.


This has come up many times before, with the usual response being Andrei
saying that nothing stands out among the large crowd of makefile
alternatives.  Has this stance changed since?



 build.d currently works for Linux, but it needs more work for Windows.
 It's almost there; it just needs a little more work.
 
 It currently mimics the makefiles, as it was ported from them, but I
 believe that once build.d is working, it can be refactored to
 something more idiomatic-D that all proficient D users can easily
 understand and debug.


[...]

OK, some questions.

1) Why not use an existing build system written in D?  There's Button,
Atila's build tool (sorry forgot what it was called), Dub (ick - sorry),
etc..

2) Won't there be bootstrapping issues to consider, if building dmd
requires build.d, but build.d needs to be compiled with dmd?


T

-- 
Valentine's Day: an occasion for florists to reach into the wallets of nominal
lovers in dire need of being reminded to profess their hypothetical love for
their long-forgotten.
Mar 19 2019
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-03-19 18:01, H. S. Teoh wrote:


 2) Won't there be bootstrapping issues to consider, if building dmd
 requires build.d, but build.d needs to be compiled with dmd?


Basically the whole compiler is written in D now. Bootstrapping is done 
by using a pre-compiled binary. For new targets cross-compiling is 
supposed to be used.

-- 
/Jacob Carlborg
Mar 19 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Mar 19, 2019 at 09:07:23PM +0100, Jacob Carlborg via Digitalmars-d
wrote:

 On 2019-03-19 18:01, H. S. Teoh wrote:
 

 2) Won't there be bootstrapping issues to consider, if building dmd
 requires build.d, but build.d needs to be compiled with dmd?


 
 Basically the whole compiler is written in D now. Bootstrapping is
 done by using a pre-compiled binary. For new targets cross-compiling
 is supposed to be used.


[...]

Precisely my point.  So now instead of a single precompiled binary
(dmd), you have two (dmd + build). When the two go out of sync, things
become "interesting".


T

-- 
Trying to define yourself is like trying to bite your own teeth. -- Alan Watts
Mar 19 2019
next sibling parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Tuesday, 19 March 2019 at 20:45:06 UTC, H. S. Teoh wrote:

 On Tue, Mar 19, 2019 at 09:07:23PM +0100, Jacob Carlborg via 
 Digitalmars-d wrote:

 On 2019-03-19 18:01, H. S. Teoh wrote:
 

 2) Won't there be bootstrapping issues to consider, if 
 building dmd requires build.d, but build.d needs to be 
 compiled with dmd?


 
 Basically the whole compiler is written in D now. 
 Bootstrapping is done by using a pre-compiled binary. For new 
 targets cross-compiling is supposed to be used.


 [...]

 Precisely my point.  So now instead of a single precompiled 
 binary
 (dmd), you have two (dmd + build). When the two go out of sync, 
 things
 become "interesting".


Is that a realistic concern? Say build suddenly lags 10 releases 
behind dmd. For that to happen, build.d would have to remain 
unchanged for that long.
Anyway, build.d could embed the version of the compiler with 
which it was built into itself, and check that against the 
version of the compiler that it calls for building. When they 
mismatch, it could rebuild itself (or print a warning).
Mar 19 2019
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Tuesday, 19 March 2019 at 22:30:20 UTC, Bastiaan Veelo wrote:


 Is that a realistic concern? Say build suddenly lags 10 
 releases behind dmd. For that to happen, build.d would have to 
 remain unchanged for that long.
 Anyway, build.d could embed the version of the compiler with 
 which it was built into itself, and check that against the 
 version of the compiler that it calls for building. When they 
 mismatch, it could rebuild itself (or print a warning).


build.d is tested in the CI with every PR.

Mike
Mar 19 2019
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Mar 19, 2019 at 10:30:20PM +0000, Bastiaan Veelo via Digitalmars-d
wrote:
[...]

 Anyway, build.d could embed the version of the compiler with which it
 was built into itself, and check that against the version of the
 compiler that it calls for building.


Actually, this leads to an interesting idea: what if build.d embeds an
executable copy of dmd within itself?  Then it will never be out of
sync, and you'll be guaranteed that a working compiler is always
available.


T

-- 
Doubtless it is a good thing to have an open mind, but a truly open mind should
be open at both ends, like the food-pipe, with the capacity for excretion as
well as absorption. -- Northrop Frye
Mar 20 2019
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2019-03-19 21:45, H. S. Teoh wrote:


 Precisely my point.  So now instead of a single precompiled binary
 (dmd), you have two (dmd + build). When the two go out of sync, things
 become "interesting".


What do you mean with "out of sync"?

-- 
/Jacob Carlborg
Mar 21 2019
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Tuesday, 19 March 2019 at 20:45:06 UTC, H. S. Teoh wrote:

 Precisely my point.  So now instead of a single precompiled 
 binary
 (dmd), you have two (dmd + build). When the two go out of sync, 
 things
 become "interesting".


 T


This is no concern at all ;-)
You see a DMD compiler is already _required_ to build DMD. We 
simply use the same host compiler to compile build.d. The CIs 
enforce that we can always compile DMD with the minimum host 
compiler and similarly they also enforce the same for build.d.

However, while we can build DMD and run its unittest without Make 
now, druntime and Phobos don't have the equivalent of `build.d` 
and `test/run.d` and some of the lesser used DMD Makefile targets 
like the docs target haven't been ported either, so it will still 
take a bit of time until we're in a post-make era.
Mar 22 2019
parent reply Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Friday, 22 March 2019 at 09:08:51 UTC, Seb wrote:

 On Tuesday, 19 March 2019 at 20:45:06 UTC, H. S. Teoh wrote:

 Precisely my point.  So now instead of a single precompiled 
 binary
 (dmd), you have two (dmd + build). When the two go out of 
 sync, things
 become "interesting".


 T


 This is no concern at all ;-)
 You see a DMD compiler is already _required_ to build DMD. We 
 simply use the same host compiler to compile build.d. The CIs 
 enforce that we can always compile DMD with the minimum host 
 compiler and similarly they also enforce the same for build.d.

 However, while we can build DMD and run its unittest without 
 Make now, druntime and Phobos don't have the equivalent of 
 `build.d` and `test/run.d` and some of the lesser used DMD 
 Makefile targets like the docs target haven't been ported 
 either, so it will still take a bit of time until we're in a 
 post-make era.


Bootstrapping is still important, though. There are situations 
where it is forbidden to install binaries on machines. Only the 
pre-installed software can be used. That's the situation for 
example at my work. We get Linux installations corresponding to 
some arbitrary levels IT has decided. The images are outdated and 
every tool or software must be compiled from sources. It's 
Kafkaesque, annoying and a total waste of time (for us, not for 
IT) but that's what I have to contend with. So to install dmd, I 
had to go through these steps:
build dmd, very old version 2.067 afaicr, then build newer 
versions. It has to be build also directly from the git repo. All 
the other nice tools to build any version (digger and so on) fail 
on our systems because of the very picky proxy. Yeah, our IT 
environment is a PITA.

So some indication somewhere which version was the last one that 
could be built without dmd would be a nice datapoint to have.
Mar 22 2019
parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Friday, 22 March 2019 at 09:49:18 UTC, Patrick Schluter wrote:

 All the other nice tools to build any version (digger and so 
 on) fail on our systems because of the very picky proxy. Yeah, 
 our IT environment is a PITA.


Could you please file some issues at 
https://github.com/CyberShadow/Digger/issues and mention what's 
missing in Digger to support this?
Mar 22 2019
prev sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Tuesday, 19 March 2019 at 17:01:43 UTC, H. S. Teoh wrote:


 1) Why not use an existing build system written in D?  There's 
 Button,
 Atila's build tool (sorry forgot what it was called), Dub (ick 
 - sorry),
 etc..


I think you're talking about Reggae 
(https://github.com/atilaneves/reggae)

I'm not against using it, but it does introduce yet another 
dependency and yet another layer of abstraction to learn and 
become familiar with.  And what are Reggae's dependencies?  I'd 
approve and merge a PR that replaces the current build system 
with one written in Reggae, but I'd rather not have to learn it.  
Also, looking at the Reggae README, it looks very makefile-ish.  
I'd rather the end result look more like procedural idiomatic D 
that I could easily add print statements to or step through with 
a debugger.

Mike
Mar 19 2019