www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [gsoc] DMD - Ideas

reply Seb <seb wilzba.ch> writes:
So this is the second thread in my series to explore potential 
GSoC projects for students and provide more detailed insights, 
ideas, impression and advise for them.

The first thread can be found here [1] and the list of all 
potential GSoC ideas is still at [2].

This thread I want to devote to DMD as it's obviously one (if not 
the) most important piece in D's infrastructure.

The wiki already suggests a few ideas for DMD:

- header generation for C/C++ [3]
- language server protocol for D based on DMD as a library [4]

And there's also DIP1014 [5] which could be implemented as part 
of a GSoC project.

Anyhow, I make a start and list my personal favorite shortcomings 
of DMD:

- better error messages (see e.g. Adam's recent thread [6])
- multiple alias this (see [7] for the last attempt)
- string interpolation (see [8] for the last attempt)

There are a few more DMD goals defined in last year's survey (see 
e.g. [9]), though some of them are too much work for the three 
months of GSoC.

 community: what other features do you miss the most from DMD 
that could be done in the span of a GSoC?

[1] 
https://forum.dlang.org/post/eftttpylxanvxjhoigqu forum.dlang.org
[2] https://wiki.dlang.org/GSOC_2019_Ideas
[3] 
https://wiki.dlang.org/GSOC_2019_Ideas#Header_generation_for_C.2FC.2B.2B
[4] 
https://wiki.dlang.org/GSOC_2019_Ideas#Language_Server_Protocol_for_D
[5] 
https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1014.md
[6] 
https://forum.dlang.org/post/amiprwpqkkmbdbhaggua forum.dlang.org
[7] https://github.com/dlang/dmd/pull/8378
[8] https://github.com/dlang/dmd/pull/7988
[9] 
https://rawgit.com/wilzbach/state-of-d/master/report.html#180373345
Mar 07
next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Thursday, 7 March 2019 at 09:53:13 UTC, Seb wrote:
  community: what other features do you miss the most from DMD 
 that could be done in the span of a GSoC?
I would like to have an additional coverage report format. Instead of having multiple lst files, I would like to have 1 file. Either in XML or JSON format. Something like Cobertura format would be great: <coverage lines-valid="463" lines-covered="299" line-rate="0.6458"> <packages> <package name="runtime.shell" line-rate="0.01666"> <class name="runtime.shell.app" filename="source\runtime\shell\app.d" line-rate="1"> <lines> <line number="9" hits="1"/> </lines> </class> <class name="runtime.shell.command" filename="source\runtime\shell\command.d" line-rate="0"> <lines> <line number="17" hits="0"/> <line number="19" hits="0"/> <line number="29" hits="0"/> <line number="30" hits="0"/> <line number="37" hits="0"/> The luxury solution of course would be, if I could hook into the coverage report logic of DMD and could write my own coverage reporter. Kind regards André
Mar 07
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 7 March 2019 at 09:53:13 UTC, Seb wrote:
  community: what other features do you miss the most from DMD 
 that could be done in the span of a GSoC?
Convert runtime hooks [1] to templates Example -------- void main() { auto bArray = new byte[5]; bArray.length = 8; // gets lowered to `_d_arraysetlengthT` } `bArray.length = 8` gets lowered by the compiler to the runtime hook `_d_arraysetlengthT` [2]. If you look at the signature for `_d_arraysetlengthT`, you'll see that it takes a runtime `TypeInfo` object. That is unfortunate because everything the compiler needs for the implementation is known at compile-time. Therefore, `_d_arraysetlengthT` should be converted to a template, and the compiler should be changed to lower `bArray.length = 8` to the template implementation. Lucia Cojocaru led the charge on this in 2017. She gave a great talk about it at DConf 2017 [3]. Her motivations for doing this work came accross much different than mine, but still valid and beneficial to D. For me, one of the problems with `TypeInfo` is that it is a class, and clesses require a great deal of runtime support in D. If the runtime hooks have a dependency on `TypeInfo` classes, then that means every D language feature that gets lowered to a runtime hook also depends on runtime support for classes. Runtime support for classes includes the GC, and that doesn't scale well to bare-metal and resource-constrained platforms that neither want, nor need, that runtime overhead. Implementing the runtime hooks as templates leverages D's all-star features such as design by introspection, metaprogramming, and CTFE which can yield performance benefits as demonstrated by Lucia. Although it was before my time, my understanding for why things are currently implemented with runtime `TypeInfo` when everything is known at compile-time is because the runtime hooks were written before D had templates. Now that D has templates, it seems, at least to me, that templates is the obvious choice. There are challenges with this effort, however. In the compiler, the lowerings are performed after the semantic phase in e2ir.d [4]. This means that currently, for example, setting the length of a dynamic array can be done in a ` safe`, `pure`, and `nothrow` context [5], but `_d_arraysetlengthT` is neither ` safe`, `pure`, nor `nothrow` [2]. The compiler is lying to us. So, if you convert these runtime hooks to templates, the lowering will be done in the semantic phase, and suddenly, you'll be forced to be honest. After a conversation with Walter, I was told to make use of `pureMalloc` and friends [6] to work around the `pure` constraint. `trusted` can be used to work around the ` safe` constraint, and instead of throwing exceptions in `nothrow` code, `assert`ions can be used. See [7] for an example implementation. Mike [1] https://wiki.dlang.org/Runtime_Hooks [2] https://github.com/dlang/druntime/blob/c86e4a0e541dc63f9e9f2ee09c7efd325c0be2d5/src/rt/lifetime.d#L1443 [3] https://www.youtube.com/watch?v=endKC3fDxqs [4] https://github.com/dlang/dmd/blob/f4f29c3e47ee30b2646cede35ed0f9bd823f0add/src/dmd/e2ir.d#L2611 [5] https://run.dlang.io/is/2aefNu [6] https://github.com/dlang/druntime/blob/29f495b5b3571484bcf4d0af2fe211d6b7d86830/src/core/memory.d#L862 [7] https://github.com/dlang/dmd/pull/8531
Mar 07
prev sibling next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Thursday, 7 March 2019 at 09:53:13 UTC, Seb wrote:
 So this is the second thread in my series to explore potential 
 GSoC projects for students and provide more detailed insights, 
 ideas, impression and advise for them.

 [...]
Compile time float to string conversion would also be great. See thread here https://forum.dlang.org/thread/uytbknagemrzwlhawxud forum.dlang.org Kind regards Andre
Mar 07
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 7 March 2019 at 09:53:13 UTC, Seb wrote:
  community: what other features do you miss the most from DMD 
 that could be done in the span of a GSoC?
`alloca` cannot be used without linking in druntime [1], and `scope` arrays should allocate on the stack [2] In DMD `alloca` is implemented in druntime, so to do dynamic stack allocation, one must link in druntime. -betterC does not link in the druntime, so dynamic stack allocation cannot be done in -betterC compiled code. This is only a limitation of DMD, however; GDC and LDC can both do dynamic stack allocation without linking in druntime. Also, it is odd that `scope` classes are allocated on the stack, but `scope` arrays are not. My understanding is that allocating `scope` classes on the stack is done only as an optimization, but I don't see why that optimization wouldn't apply equally well to `scope` arrays. Mike [1] https://issues.dlang.org/show_bug.cgi?id=19159 [2] https://issues.dlang.org/show_bug.cgi?id=18788
Mar 07
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 07, 2019 at 09:53:13AM +0000, Seb via Digitalmars-d wrote:
[...]
  community: what other features do you miss the most from DMD that
 could be done in the span of a GSoC?
[...] Enable the GC in DMD and have it not crash and/or otherwise produce wrong results / behaviour. This is critical for making dmd (and D in general) *not* look like laughing stock on low-memory machines. T -- What are you when you run out of Monet? Baroque.
Mar 07
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2019-03-07 10:53, Seb wrote:

 The wiki already suggests a few ideas for DMD:
Library related: * Source code transformations * Incremental compilation. I'm not talking about recompiling a whole module and relink. I'm talking about lexing and parsing exactly only the text that has changed in a file since last time. Then run the semantic phases and code generation on those changes New features to the language: * Struct initialization that works everywhere [1] * Better support for tuples [2] [1] https://github.com/dlang/DIPs/pull/71 [2] https://github.com/tgehr/DIPs/blob/tuple-syntax/DIPs/DIP1xxx-tg.md -- /Jacob Carlborg
Mar 08
prev sibling next sibling parent Andrea Fontana <nospam example.org> writes:
On Thursday, 7 March 2019 at 09:53:13 UTC, Seb wrote:
 So this is the second thread in my series to explore potential 
 GSoC projects for students and provide more detailed insights, 
 ideas, impression and advise for them.
This [1] is a good starting point to make wishes come true. [1] https://dlang.typeform.com/report/H1GTak/PY9NhHkcBFG0t6ig
Mar 08
prev sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 7 March 2019 at 09:53:13 UTC, Seb wrote:

  community: what other features do you miss the most from DMD 
 that could be done in the span of a GSoC?
There is currently an unfinished project in the DMD repository that is intended to replace all makefiles with a script written in D [1]. It was started by Seb [2] (Thanks! Seb) and I tried to move it along [3]. There are a number of reasons for doing this. 1. The current build system relies on Posix-y and Nix-y tooling to work. Much of that tooling is not available by default on Windows, so a number of 3rd party dependencies need to be installed first to get a build on Windows. This increases the barrier to entry for those wishing to contribute to D or build one's own compiler. 2. The build system is difficult to understand, further heightening the barrier to entry. The one thing all contributors to D have in common is the D language itself. It will be much easier to understand and debug a build system written in D than one written with makefiles. 3. Why in the world do we work so hard building extremely powerful programming languages like D to automate our lives, and then resort to inferior tools like makefiles to build our software? rdmd makes scripting in D easy an convenient. build.d currently works for Linux, but it needs more work for Windows. It's almost there; it just needs a little more work. It currently mimics the makefiles, as it was ported from them, but I believe that once build.d is working, it can be refactored to something more idiomatic-D that all proficient D users can easily understand and debug. Mike [1] https://github.com/dlang/dmd/blob/master/src/build.d [2] https://github.com/dlang/dmd/pull/8293 [3] https://github.com/dlang/dmd/pulls?q=is%3Apr+build.d+author%3AJinShil
Mar 13
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 14, 2019 at 12:16:26AM +0000, Mike Franklin via Digitalmars-d wrote:
[...]
 There is currently an unfinished project in the DMD repository that is
 intended to replace all makefiles with a script written in D [1].  It
 was started by Seb [2] (Thanks! Seb) and I tried to move it along [3].
Finally!
 There are a number of reasons for doing this.
 
 1. The current build system relies on Posix-y and Nix-y tooling to
 work.  Much of that tooling is not available by default on Windows, so
 a number of 3rd party dependencies need to be installed first to get a
 build on Windows.  This increases the barrier to entry for those
 wishing to contribute to D or build one's own compiler.
Yes, that is rather annoying. (Not that I'd know, though, since I don't use Windows.)
 2. The build system is difficult to understand, further heightening
 the barrier to entry.  The one thing all contributors to D have in
 common is the D language itself.  It will be much easier to understand
 and debug a build system written in D than one written with makefiles.
That depends on how well-written the build system is. :-P As somebody once said, "you can write assembly code in *any* language..."
 3. Why in the world do we work so hard building extremely powerful
 programming languages like D to automate our lives, and then resort to
 inferior tools like makefiles to build our software? rdmd makes
 scripting in D easy an convenient.
This has come up many times before, with the usual response being Andrei saying that nothing stands out among the large crowd of makefile alternatives. Has this stance changed since?
 build.d currently works for Linux, but it needs more work for Windows.
 It's almost there; it just needs a little more work.
 
 It currently mimics the makefiles, as it was ported from them, but I
 believe that once build.d is working, it can be refactored to
 something more idiomatic-D that all proficient D users can easily
 understand and debug.
[...] OK, some questions. 1) Why not use an existing build system written in D? There's Button, Atila's build tool (sorry forgot what it was called), Dub (ick - sorry), etc.. 2) Won't there be bootstrapping issues to consider, if building dmd requires build.d, but build.d needs to be compiled with dmd? T -- Valentine's Day: an occasion for florists to reach into the wallets of nominal lovers in dire need of being reminded to profess their hypothetical love for their long-forgotten.
Mar 19
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-03-19 18:01, H. S. Teoh wrote:

 2) Won't there be bootstrapping issues to consider, if building dmd
 requires build.d, but build.d needs to be compiled with dmd?
Basically the whole compiler is written in D now. Bootstrapping is done by using a pre-compiled binary. For new targets cross-compiling is supposed to be used. -- /Jacob Carlborg
Mar 19
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Mar 19, 2019 at 09:07:23PM +0100, Jacob Carlborg via Digitalmars-d
wrote:
 On 2019-03-19 18:01, H. S. Teoh wrote:
 
 2) Won't there be bootstrapping issues to consider, if building dmd
 requires build.d, but build.d needs to be compiled with dmd?
Basically the whole compiler is written in D now. Bootstrapping is done by using a pre-compiled binary. For new targets cross-compiling is supposed to be used.
[...] Precisely my point. So now instead of a single precompiled binary (dmd), you have two (dmd + build). When the two go out of sync, things become "interesting". T -- Trying to define yourself is like trying to bite your own teeth. -- Alan Watts
Mar 19
next sibling parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Tuesday, 19 March 2019 at 20:45:06 UTC, H. S. Teoh wrote:
 On Tue, Mar 19, 2019 at 09:07:23PM +0100, Jacob Carlborg via 
 Digitalmars-d wrote:
 On 2019-03-19 18:01, H. S. Teoh wrote:
 
 2) Won't there be bootstrapping issues to consider, if 
 building dmd requires build.d, but build.d needs to be 
 compiled with dmd?
Basically the whole compiler is written in D now. Bootstrapping is done by using a pre-compiled binary. For new targets cross-compiling is supposed to be used.
[...] Precisely my point. So now instead of a single precompiled binary (dmd), you have two (dmd + build). When the two go out of sync, things become "interesting".
Is that a realistic concern? Say build suddenly lags 10 releases behind dmd. For that to happen, build.d would have to remain unchanged for that long. Anyway, build.d could embed the version of the compiler with which it was built into itself, and check that against the version of the compiler that it calls for building. When they mismatch, it could rebuild itself (or print a warning).
Mar 19
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Tuesday, 19 March 2019 at 22:30:20 UTC, Bastiaan Veelo wrote:

 Is that a realistic concern? Say build suddenly lags 10 
 releases behind dmd. For that to happen, build.d would have to 
 remain unchanged for that long.
 Anyway, build.d could embed the version of the compiler with 
 which it was built into itself, and check that against the 
 version of the compiler that it calls for building. When they 
 mismatch, it could rebuild itself (or print a warning).
build.d is tested in the CI with every PR. Mike
Mar 19
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Mar 19, 2019 at 10:30:20PM +0000, Bastiaan Veelo via Digitalmars-d
wrote:
[...]
 Anyway, build.d could embed the version of the compiler with which it
 was built into itself, and check that against the version of the
 compiler that it calls for building.
Actually, this leads to an interesting idea: what if build.d embeds an executable copy of dmd within itself? Then it will never be out of sync, and you'll be guaranteed that a working compiler is always available. T -- Doubtless it is a good thing to have an open mind, but a truly open mind should be open at both ends, like the food-pipe, with the capacity for excretion as well as absorption. -- Northrop Frye
Mar 20
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2019-03-19 21:45, H. S. Teoh wrote:

 Precisely my point.  So now instead of a single precompiled binary
 (dmd), you have two (dmd + build). When the two go out of sync, things
 become "interesting".
What do you mean with "out of sync"? -- /Jacob Carlborg
Mar 21
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Tuesday, 19 March 2019 at 20:45:06 UTC, H. S. Teoh wrote:
 Precisely my point.  So now instead of a single precompiled 
 binary
 (dmd), you have two (dmd + build). When the two go out of sync, 
 things
 become "interesting".


 T
This is no concern at all ;-) You see a DMD compiler is already _required_ to build DMD. We simply use the same host compiler to compile build.d. The CIs enforce that we can always compile DMD with the minimum host compiler and similarly they also enforce the same for build.d. However, while we can build DMD and run its unittest without Make now, druntime and Phobos don't have the equivalent of `build.d` and `test/run.d` and some of the lesser used DMD Makefile targets like the docs target haven't been ported either, so it will still take a bit of time until we're in a post-make era.
Mar 22
parent reply Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Friday, 22 March 2019 at 09:08:51 UTC, Seb wrote:
 On Tuesday, 19 March 2019 at 20:45:06 UTC, H. S. Teoh wrote:
 Precisely my point.  So now instead of a single precompiled 
 binary
 (dmd), you have two (dmd + build). When the two go out of 
 sync, things
 become "interesting".


 T
This is no concern at all ;-) You see a DMD compiler is already _required_ to build DMD. We simply use the same host compiler to compile build.d. The CIs enforce that we can always compile DMD with the minimum host compiler and similarly they also enforce the same for build.d. However, while we can build DMD and run its unittest without Make now, druntime and Phobos don't have the equivalent of `build.d` and `test/run.d` and some of the lesser used DMD Makefile targets like the docs target haven't been ported either, so it will still take a bit of time until we're in a post-make era.
Bootstrapping is still important, though. There are situations where it is forbidden to install binaries on machines. Only the pre-installed software can be used. That's the situation for example at my work. We get Linux installations corresponding to some arbitrary levels IT has decided. The images are outdated and every tool or software must be compiled from sources. It's Kafkaesque, annoying and a total waste of time (for us, not for IT) but that's what I have to contend with. So to install dmd, I had to go through these steps: build dmd, very old version 2.067 afaicr, then build newer versions. It has to be build also directly from the git repo. All the other nice tools to build any version (digger and so on) fail on our systems because of the very picky proxy. Yeah, our IT environment is a PITA. So some indication somewhere which version was the last one that could be built without dmd would be a nice datapoint to have.
Mar 22
parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Friday, 22 March 2019 at 09:49:18 UTC, Patrick Schluter wrote:
 All the other nice tools to build any version (digger and so 
 on) fail on our systems because of the very picky proxy. Yeah, 
 our IT environment is a PITA.
Could you please file some issues at https://github.com/CyberShadow/Digger/issues and mention what's missing in Digger to support this?
Mar 22
prev sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Tuesday, 19 March 2019 at 17:01:43 UTC, H. S. Teoh wrote:

 1) Why not use an existing build system written in D?  There's 
 Button,
 Atila's build tool (sorry forgot what it was called), Dub (ick 
 - sorry),
 etc..
I think you're talking about Reggae (https://github.com/atilaneves/reggae) I'm not against using it, but it does introduce yet another dependency and yet another layer of abstraction to learn and become familiar with. And what are Reggae's dependencies? I'd approve and merge a PR that replaces the current build system with one written in Reggae, but I'd rather not have to learn it. Also, looking at the Reggae README, it looks very makefile-ish. I'd rather the end result look more like procedural idiomatic D that I could easily add print statements to or step through with a debugger. Mike
Mar 19