www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Appreciating D

reply IchorDev <zxinsworld gmail.com> writes:
This is going to be a long one.

I have been thinking about [this blog 
post](https://www.yet-another-blog.com/porting_the_game_to_jai_part0/) a lot
recently. It documents how the author—now presumed dead—first used D in
2019 as a replacement to C++ for programming video-games. They discuss how they
didn't like the experience of using D, and why they chose to port their code to
Jai, an unfinished language which is not available to the general public. They
neglect to mention the future implications of the language's unfinished
unreleased state for libraryavailability, however.

In the present day it's only nearly two years later, Jai has 
still not materialised a public release despite its seemingly 
avid community, and around half of the problems with D mentioned 
in the post have already been solved. I think this is a great 
indication of the strength of D's community. Despite some recent 
roadblocks in getting new features into the language, it has 
still evolved significantly over the course of a mere two years. 
I'd like to now go over some of the issues the author had with D 
at the time and review them for the sake of appreciating D's 
growth, and analysing some of their misconceptions about the 
language.



 After bouncing between the two available D compilers for 
 Windows (dmd and ldc2) for about 4 years, I find the state of D 
 on Windows to resemble something I would expect from a hobby 
 project, [...]
 From the following problems that I have encountered over the 
 years, by far the biggest one is that debug info on Windows is 
 completely broken:
 [...]
 I was told that “DMD has historically had a lot of issues with 
 [...] debug info” [...] but unfortunately most of [these 
 issues] apply to both compilers, not just dmd.


First of all, GDC is also [available for 
Windows](https://winlibs.com), at least in some capacity.
I cannot speak for the quality of D's compilers on Windows (since 
I have banned it from my house), however I can say that since 
they experienced issues with LDC2 I suspect that a lot of their 
issues with debugging could stem from Visual Studio, or how it 
interfaces with whichever debugger it's using. We're never 
informed whether it was VSCode or the full VS IDE though, so the 
answer remains nebulous.


 - mixins (D’s macro equivalent) generate debug info in a way 
 that causes the debugger to not find the correct file (so you 
 step through disassembly)


The `-mixin` flag has always solved this problem for me, and I 
don't see why it wouldn't work in Windows.


 [...] there are other issues and shortcomings, to a significant 
 part in metaprogramming:
 - different compiler phases interact in weird ways that lead to 
 surprises in metaprogramming, while generating misleading 
 errors [...] like order independent declaration in global 
 scopes sometimes breaking with `mixin`s


I feel that this is to be expected, and this behaviour can be 
easily logically identified with [a bit of 
knowledge](https://dlang.org/spec/intro.html#phases-of-compilation). A compiler
is not magical, it cannot know things it has not gone out and discovered yet.
Usually D doesn't require forward declaration because it does a pass that
alleviates the need for it, but without giving mixins their own pass, they
naturally result in sequentially parsed code. D's backend design has a strong
preference for shorter compile times, and adding another pass would be a huge
detriment to compile times.
D compilers definitely have an error message problem, though. 
Since this post was written many previously confusing error 
messages (especially for missing semicolons & closing 
parenthesis) have gotten a lot better, but you'll always get 
insane error messages from the depths of a fever dream when the 
compiler encounters a fringe error (or possibly even a bug) 
inside of a 7-layer nested mixin from a CTFE lambda inside an 
instantiation of an aliased template.


 or like namespaced names not resolving in mixins unless you use 
 a [special kind of string 
 literal](https://dlang.org/spec/lex.html#token_strings)


This is a very strange one. First of all, D does not really use 
'namespaces', instead we use have scopes. What I *assume* they 
mean is fully qualified identifiers, but I cannot find any 
evidence for anything like this in the history of the D 
specification for mixins or token strings. The other possibility 
is that they're using D's 'namespace' variant of `extern(C++)` 
(which I do not recommend—use the string version instead), since 
they mention C++ interoperability. Perhaps one of those was once 
an implementation detail, but it's also a patently illogical 
idea. Mixins parse strings into a regular D 
expressions/declarations, so why would they ever have a special 
case like this?


 or like `static foreach` not being able to insert multiple 
 `else if`s after an `if`


Here we see a fundamental misunderstanding of D's 
meta-programming. The body of a `static foreach` inlines a series 
of [Declaration 
Statement](https://dlang.org/spec/statement.html#declaration-statement)s. None
of D's meta-programming facilities allow you to write incomplete
declarations/expressions the way that C's macros do, because of the disastrous
effects this has had on C's readability. If you want to write incomplete
declarations like having `if` and `else if` declared with meta-programming, use
string concatenation with a mixin:
```d
void main(){
	int n;
	import std.writeln;
	mixin((string[] list){
		string ret = "if(n == "~list[0]~"){\nwriteln(`"~list[0]~"`);\n";
		foreach(item; list[1..$]){
			ret ~= "}else if(n == "~item~"){\nwriteln(`"~item~"`);\n";
		}
		return ret ~ "}";
	}(["1", "2", "3"]));
}
```


 ldc2 is awfully slow to compile:
 It takes 1 minute in debug and over 5 minutes in release mode 
 to compile the game. There’s also the issue of ldc2 somehow 
 needing 8GB of RAM in debug and 11.5GB in release mode for 
 compiling a 20MB executable from 2MB of source files, thus 
 causing regular near-death experiences for my laptop.


When hearing about compile times this slow, and using this much 
memory, the first thing that comes to mind is **severe** template 
abuse, which is a compile time bottleneck I've experienced with 
LDC2 before, especially when building for release. What's the 
solution? First of all, acknowledge that whenever you use 
meta-programming, you're making the compiler **run** that code, 
so if that code takes a long time to execute then that will 
necessarily inflate your compile times. No compiler design can 
fix that. A big part of the issue for me was re-instantiating 
template functions in nested `static foreach` loops. Instead, I 
would recommend using CTFE functions that create one big mixin.


 but [ldc2 is] sometimes the only choice because dmd has bugs:
 I’ve encountered a couple of bugs over the years that stopped 
 my program from building, the latest one being this [codegen 
 bug](https://issues.dlang.org/show_bug.cgi?id=23195) when 
 interfacing with C++.


The bug mentioned has been [fixed for some time 
now](https://github.com/dlang/dmd/pull/14651#event-8476344353), 
and importantly, was introduced due to an error in Microsoft's 
own documentation about the calling convention.


 D offers a *betterC* mode that among other things disables 
 garbage collection. However, when using this mode, the standard 
 library does not compile and meta programming is significantly 
 hampered:
 Disabling the garbage collector not only disables it for your 
 compiled code, but also for the code executing at compile time. 
 One major way of metaprogramming in D is injecting new code 
 into the program. That code is a string, and those strings need 
 to be concatenated, and string concatenation uses the garbage 
 collector. So … it just doesn’t compile.


There is quite the errata in this section. Specifically, BetterC 
replaces DRuntime with a wrapper that hooks into C's runtime, 
which makes some basic DRuntime-dependant features like `assert` 
still work. The GC is part of DRuntime, so out it goes. 
Phobos—D's standard library—is *partially* unavailable, with most 
meta-programming facilities (like `std.traits`/`std.meta`) still 
being available.
You might've noticed a discrepancy there. Why would removing the 
garbage collector from the *runtime* stop GC being used at 
*compile-time*? The compiler's interpreter isn't using the same 
runtime as our generated run-time code, right? No, of course it's 
not. D compilers don't even use the GC at compile-time by 
default, although it can be enabled for lower memory consumption. 
What this person has clearly encountered is that run-time 
functions that use the GC/DRuntime naturally don't compile with 
BetterC (or produce linker errors) and so they have naturally 
concluded that CTFE string concatenation is impossible with 
BetterC. This is simply wrong. When you write a run-time function 
(**whether or not** you use it in run-time code) the compiler 
**will** try to generate code for it. There's a way to make 
CTFE-only code-paths in a function, but this feature only works 
with `if`, not `static if`, so it's not useful to us here. What 
you need to do is create a function that the compiler won't try 
to generate run-time code for. You need a lambda:
```d
//not a lambda, will generate run-time code:
string catRT(string a, string b) => a ~ b; //Error: array 
concatenation of expression `a ~ b` requires the GC which is not 
available with -betterC

//a lambda assigned to an enum:
enum catCT = (string a, string b) => a ~ b; //OK
```
Of course, if you call this lambda in run-time code it will 
produce an error (from linking, specifically).


 Bad debug info and the de-facto need for garbage collection are 
 dealbreakers


Garbage collection being a deal-breaker is weird to me as a D 
user, but somewhat understandable for real-time speed-critical 
applications. However, D is a perfectly capable language without 
DRuntime, as you can easily write a custom templated 'array' 
struct that overloads `~` and uses existing C/C++ libraries 
instead of Phobos. If you don't care about executable size, just 
mix GC with manual memory allocation. You'll get to use a lot 
more D libraries that way. Unless you pick a good allocator or 
swap a lot of large long-term GC allocations to manual ones (to 
reduce GC heap scanning) your net time savings will likely be ~0 
anyway.

See this excellent write-up about the matter from Bit Bashing for 
more: [Garbage Collection for Systems 
Programmers](https://bitbashing.io/gc-for-systems-programmers.html)


 **Why is there a single language without the ability to pass 
 parameters by name in 2022??**


The [DIP for named 
parameters](https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1030.md)
was accepted back in 2020, so anyone could've seen it coming, but now we've
finally got our hands on them! Hurrah!

And that's all! If you like long blog posts about D I recommend 
reading some from [Bit Bashing](https://bitbashing.io), and [The 
Art of Machinery](https://theartofmachinery.com).
Jul 13
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Thanks for taking the time to write an interesting analysis!
Jul 13
prev sibling next sibling parent reply Ruby The Roobster <michaeleverestc79 gmail.com> writes:
Good post, but there is one issue:  GDC is NOT available for 
windows in any capacity.  It apparently got removed from winlibs 
a while back, for whatever reason, and is still removed to this 
day.
Jul 14
parent IchorDev <zxinsworld gmail.com> writes:
On Sunday, 14 July 2024 at 13:38:27 UTC, Ruby The Roobster wrote:

 Good post, but there is one issue:  GDC is NOT available for 
 windows in any capacity.  It apparently got removed from 
 winlibs a while back, for whatever reason, and is still removed 
 to this day.


Oh, thanks for the correction! Their website mentioned D so I was 
inclined to believe it, but I guess it would hardly be the first 
time I see a website for a FOSS project that’s incredibly out of 
date. (For example: GDC’s own website)
Jul 14
prev sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Saturday, 13 July 2024 at 16:33:25 UTC, IchorDev wrote:

 This is going to be a long one.

 I have been thinking about [this blog 
 post](https://www.yet-another-blog.com/porting_the_game_to_jai_part0/) a lot
recently. It documents how the author—now presumed dead—first used D in
2019 as a replacement to C++ for programming video-games. They discuss how they
didn't like the experience of using D, and why they chose to port their code to
Jai, an unfinished language which is not available to the general public. They
neglect to mention the future implications of the language's unfinished
unreleased state for libraryavailability, however.

 [...]


Thanks for the summary
Jul 14