www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Discussion: Porting 58000 lines of D and C++ to jai, Part 0: Why and

reply Kyle Ingraham <kyle kyleingraham.com> writes:
Hi all. I came across this article on Hacker News: 
https://www.yet-another-blog.com/porting_the_game_to_jai_part0/

I’m interested in reading your thoughts on the ‘Why not D’ 
section. Anyone have experiences that match the specific points 
there? I haven’t experienced them but I also don’t debug on 
Windows. I also find D’s documentation to be excellent.

I’m super-positive on D but also interested in whether there are 
others with the same issues as the article’s author.
Nov 23 2022
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 24/11/2022 6:20 AM, Kyle Ingraham wrote:

 I’m interested in reading your thoughts on the ‘Why not D’ section. 
 Anyone have experiences that match the specific points there? I haven’t 
 experienced them but I also don’t debug on Windows. I also find D’s 
 documentation to be excellent.


I haven't experienced the exact set of issues listed, but yeah debugging 
can be quite janky even when it works. If I can help it I will not start 
up VS to debug, it messes with window order OS wide, not fun.


 I’m super-positive on D but also interested in whether there are others 
 with the same issues as the article’s author.


I'm quite surprised that the issues surrounding shared libraries wasn't 
mentioned. That's the big one I'm facing. Windows is basically a write 
off for dmd and there isn't much support on the compiler dev side to fix 
this.
Nov 23 2022
prev sibling next sibling parent 12345swordy <alexanderheistermann gmail.com> writes:
On Wednesday, 23 November 2022 at 17:20:54 UTC, Kyle Ingraham 
wrote:

 Hi all. I came across this article on Hacker News: 
 https://www.yet-another-blog.com/porting_the_game_to_jai_part0/

 I’m interested in reading your thoughts on the ‘Why not D’ 
 section. Anyone have experiences that match the specific points 
 there? I haven’t experienced them but I also don’t debug on 
 Windows. I also find D’s documentation to be excellent.

 I’m super-positive on D but also interested in whether there 
 are others with the same issues as the article’s author.


I have with regards to debugging on windows. It is unironically 
so terrible that I have to stop using the d programming language 
all together. The difficulty in fixing requires too much time 
from me and it a requires a full time developer just to fix this. 
I can't recommend programming d on windows platform for serious 
work, due to this major issue.

- Alex
Nov 23 2022
prev sibling next sibling parent reply Johan <j j.nl> writes:
On Wednesday, 23 November 2022 at 17:20:54 UTC, Kyle Ingraham 
wrote:

 Hi all. I came across this article on Hacker News: 
 https://www.yet-another-blog.com/porting_the_game_to_jai_part0/

 I’m interested in reading your thoughts on the ‘Why not D’ 
 section. Anyone have experiences that match the specific points 
 there? I haven’t experienced them but I also don’t debug on 
 Windows. I also find D’s documentation to be excellent.


About the compilation speed, I think the author does not yet 
realize that you have to think about performance of compile-time 
execution, similarly as with runtime execution. The algorithm 
used matters a lot, also at compile time. Spending 60sec on that 
small of a codebase is not my experience (*), so I think it's 
just a piece of compile-time code that is consuming most of the 
time (note that his optimized build only takes 2x longer).

-Johan

(*) >10 times larger linecount project with lots of compile-time 
stuff and template complexity, 1.8GB main executable binary in 
debug mode, about ten 100MB executables, just tested: builds in 
300seconds. Just semantic analysis (compile without machine 
codegen) takes 130seconds.
Nov 23 2022
parent TheGag96 <thegag96 gmail.com> writes:
On Wednesday, 23 November 2022 at 19:42:22 UTC, Johan wrote:

 About the compilation speed, I think the author does not yet 
 realize that you have to think about performance of 
 compile-time execution, similarly as with runtime execution. 
 The algorithm used matters a lot, also at compile time. 
 Spending 60sec on that small of a codebase is not my experience 
 (*), so I think it's just a piece of compile-time code that is 
 consuming most of the time (note that his optimized build only 
 takes 2x longer).


I'm very interested in seeing how using Jai turns out for this 
guy, but I was wondering about this too... Just what kind of code 
has he been writing to cause D to take a whole minute to 
compile?? Is he compiling each D file separately? I can 
understand the rest of his criticisms but I think this problem 
may be solely his own, even if our CTFE engine and template 
instantiation isn't the fastest.
Nov 24 2022
prev sibling next sibling parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Wednesday, 23 November 2022 at 17:20:54 UTC, Kyle Ingraham 
wrote:

 Hi all. I came across this article on Hacker News: 
 https://www.yet-another-blog.com/porting_the_game_to_jai_part0/

 I’m interested in reading your thoughts on the ‘Why not D’ 
 section. Anyone have experiences that match the specific points 
 there? I haven’t experienced them but I also don’t debug on 
 Windows. I also find D’s documentation to be excellent.

 I’m super-positive on D but also interested in whether there 
 are others with the same issues as the article’s author.


Some valid criticisms, specially regarding to debugging, it has 
improved so there is that


 Reducing noise in the code due to various syntax improvements


This one i agree too, it's nice when the language allows you to 
be expressive without being noisy

The Enum Type Inference DIP is a move in the right direction imo


 Replacing build-scripts with jai code


I also like that idea, i wonder if dub could support something 
like it, at the end of the day, dub just parse a json into a 
struct

```D
// build.d

import dub;

void build(Builder* builder)
{
     builder.target_type = .exe;
     builder.target_name = "game";

     auto config = builder.create_configuration("config_a");
     config.(..)

     config.prebuild_commands ~= () {
         // do stuff
     };
}

```

And then dub could just run that function directly instead of 
parsing a json


 Reducing compile times from about 60s right now to under 5s, 
 hopefully around 1s


This one i find suspicious and very surprising

My 30+k LOC game rebuilds fully in just 1.2 sec, i don't touch 
phobos and i'm light on templates

On picks D because it's fast at compilation, so by the time it 
reaches 60s i wonder why he didn't think there is a problem 
somewhere before... like at 2-5 seconds it already is slow, at 
that time you need to guess that the code you wrote is cat piss 
and it is time to reflect on what you are doing..

I suggest that autho to stick to D for fast iteration and to ping 
the forum whenever there is a bug that's blocking you...

That's why it's important to be pro active, most users won't 
complain about issues, they'll just move on..

Programming languages should:

- be easy to debug out of the box
- be easy to write/read without being verbose
- be fast at compilation
- be fast at runtime

Once my game will be at a visually pleasant state, i'll write a 
proper blog post, i'll prove these people wrong
Nov 23 2022
next sibling parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Wednesday, 23 November 2022 at 21:17:42 UTC, ryuukk_ wrote:

 I suggest that author to stick to DMD (..)


Damn, typos..
Nov 23 2022
parent Sergey <kornburn yandex.ru> writes:
On Wednesday, 23 November 2022 at 21:20:17 UTC, ryuukk_ wrote:

 On Wednesday, 23 November 2022 at 21:17:42 UTC, ryuukk_ wrote:

 I suggest that author to stick to DMD (..)


 Damn, typos..


He wrote that DMD is not acceptable for him, that the debug info 
in DMD is just wrong somehow.. “DMD has historically had a lot of 
issues with the quality of its debug info”
I think that's why the proposal: for development iteration use 
DMD, for production LDC is not suitable for his code and his case.
Nov 23 2022
prev sibling parent Kyle Ingraham <kyle kyleingraham.com> writes:
On Wednesday, 23 November 2022 at 21:17:42 UTC, ryuukk_ wrote:

 Once my game will be at a visually pleasant state, i'll write a 
 proper blog post, i'll prove these people wrong


I’d definitely give that a read. Looking forward to it 👍.
Nov 23 2022
prev sibling next sibling parent reply Guillaume Piolat <first.last spam.org> writes:
On Wednesday, 23 November 2022 at 17:20:54 UTC, Kyle Ingraham 
wrote:

 I’m super-positive on D but also interested in whether there 
 are others with the same issues as the article’s author.



I'm using VisualD and yeah it has become a bit strange over time, 
with debugging values being shown wrong, it also depends what's 
your compiler and whether youa re using Mago or isual studio 
debugger, breakpoints sometimes not triggering unless you 
rebuild... I'm pretty sure it used to be a bit better.

... but the _last_ thing I would do is rewrite a whole codebase 
just for such a reason. It's not even clear Jai has a Visual 
Studio extension at all.
Nov 23 2022
parent reply Kyle Ingraham <kyle kyleingraham.com> writes:
On Wednesday, 23 November 2022 at 22:07:01 UTC, Guillaume Piolat 
wrote:

 On Wednesday, 23 November 2022 at 17:20:54 UTC, Kyle Ingraham 
 wrote:

 I’m super-positive on D but also interested in whether there 
 are others with the same issues as the article’s author.



 I'm using VisualD and yeah it has become a bit strange over 
 time, with debugging values being shown wrong, it also depends 
 what's your compiler and whether youa re using Mago or isual 
 studio debugger, breakpoints sometimes not triggering unless 
 you rebuild... I'm pretty sure it used to be a bit better.

 ... but the _last_ thing I would do is rewrite a whole codebase 
 just for such a reason. It's not even clear Jai has a Visual 
 Studio extension at all.


When you say ‘shown wrong’ do you mean they’re hard to read 
(wrong format, mangled names) or that the values are wrong? Do 
you know if there are bugs reported for that problem?
Nov 23 2022
parent Guillaume Piolat <first.last spam.org> writes:
On Wednesday, 23 November 2022 at 23:44:48 UTC, Kyle Ingraham 
wrote:

 When you say ‘shown wrong’ do you mean they’re hard to read 
 (wrong format, mangled names) or that the values are wrong?


Wrong.
btw this sort of things also happens in C++


 Do you know if there are bugs reported for that problem?


I don't know. When this happen, I change the debugging engine or 
rebuild, and it makes it work. I also have disabled semantic 
analysis and completion because it crashed often or took too much 
memory.
Nov 24 2022
prev sibling next sibling parent reply IGotD- <nise nise.com> writes:
On Wednesday, 23 November 2022 at 17:20:54 UTC, Kyle Ingraham 
wrote:

 Hi all. I came across this article on Hacker News: 
 https://www.yet-another-blog.com/porting_the_game_to_jai_part0/

 I’m interested in reading your thoughts on the ‘Why not D’ 
 section. Anyone have experiences that match the specific points 
 there? I haven’t experienced them but I also don’t debug on 
 Windows. I also find D’s documentation to be excellent.

 I’m super-positive on D but also interested in whether there 
 are others with the same issues as the article’s author.


Just a quick question about Jai, can anybody obtain the Jai 
compiler today?
Nov 23 2022
parent Kyle Ingraham <kyle kyleingraham.com> writes:
On Wednesday, 23 November 2022 at 22:29:24 UTC, IGotD- wrote:

 On Wednesday, 23 November 2022 at 17:20:54 UTC, Kyle Ingraham 
 wrote:

 Hi all. I came across this article on Hacker News: 
 https://www.yet-another-blog.com/porting_the_game_to_jai_part0/

 I’m interested in reading your thoughts on the ‘Why not D’ 
 section. Anyone have experiences that match the specific 
 points there? I haven’t experienced them but I also don’t 
 debug on Windows. I also find D’s documentation to be 
 excellent.

 I’m super-positive on D but also interested in whether there 
 are others with the same issues as the article’s author.


 Just a quick question about Jai, can anybody obtain the Jai 
 compiler today?


It looks like it’s accessible through a closed beta: 
https://github.com/Jai-Community/Jai-Community-Library/wiki/Overview#history
Nov 23 2022
prev sibling next sibling parent reply Hipreme <msnmancini hotmail.com> writes:
I'm developing my engine literally on Windows since the start, 
and I'm even doing Xbox development for it, so, I think my 
experience with D should prove useful.



 90% of times, the this pointer is missing or wrong


- The only time I found `this` missing was in a bug I just sent 
on issues.dlang: when using with anonymous classes inside 
functions, that seems to be a bug on DMD itself.


 variables on function stacks are often partially or completely 
 missing or wrong


- Never here


 values of variables are sometimes reported wrong without any 
 other visible issues


- Nope



 static foreach expansions are not handled well and confuse the 
 debugger


- I haven't actually debugged this kind of code


 mixins (D’s macro equivalent) generate debug info in a way that 
 causes the debugger to not find the correct file (so you step 
 through disassembly)


- Or you can use -mixin=targetFileToMixin.d, which makes it a lot 
easier to understand what is causing that. All the 
metaprogramming facilities in D doesn't work correctly in all its 
tooling such as code-d or VisualD.


 moving the instruction pointer back to the previous line in 
 visual studio often crashes the program on the next instruction


- I would not play with instruction pointer, so unsafe to do in 
any language



 different compiler phases interact in weird ways that lead to 
 surprises in metaprogramming, while generating misleading 
 errors234


- Yes, I actually got a problem like that when dealing with 
libraries, static if + versions.


 ldc2 is awfully slow to compile5, but sometimes the only choice 
 because dmd has bugs6


- Currently for me it is quite the opposite, I can't build DLLs 
correctly with LDC, and when building with LDC, my writeln/printf 
becomes really buggy.



 D offers a betterC mode that among other things disables 
 garbage collection. However, when using this mode, the standard 
 library does not compile and meta programming7 is significantly 
 hampered.


- BetterC bad


 the documentation is lacking


- Which documentation? Most of the time I could not find was when 
I didn't read the documentation correctly and someone was able to 
point to me



 like order independent declaration in global scopes sometimes 
 breaking with mixins ↩︎


- I have documented a bug related to that


 or like namespaced names not resolving in mixins unless you use 
 a special kind of string literal ↩︎


- Never had


 or like static foreach not being able to insert multiple else 
 ifs after an if ↩︎


- Agreed, if you can put `case` with static foreach, you should 
be able to put `else if`


*Summary*:
I can see the guy over there made 58KLoc of pure template, 
metaprogramming and CTFE abuse. 60 seconds of build time is just 
unacceptable. I do believe though that a project like that is a 
nice case study for D to evolve itself.

Unfortunately, it is sad to see that D is quite in that state on 
its unique features, like being advertising with CTFE and not 
being able to actually use CTFE because it can slow the build 
quite a lot (there was a CTFE cache PR somewhere, saying that it 
was able to make it build much faster, unfortunately, it didn't 
go onwards dunno why).


Although I would not switch D for another language, I don't 
completely disagree with what is said in that blog. And that is 
what I and some other people says: adding new features to D 
should not be priority n° 1, cuz there is a lot of other broken 
things over there that just ruins the developer experience.

Having a clear and better compilation flow with version/static 
if/mixin template/mixin, generating better debug information on 
metaprogramming side (which could also help tooling such as 
code-d and visual D), and after that, making CTFE faster (or 
cached) could make D better in what D offers.

If you code D like C or Java, you won't have much pain with it, 
but if you start using mixin template/alias and other features 
such as that, you'll start to be bitten.
Nov 24 2022
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Thursday, 24 November 2022 at 11:37:57 UTC, Hipreme wrote:

 - Agreed, if you can put `case` with static foreach, you should 
 be able to put `else if`


This is not a case of "if doesn't work with this normal feature", 
it's a case of "case is extremely, **extremely** strange." Case 
statements work much more like goto labels than statements. For 
instance, this is valid code:

     import std.stdio;
     void switchtest(int i, bool b)
     {
         switch (i)
         {
             case 1:
                 if (b)
                 {
                     case 2:
                         writefln!"Case 1b or 2: %s, %s"(i, b);
                         return;
                 }
                 break;
             default: break;
         }
         assert(false);
     }

     void main()
     {
         switchtest(1, true);
         switchtest(2, false);
     }
Nov 24 2022
next sibling parent FeepingCreature <feepingcreature gmail.com> writes:
On Thursday, 24 November 2022 at 12:04:27 UTC, FeepingCreature 
wrote:

 For instance, this is valid code:

     import std.stdio;
     void switchtest(int i, bool b)
     {
         switch (i)
         {
             case 1:
                 if (b)
                 {
                     case 2:
                         writefln!"Case 1b or 2: %s, %s"(i, b);
                         return;
                 }
                 break;
             default: break;
         }
         assert(false);
     }

     void main()
     {
         switchtest(1, true);
         switchtest(2, false);
     }


Ie. to explain what's happening here:

The syntax tree structure of `if/else` is `if (EXPRESSION) 
STATEMENT [else STATEMENT]`. `static foreach` does not match 
`else`, so it fails to parse.

The syntax tree structure of `switch/case` is not, as you would 
expect, `switch (EXPRESSION) { [case EXPRESSION: STATEMENT*]* }`.

Instead, it's `switch (EXPRESSION) STATEMENT` and `STATEMENT = 
... | CASE_STATEMENT`, `CASE_STATEMENT = case EXPRESSION:`, where 
the CASE_STATEMENT is *semantically* (not syntactically) verified 
to occur inside `switch`, with no other restrictions on their 
placement. That's the only reason it works with `static foreach`.

(All parse rules made up from memory, not taken from the spec.)
Nov 24 2022
prev sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On Thursday, 24 November 2022 at 12:04:27 UTC, FeepingCreature 
wrote:

 On Thursday, 24 November 2022 at 11:37:57 UTC, Hipreme wrote:

 - Agreed, if you can put `case` with static foreach, you 
 should be able to put `else if`


 This is not a case of "if doesn't work with this normal 
 feature", it's a case of "case is extremely, **extremely** 
 strange." Case statements work much more like goto labels than 
 statements. For instance, this is valid code:


See also [Duff's 
device](https://en.wikipedia.org/wiki/Duff%27s_device).
Dec 02 2022
prev sibling next sibling parent cc <cc nevernet.com> writes:
On Wednesday, 23 November 2022 at 17:20:54 UTC, Kyle Ingraham 
wrote:

 I’m interested in reading your thoughts on the ‘Why not D’ 
 section. Anyone have experiences that match the specific points 
 there? I haven’t experienced them but I also don’t debug on 
 Windows. I also find D’s documentation to be excellent.


Does this make sense to anyone without elaboration:

 90% of times, the `this` pointer is missing or wrong


Can't say I've ever noticed anything like that suddenly springing 
up, certainly not *90% of the time*.
Nov 24 2022
prev sibling next sibling parent reply Dukc <ajieskola gmail.com> writes:
On Wednesday, 23 November 2022 at 17:20:54 UTC, Kyle Ingraham 
wrote:

 Hi all. I came across this article on Hacker News: 
 https://www.yet-another-blog.com/porting_the_game_to_jai_part0/

 I’m interested in reading your thoughts on the ‘Why not D’ 
 section. Anyone have experiences that match the specific points 
 there? I haven’t experienced them but I also don’t debug on 
 Windows. I also find D’s documentation to be excellent.


That article has an interesting take. It's gripes with D aren't 
the usual ones, maybe debugging experience excluded.

D's documentation is good but not top-notch IMO. It's easy to 
browse and usually has examples, but still sometimes lacks detail 
with regards to corner cases. It's still better than most, so I 
find complaining about that surprising. Compare Phobos docs to 
[Nix standard API 
docs](https://nixos.org/manual/nixpkgs/stable/#preface) for 
instance. At least for me it's far easier to find what I want 
from Phobos docs, yet even NixPkgs documentation should probably 
be considered acceptable - at least it is correct and usually 
covers what you need when you look hard enough.

I agree with others here that he must have some problem with the 
compile time he could solve without switching the language. D 
usually compiles far faster than C++. It may be because of his 
memory usage, perhaps that leads to excessive disk swapping. 
Should be solvable too.
Nov 24 2022
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Friday, 25 November 2022 at 07:31:59 UTC, Dukc wrote:

 I agree with others here that he must have some problem with 
 the compile time he could solve without switching the language. 
 D usually compiles far faster than C++. It may be because of 
 his memory usage, perhaps that leads to excessive disk 
 swapping. Should be solvable too.


D has a tendency to give you lots of features, invite you to 
commit to them, then punish you somewhere way down the line when 
you can't back out. For instance, format!`` is very clever, but 
it's probably not beneficial for compile speed to have thousands 
of recursive instantiations of the formatting code in the 
compilation set.
Nov 25 2022
next sibling parent Dukc <ajieskola gmail.com> writes:
On Friday, 25 November 2022 at 08:23:22 UTC, FeepingCreature 
wrote:

 On Friday, 25 November 2022 at 07:31:59 UTC, Dukc wrote:

 I agree with others here that he must have some problem with 
 the compile time he could solve without switching the 
 language. D usually compiles far faster than C++. It may be 
 because of his memory usage, perhaps that leads to excessive 
 disk swapping. Should be solvable too.


 D has a tendency to give you lots of features, invite you to 
 commit to them, then punish you somewhere way down the line 
 when you can't back out. For instance, format!`` is very 
 clever, but it's probably not beneficial for compile speed to 
 have thousands of recursive instantiations of the formatting 
 code in the compilation set.


Still, by the time your compile times are getting longer but are 
still tolerable, you can investigate and stop generating further 
template bloat. The compile times don't jump from 3 sec to a 
minute overnight, unless it's something that can be fixed easily.

Of course, if the problem of the blog poster is having continued 
to soldier on with compile-time format or regexp or similar, then 
there might not be an easy fix. Still easier than switching 
language though.
Nov 25 2022
prev sibling parent FeepingCreature <feepingcreature gmail.com> writes:
On Friday, 25 November 2022 at 08:23:22 UTC, FeepingCreature 
wrote:

 On Friday, 25 November 2022 at 07:31:59 UTC, Dukc wrote:

 I agree with others here that he must have some problem with 
 the compile time he could solve without switching the 
 language. D usually compiles far faster than C++. It may be 
 because of his memory usage, perhaps that leads to excessive 
 disk swapping. Should be solvable too.


 D has a tendency to give you lots of features, invite you to 
 commit to them, then punish you somewhere way down the line 
 when you can't back out. For instance, format!`` is very 
 clever, but it's probably not beneficial for compile speed to 
 have thousands of recursive instantiations of the formatting 
 code in the compilation set.


Update: Correction! I've benchmarked it and this surprisingly 
appears to not actually make much difference. A lot of it seems 
to be going to boilerplate, unsurprisingly.

I wish `-vtemplates` gave you sum compiler time spent per 
template.
Nov 25 2022
prev sibling parent Kyle Ingraham <kyle kyleingraham.com> writes:
On Wednesday, 23 November 2022 at 17:20:54 UTC, Kyle Ingraham 
wrote:

 Hi all. I came across this article on Hacker News: 
 https://www.yet-another-blog.com/porting_the_game_to_jai_part0/

 I’m interested in reading your thoughts on the ‘Why not D’ 
 section. Anyone have experiences that match the specific points 
 there? I haven’t experienced them but I also don’t debug on 
 Windows. I also find D’s documentation to be excellent.

 I’m super-positive on D but also interested in whether there 
 are others with the same issues as the article’s author.


Thanks everyone for the replies. The context they provided was a 
great compliment to the article. There are some good learnings 
here too.
Nov 29 2022