www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - The Future of D Runtime

reply Adam Wilson <flyboynw gmail.com> writes:
History and Problem
-------------------

Now that the Editions DIP is up public comment it is time to have 
a frank discussion about where the D Runtime (DRT for the 
remainder) fits into the puzzle that Editions introduces into the 
D ecosystem. The reason I bring this up is that it appears that, 
at present, most people treat DRT as a particularly vexing 
extension of the compiler that nobody wants to touch. As a 
result, the presumption seems to be that, with the advent of 
editions, we should just freeze DRT wherever it is when the first 
edition drops and never touch it again because doing so might 
break something across editions. While this position is certainly 
expedient, I would argue that this position is going to severely 
limit the future of Phobos and D.

Furthermore, I would argue that this position makes a presumption 
that all DRT should ever be is a compiler extension library. I 
propose that this view is both incorrect and a significant 
limitation on what we can do with Phobos 3 and even future 
editions.

I want to call back to DConf 2023 where I spent much of 
conference working to undo the deprecation of the ODBC bindings 
in Phobos because they were being replaced with `version 
(Windows)` bindings in DRT, which is utter nonsense, ODBC can be 
used on POSIX quite easily (and I do every day). The explanation 
I was given as to why this was done is that the bindings have 
nothing to do with system resources, but because the bindings 
were provided by DRT they should be removed from Phobos because 
Phobos is not in the business of providing library bindings.

This creates a rule for Phobos. Phobos does not expose system 
resources or bindings. IMO this is a good rule. Phobos is 
bespoke, and that is the proper situation. The standard library 
really should not be in the business of exposing system level 
constructs.

Then why not use the DRT bindings? Because the bindings in DRT 
are automatically generated and are fundamentally broken because 
of it. At the time it was noted to me that those bindings really 
should not be in DRT *either* because DRT isn’t supposed to 
provide access to non-System resources like ODBC and they were 
only there by accident because the ODBC headers got swept into 
the auto-generation process.

This creates a rule for DRT. DRT is only to expose system 
resources, nothing more. And indeed, this is a good rule as well. 
The purpose of a runtime is to expose system resources in a 
uniform manner, and to the extent that DRT does this, it does a 
fantastic job.

Currently we call DRT a runtime, but when you catalog what it 
does, it’s really a “compiler support library that provides some 
runtime services.” There is no argument from me that the GC is a 
runtime service, the same for threads, but the rest of the listed 
items that DRT provides are very definitely aimed at enabling the 
compiler to do things like AA’s for example. AA’s are a feature 
that is only accessible from the language hence “compiler 
support”.

The most often given reason for keeping DRT as-is is that “we 
only need to port DRT to a new platform, Phobos is platform 
independent.” I submit that this is factually incorrect. There 
are currently 31 files in Phobos with `version (WinXXXX)` 
statements in them. 14 with `version (linux)`. And IIRC everybody 
who has tried a port D to a new platform recently has just not 
bothered to port Phobos at all. To be sure, some of these version 
statements are benign, but take a look at `std.stdio`, I wish 
whoever wants to port that much luck in their endeavors. It has 
32 instances `version (WinXXXX)` alone. And that is our basic I/O 
module.

Runtimes are not compiler support libraries, they are 
application-system interfaces. The .NET runtime does much more 
than threads and the GC. The Java runtime does much more than 
threads and the GC. The C runtime does much more than memory 
management. I could go on, but you get the point.

The purpose of a runtime is to provide a system resource access 
layer for the standard library. Consider the C Runtime and the C 
Standard Library. The C Runtime provides the system specific 
implementation, and the Standard Library provides the 
standardized application interface. No matter what system you are 
on, calling `printf` will print a string to the terminal via the 
runtime. If you get a `FILE*` you will work with a file on disk 
no matter what filesystem the system is using. These are not 
compiler supports. They are a universal system-application 
interface.

An example of why this matters is found in `std.stdio`. We 
currently use the C Runtime to do basic I/O operations. While 
that works and is expedient, it limits us. For example, doing 
colored terminal output is so difficult nobody does it, or, we 
are limited to the C file interface, etc. To gain access to more 
advanced capabilities, we need to use the system API’s, but these 
are diverse, and putting them in Phobos would significantly 
expand the code required and make maintenance a nightmare. What 
is interesting about this is that, in fact, D has two runtimes. 
The C Runtime, which we use as a universal system interface, and 
the DRT which is a bunch of compiler support tools that the CRT 
doesn’t have.

Proposed Design
---------------

My proposal is that we expand DRT into a universal system 
interface for the standard library. DRT would contain low-level 
interfaces for Terminal and File I/O, threads, event-loops, 
sockets, cryptography, anything that would normally access a 
system API. Why go to all this effort? Simple, so that we can 
move beyond the limitations imposed by the C Runtime. Using the 
CRT was certainly expedient in the early days of D, but we’ve 
grown beyond that and it’s time to upgrade our capabilities.

However, this will significantly expand the scope of DRT and will 
make porting harder. I think the answer to this problem came out 
of left field in a conversation I had last Thursday with an 
ex-Google engineer who has no prior knowledge of D. What he said 
was: “you need to shard your runtime.” This statement caused 
several things to “click” and got me started writing this missive.

Before I get into the design of the DRT I want to propose rules 
that will allow us to continue to evolve DRT in the future 
without breaking past editions. From an ABI standpoint, there are 
only three actions that can be taken: Add, Rename, Remove.
1. Remove. Banned. We cannot remove symbols from the DRT API. But 
we can rename them provided we follow the rename rules.
2. Rename. Renames include changing the parameters in any way. We 
can support this by offering either a shim that redirects to the 
new method or leaving the original method alone and building a 
new one next to it.
3. Add. The is probably the easiest. New symbols can be added at 
any time.
If we follow these rules, then we can safely use the *latest* DRT 
version with all prior editions.

For DRT I propose a sharded design with a split between the 
compiler support modules and the universal system interface 
shards. What this means is that there will be a “Core-DRT” that 
only contains things that the compiler needs to function. Then, 
for each subsystem in Phobos, there would be a corresponding DRT 
subsystem that can be built and linked (statically or 
dynamically). This means that the only mandatory component that 
would have to be ported is the “Core-DRT” component for the 
compiler. The porting engineer would then be free to choose to 
port as much or as little of what remains of DRT and know that 
the corresponding Phobos subsystem is guaranteed to work once the 
port is complete. Another to view these components is as an 
“onion” where each one builds on the one above it. Core must 
exist for System, System exists for Cryptography, etc. Below is 
a, likely incomplete, list of potential runtime shards.

Core:
1. Compiler Hooks
2. GC
3. Threads
4. Event Loop (in later iterations)

System:
1. Console/Terminal I/O
2. File I/O

Cryptography:
1. System Cryptography Primitives
2. Optional OpenSSL extended Cryptography Primitives (to support 
what the OS does not).

Network:
1. Sockets
2. SSL/TLS

I’m not married to the above so please destroy. One thing that I 
think would be beneficial for ease-of-porting is researching if 
it would be possible to move Threads and the Event Loop out of 
Core and into the System component then provide a mechanism for 
notifying the GC of thread creation/deletion without actually 
requiring the Threads (and thus the Event Loop) implementation to 
be in the Core DRT so that people porting to systems that don’t 
have Threads (or simply don’t want to support them) don’t have to 
port them to get basic D code running.

The goal of Core is to be the most reduced component of the 
runtime that can still produce a functional program in D. The 
larger goal of this DRT expansion project is to eventually drop 
the requirement for the C Runtime. By doing so we free ourselves 
from the limitations of the CRT, such as being unable to handle 
Console Colors on systems that support them, and we remove a 
layer of abstraction by calling directly into the system API’s. 
This does mean more work for porting in terms of needing to call 
the system API’s directly as opposed to using the CRT, however, 
one option could be to provide an opt-in CRT-enabled build of DRT 
that uses the CRT for a limited set of features.

I want to point out that these are ideas intended to address the 
limitations of the current situation and the proposal to make 
changes is a result of looking at the work we’re going to have to 
do with Phobos 3. I fully understand that I can be wrong on 
things here, but the present situation with DRT is not going to 
remain tenable as we move forward with expanding Phobos.

Another angle that might be worth pursuing is building DRT up 
into a Runtime that can be used by many languages, similar to how 
the CRT is used. It would also increase the appeal of D to folks 
looking for a platform to start building a language on, like how 
an ARM backend will help the hobbyist language community. I 
understand that LLVM and GCC are there, but they are not simple 
to work with. If we can provide an ecosystem that is simple to 
work with for language experimentation and development, we can 
capture people looking for a tool to start designing languages 
quickly and still provide them a path to integrate with LLVM or 
GCC via our existing integrations with those tools.

These are ideas, not directions, so please destroy.
Jun 05
next sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
The GC directly depends upon a thread abstraction to provide suspending 
of threads. This requires on Posix to setup signals on the thread. For 
these reasons it must go together.

It wouldn't be a bad idea to have both a C threads implementation and a 
singleton one.
Jun 05
prev sibling next sibling parent Dukc <ajieskola gmail.com> writes:
Adam Wilson kirjoitti 6.6.2024 klo 2.58:

 Proposed Design
 ---------------
 
 My proposal is that we expand DRT into a universal system interface for 
 the standard library. DRT would contain low-level interfaces for 
 Terminal and File I/O, threads, event-loops, sockets, cryptography, 
 anything that would normally access a system API. Why go to all this 
 effort? Simple, so that we can move beyond the limitations imposed by 
 the C Runtime. Using the CRT was certainly expedient in the early days 
 of D, but we’ve grown beyond that and it’s time to upgrade our 
 capabilities >


I very much like the general idea here. I just don't think we should 
consider non-core shards as part of what we call the runtime. I believe 
"runtime" means what you call "compiler support". A monolithic runtime 
can also be a platform abstraction for the standard library like you 
write, but that's not what makes it a runtime.

I think the core DRuntime should have those stability guarantees you 
write about, and be considered part of the language, not of the standard 
library. But for other shards, I think they should either be part of 
Phobos, or alternatively a separate component that isn't DRuntime nor 
part of the spec/compiler, but not of Phobos either. If they have a 
similar non-removal guarantee as the DRuntime like you suggested, 
there's no need to version them like Phobos will be versioned between 2 
and 3.
Jun 06
prev sibling next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Wednesday, 5 June 2024 at 23:58:14 UTC, Adam Wilson wrote:

 [...]


I think DRT only needs to concern itself with supporting language 
features. Anything else needs to go elsewhere.
Jun 06
next sibling parent reply max haughton <maxhaton gmail.com> writes:
On Thursday, 6 June 2024 at 18:00:56 UTC, Sebastiaan Koppe wrote:

 On Wednesday, 5 June 2024 at 23:58:14 UTC, Adam Wilson wrote:

 [...]


 I think DRT only needs to concern itself with supporting 
 language features. Anything else needs to go elsewhere.


I agree. If nothing else because you can start a new phobos 
yesterday whereas making changes to druntime has all kinds of 
other high-dimensional treadeoffs to faff around with e.g. the 
druntime a user actually gets is already often slightly forked in 
that it's often not from dmd.

The way to get a stable foundation that can be ported easily to 
the likes of wasm and so on is not making druntime bigger. You 
end up with a good result by doing so but it would be like taking 
a shortcut through a maze rather than the road - fragile.
Jun 06
parent Adam Wilson <flyboynw gmail.com> writes:
On Thursday, 6 June 2024 at 23:58:07 UTC, max haughton wrote:


 The way to get a stable foundation that can be ported easily to 
 the likes of wasm and so on is not making druntime bigger. You 
 end up with a good result by doing so but it would be like 
 taking a shortcut through a maze rather than the road - fragile.


I think you misunderstood what I am proposing. I want to split 
out the necessary compiler symbols into a "Mini-DRT" that 
represents the minimum required to make the application function. 
Then push the rest up higher in the onion.

I'm trying to make porting easier. And the best way to do that is 
layering.
Jun 08
prev sibling next sibling parent Adam Wilson <flyboynw gmail.com> writes:
On Thursday, 6 June 2024 at 18:00:56 UTC, Sebastiaan Koppe wrote:

 On Wednesday, 5 June 2024 at 23:58:14 UTC, Adam Wilson wrote:

 [...]


 I think DRT only needs to concern itself with supporting 
 language features. Anything else needs to go elsewhere.


Where then?

A runtime is a collection of code that is executed, at runtime, 
by applications. The compiler doesn't execute code in the DRT 
while it's compiling, so it's not really a compiler support 
library. The only thing that the compiler needs to know about the 
runtime is that the symbols:
1. Exist.
2. Follow the compilers emitted ABI for them so that they'll link.

The compiler is only tangentially interested in the runtime 
insofar as it emits well-known symbols that are contained in the 
runtime. There is nothing stopping DMD from emitting Phobos 
symbols, and indeed, on a few occasions, it does. (`std.math` 
IIRC) So by this definition of "Runtime" all of Phobos is part of 
DRT since technically any Phobos symbol is a compiler support 
symbol as well.

Which, consequently, is why Phobos and DRT are built and shipped 
together. That fact should be enough to end this prattle about 
keeping DRT solely for the compiler. The hard reality is that the 
runtime is *far* more closely associated with the standard 
library than the compiler. And it *always will be*. If somebody 
wanted to link in a new runtime, the compiler would not know or 
care as long as the symbols it relied on were available and 
mangled the same. IIRC this is what Tango's runtime did. But if 
somebody wants to try to make Phobos work with that same new 
runtime ... well, I guess we'll see them year or two later with a 
lot less hair.

This becomes a blocker if the goal is to make Phobos 
"source-only" to ease the transition to Editions, as we *must* 
move code out of Phobos and into DRT since we would no longer be 
able to ensure that the Phobos symbols DMD emits are actually 
compiled into the binary.

I am not proposing that we get rid of DRT or it's utility in 
language feature support, only that we accept it's mission for 
what it is, to be the universal system interface. It's either 
that or we continue to pretend that the split between DRT and 
Phobos is an actual thing.

Runtimes do not serve the compiler, they serve the applications 
that the compiler builds. Yes the compiler needs to be aware of 
the runtime, but that's it. I think we are trying to hard to keep 
DRT "small" to make porting easier, when all we've really done is 
create a situation where we have three runtimes: CRT, DRT, and 
Phobos.

Mission accomplished in the most uselessly restrictive and 
excessive way possible. Yay for D!

TL;DR: The runtime is far more important to libraries and 
applications than it ever will be to the compiler.

So, where should all of this system interface code go?
Jun 08
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 6 June 2024 at 18:00:56 UTC, Sebastiaan Koppe wrote:

 On Wednesday, 5 June 2024 at 23:58:14 UTC, Adam Wilson wrote:

 [...]


 I think DRT only needs to concern itself with supporting 
 language features. Anything else needs to go elsewhere.


Same here, and that it should be pay-as-you-go and as small as 
possible. Anything else should be regular D code.
Jun 17
parent reply Adam Wilson <flyboynw gmail.com> writes:
On Monday, 17 June 2024 at 22:58:13 UTC, Atila Neves wrote:

 On Thursday, 6 June 2024 at 18:00:56 UTC, Sebastiaan Koppe 
 wrote:

 I think DRT only needs to concern itself with supporting 
 language features. Anything else needs to go elsewhere.


 Same here, and that it should be pay-as-you-go and as small as 
 possible. Anything else should be regular D code.


Ok, then were does the Event Loop go? Or the `std.math` methods 
that the compiler hooks? The basic problem with the premise is 
that you're effectively saying "we're never going to do more than 
DRT does today".

If we put this stuff directly into Phobos then we're back to a 
compiled library as we'll be linking other static/dynamic 
libraries in. Furthermore this is going to lead to an file-size 
explosion in DRT. Very likely what you end up with is a bunch of 
"internal" modules like `std.concurrency.internal.windows` which 
is basically the same thing as a putting them in DRT.

What we are proposing *is* is a PAYGO solution. There is a 
mini-runtime that is just the basics needed to allow the compiler 
to link code. After that there will be larger rings around the 
mini-runtime that support higher level features. If you're 
porting, you must port the mini-runtime or the app won't link, 
but if you don't port the higher level features, you just want be 
able to use them.
Jun 17
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 18 June 2024 at 06:12:53 UTC, Adam Wilson wrote:

 On Monday, 17 June 2024 at 22:58:13 UTC, Atila Neves wrote:

 On Thursday, 6 June 2024 at 18:00:56 UTC, Sebastiaan Koppe 
 wrote:

 I think DRT only needs to concern itself with supporting 
 language features. Anything else needs to go elsewhere.


 Same here, and that it should be pay-as-you-go and as small as 
 possible. Anything else should be regular D code.


 Ok, then were does the Event Loop go?


What event loop? Unless we add a language feature for this I 
don't see why it would belong in drt.


 Or the `std.math` methods that the compiler hooks?


Intrinsics?


 The basic problem with the premise is that you're effectively 
 saying "we're never going to do more than DRT does today".


Not in drt itself, no. But why would we have to?
Jun 18
next sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 19/06/2024 5:29 AM, Atila Neves wrote:

     Ok, then were does the Event Loop go?
 
 What event loop? Unless we add a language feature for this I don't see 
 why it would belong in drt.


Adam is using different meanings to terminology we use.

A runtime in literature can also refer to the standard library, I did 
warn him about this particular confusion as it does not match the D 
communities understanding of the word.

Basically what he is suggesting is that there is a series of libraries 
starting from what is needed to make ``new`` clean up (it doesn't have 
to), eventloop and other core dependencies that things in our standard 
library require to have a binary linked against for.

Either way it needs a binary to be inside of. Regardless of if we call 
it part of phobos or druntime. It can't live as an import only library.
Jun 18
prev sibling parent Dukc <ajieskola gmail.com> writes:
Atila Neves kirjoitti 18.6.2024 klo 20.29:

 
 Not in drt itself, no. But why would we have to?
 
 


He already explained it in the original post:


 The most often given reason for keeping DRT as-is is that “we only need to
port DRT to a new platform, Phobos is platform independent.” I submit that
this is factually incorrect. There are currently 31 files in Phobos with
`version (WinXXXX)` statements in them. 14 with `version (linux)`. And IIRC
everybody who has tried a port D to a new platform recently has just not
bothered to port Phobos at all. To be sure, some of these version statements
are benign, but take a look at `std.stdio`, I wish whoever wants to port that
much luck in their endeavors. It has 32 instances `version (WinXXXX)` alone.
And that is our basic I/O module.
 
 [snip]
 
 An example of why this matters is found in `std.stdio`. We currently use the C
Runtime to do basic I/O operations. While that works and is expedient, it
limits us. For example, doing colored terminal output is so difficult nobody
does it, or, we are limited to the C file interface, etc. To gain access to
more advanced capabilities, we need to use the system API’s, but these are
diverse, and putting them in Phobos would significantly expand the code
required and make maintenance a nightmare. What is interesting about this is
that, in fact, D has two runtimes. The C Runtime, which we use as a universal
system interface, and the DRT which is a bunch of compiler support tools that
the CRT doesn’t have. 


To do IO in a modern way (as opposed to the standard C way) an event 
loop is probably needed.

If you read the rest of his post, you can see it's arguable if he is 
actually suggesting doing that in the runtime. He is suggesting to 
"shard" the runtime so that there would be a core "shard" that does what 
the runtime does now, and other shards that do stuff like event loop, 
io, file access primitives and so on, so that Phobos could be built on 
those without platform-specific code in it. Those other shards would all 
be optional, so that if you don't use them (or parts of Phobos or other 
libraries that depend on them), you could still write D on platform the 
optional shards aren't ported to.

When Adam writes about the runtime, he means all the proposed shards. I 
personally feel runtime should only mean the core shard, but this is 
only a difference in terminology. Either way, the proposal is to 
implement the event loop in one of the optional shards, not the core shard.
Jun 19
prev sibling next sibling parent reply Mathias Lang <geod24 gmail.com> writes:
On Wednesday, 5 June 2024 at 23:58:14 UTC, Adam Wilson wrote:

 Before I get into the design of the DRT I want to propose rules 
 that will allow us to continue to evolve DRT in the future 
 without breaking past editions. [...]


We have to take into account the combinatorial explosion that 
comes with the `rename` method. For example, should you choose to 
change the `Throwable` interface (because it's old and really not 
good), you would need to make sure that everything that uses it 
is compatible with it. There are known ways to do it in other 
languages (https://wiki.qt.io/D-Pointer) but I think the rules 
you proposed are incomplete as they only work for functions, not 
types.


 For DRT I propose a sharded design with a split between the 
 compiler support modules and the universal system interface 
 shards. [...]


Layering things / encapsulating them is good, but you also need a 
way for inner layers to refer to outer layers, otherwise you 
severely limit the capabilities of your system.
Take for example unittests: we would like to provide a much 
better out-of-the box experience for unittests. You should get 
colors, summary, the ability to run them in parallel, or even a 
filewatcher that auto-recompiler them, out of the box. We *might* 
consider unittests to be in `Core`, but I'm sure the rest is not.


 Simple, so that we can move beyond the limitations imposed by 
 the C Runtime.


A resounding yes. Keeping the ability to easily bind to the 
CRuntime is useful (e.g. being able to spin up a socket and just 
look at the C documentation), but because it's "good enough", it 
was never good at all.
Jun 06
next sibling parent Adam Wilson <flyboynw gmail.com> writes:
On Thursday, 6 June 2024 at 18:56:59 UTC, Mathias Lang wrote:

 On Wednesday, 5 June 2024 at 23:58:14 UTC, Adam Wilson wrote:

 Before I get into the design of the DRT I want to propose 
 rules that will allow us to continue to evolve DRT in the 
 future without breaking past editions. [...]


 We have to take into account the combinatorial explosion that 
 comes with the `rename` method. For example, should you choose 
 to change the `Throwable` interface (because it's old and 
 really not good), you would need to make sure that everything 
 that uses it is compatible with it. There are known ways to do 
 it in other languages (https://wiki.qt.io/D-Pointer) but I 
 think the rules you proposed are incomplete as they only work 
 for functions, not types.


I'm actually a fan of that approach for data structures, I've 
seen it referred to as either a `pImpl` for 
"Pointer-to-implementation" or a "Handle". Realistically, for 
editions this is likely the direction we need to go.

As for the combinatorial explosion problem. I'd rather have that 
than arbitrarily cutting off development of DRT. Yes, it will 
result in an ever increasing number of symbols. No way around 
that, in most cases we should be able to do redirections to newer 
code and leave the old methods as forwarding-stubs. Yes there 
will be cases where we can't But it doesn't have to be a 
geometric expansion.


 For DRT I propose a sharded design with a split between the 
 compiler support modules and the universal system interface 
 shards. [...]


 Layering things / encapsulating them is good, but you also need 
 a way for inner layers to refer to outer layers, otherwise you 
 severely limit the capabilities of your system.
 Take for example unittests: we would like to provide a much 
 better out-of-the box experience for unittests. You should get 
 colors, summary, the ability to run them in parallel, or even a 
 filewatcher that auto-recompiler them, out of the box. We 
 *might* consider unittests to be in `Core`, but I'm sure the 
 rest is not.


IMO, the FileWatcher/Re-builder is something that I've always 
seen done outside of the compiler. The compiler compiles things, 
we need to avoid trying to make it the "Everything Program". I 
feel the same way about proposals that try to make the compiler 
responsible for dependency management for the same reason.


 Simple, so that we can move beyond the limitations imposed by 
 the C Runtime.


 A resounding yes. Keeping the ability to easily bind to the 
 CRuntime is useful (e.g. being able to spin up a socket and 
 just look at the C documentation), but because it's "good 
 enough", it was never good at all.


Agreed.
Jun 08
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/6/2024 11:56 AM, Mathias Lang wrote:

 A resounding yes. Keeping the ability to easily bind to the CRuntime is useful 
 (e.g. being able to spin up a socket and just look at the C documentation),
but 
 because it's "good enough", it was never good at all.


By CRT, I mean things like strlen, printf, and setting up the arguments for 
main(). Not sockets!
Jun 10
parent Adam Wilson <flyboynw gmail.com> writes:
On Tuesday, 11 June 2024 at 00:03:29 UTC, Walter Bright wrote:

 By CRT, I mean things like strlen, printf, and setting up the 
 arguments for main(). Not sockets!


If anybody is using `strlen` in 2024 I will personally hunt them 
down and break every bone in their hands with a ball-peen 
hammer...

And one of these days we're going to need to have a conversation 
about `printf`...
Jun 11
prev sibling next sibling parent reply max haughton <maxhaton gmail.com> writes:
On Wednesday, 5 June 2024 at 23:58:14 UTC, Adam Wilson wrote:

 For example, doing colored terminal output is so difficult 
 nobody does it, or, we are limited to the C file interface, 
 etc. To gain access to more advanced capabilities, we need to 
 use the system API’s, but these are diverse, and putting them 
 in Phobos would significantly expand the code required and make 
 maintenance a nightmare.


Firstly people do colour, sometimes not very well but it's 
clearly not "so difficult nobody does it". There should be a 
higher level way of doing it shipping with the language but 
that's more of a question of high-level tradeoffs than anything 
to do with druntime.

Secondly, I think this reasoning leads towards a trap: Placing an 
arbitrary division between phobos and druntime, or more 
charitably placing that division in the wrong place, can lead to 
a lot of needless debate or lost productivity.

Unless the compiler (i.e. containing druntime) and phobos were in 
a repo together you can end up in the same situation we had 
before with the compiler and runtime being separate leading to 
have numbers of double-PRs and so on, only squared.

While we still have a runtime I don't think phobos should have 
all of the nuts and bolts in it but in contrast to "maintenance 
would be a nightmare" I posit that this be preferable to many 
alternatives: be glad you can do the maintenance in the first 
place. When you split things up across projects (or worse repos) 
you lose the ability to do atomic commits, test (easily, 
buildkite isn't easy) at the same time for example.



Aside from that I do like really the idea of ditching the C API 
where easy e.g. purely as a gimmick I've always liked the idea of 
hello world inlining down to a system call where applicable.

There are things where the C runtime does cover up a lot of 
hardware-specific details that we really don't need to care about 
but other things where it's a pile of crap and worth ditching.
Jun 06
parent Adam Wilson <flyboynw gmail.com> writes:
On Thursday, 6 June 2024 at 23:50:06 UTC, max haughton wrote:


 Secondly, I think this reasoning leads towards a trap: Placing 
 an arbitrary division between phobos and druntime, or more 
 charitably placing that division in the wrong place, can lead 
 to a lot of needless debate or lost productivity.


This is actually the heart of what I am trying to resolve. Right 
now people are treating DRT as some sort of particular annoying 
compiler extension. I wrote more about this elsewhere so I won't 
duplicate.


 Unless the compiler (i.e. containing druntime) and phobos were 
 in a repo together you can end up in the same situation we had 
 before with the compiler and runtime being separate leading to 
 have numbers of double-PRs and so on, only squared.

 While we still have a runtime I don't think phobos should have 
 all of the nuts and bolts in it but in contrast to "maintenance 
 would be a nightmare" I posit that this be preferable to many 
 alternatives: be glad you can do the maintenance in the first 
 place. When you split things up across projects (or worse 
 repos) you lose the ability to do atomic commits, test (easily, 
 buildkite isn't easy) at the same time for example.


Is this a critique of my idea or the way things are today? 
Because I talked with Mike at DConf last year about merging 
Phobos into the DMD repo and he told me then that the plan has 
always been to turn D into a mono-repo project, precisely for the 
reasons you have laid out above. My assumption is that the reason 
this has not been done yet is because of the difficultly of 
getting the CI infra stood-up correctly in the DMD repo and not 
some ideological reason.

In any case, I put this on the agenda for the Monthly meeting 
next week.


 Aside from that I do like really the idea of ditching the C API 
 where easy e.g. purely as a gimmick I've always liked the idea 
 of hello world inlining down to a system call where applicable.

 There are things where the C runtime does cover up a lot of 
 hardware-specific details that we really don't need to care 
 about but other things where it's a pile of crap and worth 
 ditching.


You and me both.
Jun 08
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Is relying on the C runtime library really a problem? It's probably the most 
debugged library in history, and it's small and lightweight.
Jun 06
next sibling parent rkompass <rkompass gmx.de> writes:
On Friday, 7 June 2024 at 01:23:03 UTC, Walter Bright wrote:

 Is relying on the C runtime library really a problem? It's 
 probably the most debugged library in history, and it's small 
 and lightweight.


My impression is:
If D wants to be successful as a universal systems programming 
language then relying on the C runtime may become a problem.
And splitting the D runtime as proposed, with many configuration 
options may be a solution.
Jun 08
prev sibling next sibling parent reply monkyyy <crazymonkyyy gmail.com> writes:
On Friday, 7 June 2024 at 01:23:03 UTC, Walter Bright wrote:

 Is relying on the C runtime library really a problem? It's 
 probably the most debugged library in history, and it's small 
 and lightweight.


Depends on goals, if your targeting moving d to a higher level, 
wasm+libc will just suck(they broke file i/o despite w3c lying, 
it will always be a weird edge case you have to specifically 
support) and I think a go/swift apooch  of an std making a non-c 
api will probably be best

If you want to compete on the low level zig competing with c 
involves competing with libc; other platforms, new chips; maybe 
things break weirdly or they have bad workarounds

if you want to keep d exactly where it is; I cant imagine much 
reason to change libc dependence its fine for windows and linux 
and fake linux; so is there going to be a major push for wasm or 
embedded?
Jun 08
next sibling parent reply Adam Wilson <flyboynw gmail.com> writes:
On Saturday, 8 June 2024 at 21:15:39 UTC, monkyyy wrote:

 On Friday, 7 June 2024 at 01:23:03 UTC, Walter Bright wrote:

 Is relying on the C runtime library really a problem? It's 
 probably the most debugged library in history, and it's small 
 and lightweight.


 Depends on goals, if your targeting moving d to a higher level, 
 wasm+libc will just suck(they broke file i/o despite w3c lying, 
 it will always be a weird edge case you have to specifically 
 support) and I think a go/swift apooch  of an std making a 
 non-c api will probably be best

 If you want to compete on the low level zig competing with c 
 involves competing with libc; other platforms, new chips; maybe 
 things break weirdly or they have bad workarounds

 if you want to keep d exactly where it is; I cant imagine much 
 reason to change libc dependence its fine for windows and linux 
 and fake linux; so is there going to be a major push for wasm 
 or embedded?


I broadly agree with this assessment.

Moving up level necessarily means broadening out beyond the CRT.

Moving down is something we're not well equipped to handle as you 
start to compete on execution speed, which means esoteric 
back-end optimizations, which is something DMD sucks at so we use 
LDC/GDC. And since LLVM and GCC already exist and we're already 
using them, we've already admitted that we're not going down that 
path.

Staying where we are means stagnation. Let's not do that.

And WASM keeps coming up as a priority for DLF so I think we can 
all see where this is headed.
Jun 08
parent monkyyy <crazymonkyyy gmail.com> writes:
On Sunday, 9 June 2024 at 03:22:07 UTC, Adam Wilson wrote:

 
 Moving down is something we're not well equipped to handle as 
 you start to compete on execution speed, which means esoteric 
 back-end optimizations


Im not sure how esoteric it is; I think you could have mixins 
generate inline asm syscalls and just grab the tables and then 
like a data structure for simd thats mostly op overloads
Jun 09
prev sibling parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Saturday, 8 June 2024 at 21:15:39 UTC, monkyyy wrote:


 if you want to keep d exactly where it is; I cant imagine much 
 reason to change libc dependence its fine for windows and linux 
 and fake linux; so is there going to be a major push for wasm 
 or embedded?


As I remember, druntime uses not more than 10 CRT calls: 
allocate/free memory, sockets, threads. Thats all.

It upsets me that we have a huge core.stdc.* - it would be great 
to move these files somewhere
Jun 09
parent reply Adam Wilson <flyboynw gmail.com> writes:
On Sunday, 9 June 2024 at 16:58:33 UTC, Denis Feklushkin wrote:

 On Saturday, 8 June 2024 at 21:15:39 UTC, monkyyy wrote:

 As I remember, druntime uses not more than 10 CRT calls: 
 allocate/free memory, sockets, threads. Thats all.

 It upsets me that we have a huge core.stdc.* - it would be 
 great to move these files somewhere


I don't know how many it is, but it's definitely more than 10, 
just the File I/O would be more than 10.

I agree that the system "DI" files do not belong in the Runtime 
and Rikki has proposed that we move all of the system API 
interface DI files out to a separate location. Maybe we could do 
something like `windows.*`, `posix.*`, `macos.*`, `crt.*`?
Jun 11
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Tuesday, 11 June 2024 at 08:53:21 UTC, Adam Wilson wrote:


 interface DI files out to a separate location. Maybe we could 
 do something like `windows.*`, `posix.*`, `macos.*`, `crt.*`?


I proposed a radical solution that changes all this (but the 
community's opinion was polarized on it):

https://github.com/dlang/dmd/pull/15887
Jun 12
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Wednesday, 12 June 2024 at 08:46:47 UTC, Denis Feklushkin 
wrote:

 On Tuesday, 11 June 2024 at 08:53:21 UTC, Adam Wilson wrote:


 interface DI files out to a separate location. Maybe we could 
 do something like `windows.*`, `posix.*`, `macos.*`, `crt.*`?


 I proposed a radical solution that changes all this (but the 
 community's opinion was polarized on it):

 https://github.com/dlang/dmd/pull/15887


You introduce a makefile on top of a cryptic script to achieve 
it, it is over engineered
Jun 12
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Wednesday, 12 June 2024 at 09:31:13 UTC, ryuukk_ wrote:

 On Wednesday, 12 June 2024 at 08:46:47 UTC, Denis Feklushkin 
 wrote:

 On Tuesday, 11 June 2024 at 08:53:21 UTC, Adam Wilson wrote:


 interface DI files out to a separate location. Maybe we could 
 do something like `windows.*`, `posix.*`, `macos.*`, `crt.*`?


 I proposed a radical solution that changes all this (but the 
 community's opinion was polarized on it):

 https://github.com/dlang/dmd/pull/15887


 You introduce a makefile on top of a cryptic script to achieve 
 it, it is over engineered


Please, spend 5 minutes to understand how this script works. It 
just contains two cycles. On D in will be ~5 lines, I think

I also don't like Bash, but this is the only way to make code 
portable for all our supported platforms
Jun 12
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
 Please, spend 5 minutes to understand how this script works


This is the exact problem of druntime, it does bunch of stuff it 
shouldn't be doing

If the language sucks to achieve what you want to achieve, i'd 
personally work my way in improving the compiler, i tried once 
https://github.com/dlang/dmd/pull/15479
Jun 12
parent Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Wednesday, 12 June 2024 at 11:39:00 UTC, ryuukk_ wrote:

 Please, spend 5 minutes to understand how this script works


 This is the exact problem of druntime, it does bunch of stuff 
 it shouldn't be doing


The script does exactly what is needed. And only at the druntime 
build stage, so majority of users will never encounter it


 If the language sucks to achieve what you want to achieve


It's not about D language, it's about only design (organsation) 
of the druntime source code
Jun 12
prev sibling parent reply Adam Wilson <flyboynw gmail.com> writes:
On Friday, 7 June 2024 at 01:23:03 UTC, Walter Bright wrote:

 Is relying on the C runtime library really a problem? It's 
 probably the most debugged library in history, and it's small 
 and lightweight.


For example, let's say you want to do some asynchronous I/O. 
Forget the CRT, it just doesn't do that. So off you go to the 
system API's anyways. Or kludges like Photon. (Don't get me 
wrong, Photon is a very cool piece of work, but the nature of 
it's implementation means that it suffers from the inherent 
limitations and drawbacks of fibers, and does so in a highly 
obfuscated way.)

The point is more that if we want to do useful things in the 
modern world that exists beyond the CRT, then we have to work 
around it, and if we have to work around it anyways, why are we 
using it at all?

If you go with the System API's the world is your oyster. Yes, 
there is more work upfront, but the number of capabilities we 
would be able to enable is immense.
Jun 08
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Sunday, 9 June 2024 at 03:10:26 UTC, Adam Wilson wrote:

 On Friday, 7 June 2024 at 01:23:03 UTC, Walter Bright wrote:

 Is relying on the C runtime library really a problem? It's 
 probably the most debugged library in history, and it's small 
 and lightweight.


 For example, let's say you want to do some asynchronous I/O. 
 Forget the CRT, it just doesn't do that. So off you go to the 
 system API's anyways.


And on Posix libc is the systems API.


 Or kludges like Photon. (Don't get me wrong, Photon is a very 
 cool piece of work, but the nature of it's implementation means 
 that it suffers from the inherent limitations and drawbacks of 
 fibers, and does so in a highly obfuscated way.)


I find critique of stackful coroutines really weak to be honest. 
Implementing n:m or 1:n threads is kernel doesn’t scale, but  
combining async io with user-space fibers works beautifully. The 
only problem I see is stack sizes, and even there we can just 
reserve a lot more on x64 it’s not a problem at all.

Java after all these years is putting lots of effort to support 
virtual threads,  that would be introduced along side with normal 
threads.
Go is highly popular and doing just fine without even having 
normal threads.


 The point is more that if we want to do useful things in the 
 modern world that exists beyond the CRT, then we have to work 
 around it, and if we have to work around it anyways, why are we 
 using it at all?


On Windows I have no idea why we need to bind to libc at all. 
Synchronize with C’s io?

On Posix mostly syscall interface. That and malloc/free, 
memcpy/memset (simply because it’s optimized to death). All of 
the rest is legacy garbage no one is going to touch anyway.


 If you go with the System API's the world is your oyster. Yes, 
 there is more work upfront, but the number of capabilities we 
 would be able to enable is immense.


Being a system language D it allows anyone to use system APIs, 
meaning it’s easy to step on the toes of DRT if it uses lots of 
them.
Jun 10
parent reply Adam Wilson <flyboynw gmail.com> writes:
On Monday, 10 June 2024 at 09:36:14 UTC, Dmitry Olshansky wrote:

 And on Posix libc is the systems API.


For some things yes, but if you want to do anything with Async 
I/O you're going to switch over to something like `select` or 
`io_uring`.


 Or kludges like Photon. (Don't get me wrong, Photon is a very 
 cool piece of work, but the nature of it's implementation 
 means that it suffers from the inherent limitations and 
 drawbacks of fibers, and does so in a highly obfuscated way.)


 I find critique of stackful coroutines really weak to be 
 honest. Implementing n:m or 1:n threads is kernel doesn’t 
 scale, but  combining async io with user-space fibers works 
 beautifully. The only problem I see is stack sizes, and even 
 there we can just reserve a lot more on x64 it’s not a problem 
 at all.

 Java after all these years is putting lots of effort to support 
 virtual threads,  that would be introduced along side with 
 normal threads.
 Go is highly popular and doing just fine without even having 
 normal threads.


That strikes me as more of an opinion than objective fact. I led 
a detailed discussion of this topic on Discord. The end result 
was that the stack size issue ends up being catastrophic in 
non-trivial workloads. Vibe went with a 16MB stack size for 
precisely this reason, which means that to handle 65536 
simultaneous connections, I need a server with *1TB* of RAM. The 
reason for that is that due to performance concerns, we turn off 
over-commit and thus allocating 16MB per stack means that you are 
fully committing 16MB of physical RAM. Go/Java/.NET, can all 
handle 10x that number of connections on a server with 128GB of 
RAM, so that's the bar we have to meet.

No other language suffers this problem, not even Go. The reason 
is that all languages that successfully use Fibers, use 
dynamically expanding stacks, but this means using a precise 
stack-scanning moving GC. Something that D, so long as Walter is 
among the living, will never have.

Stackless coroutines also do not suffer this problem, which is 
why .NET and Rust use them.

Here are some resources, including a real-world example from 
CloudFlare:
https://devblogs.microsoft.com/oldnewthing/20191011-00/?p=102989
https://www.open-std.org/JTC1/SC22/WG21/docs/papers/2018/p1364r0.pdf
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1520r0.pdf
https://blog.cloudflare.com/how-stacks-are-handled-in-go


 On Windows I have no idea why we need to bind to libc at all. 
 Synchronize with C’s io?


Beyond the fact that we currently use libc, I don't see any 
reason for us to either.


 On Posix mostly syscall interface. That and malloc/free, 
 memcpy/memset (simply because it’s optimized to death). All of 
 the rest is legacy garbage no one is going to touch anyway.


And in the case of malloc/free, replacing those with different 
allocators has become quite the rage lately. jemalloc seems 
particularly popular. IIRC, Deadalnix's new GC uses a different 
allocator from libc. So even fewer reasons.


 Being a system language D it allows anyone to use system APIs, 
 meaning it’s easy to step on the toes of DRT if it uses lots of 
 them.


Can you explain what you mean? I am not sure how we'd step on 
DRT's toes if applications use a lot of System API's?
Jun 11
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 11/06/2024 9:25 PM, Adam Wilson wrote:

 On Monday, 10 June 2024 at 09:36:14 UTC, Dmitry Olshansky wrote:
 
     And on Posix libc is the systems API.
 
 For some things yes, but if you want to do anything with Async I/O 
 you're going to switch over to something like |select| or |io_uring|.


Not select, that's the hello world of polling functions.

Right now Posix doesn't have anything for async, linux has epoll which 
can be the equivalent to IOCP.
Jun 11
parent Adam Wilson <flyboynw gmail.com> writes:
On Tuesday, 11 June 2024 at 09:30:17 UTC, Richard (Rikki) Andrew 
Cattermole wrote:

 Not select, that's the hello world of polling functions.

 Right now Posix doesn't have anything for async, linux has 
 epoll which can be the equivalent to IOCP.


Oh, I know. I was just working off you prior use of it as an 
example.
Jun 11
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Tuesday, 11 June 2024 at 09:25:15 UTC, Adam Wilson wrote:

 On Monday, 10 June 2024 at 09:36:14 UTC, Dmitry Olshansky wrote:

 And on Posix libc is the systems API.


 For some things yes, but if you want to do anything with Async 
 I/O you're going to switch over to something like `select` or 
 `io_uring`.


Select is a system call so libc. io_uring is a system call with a 
dedicated library.


 Or kludges like Photon. (Don't get me wrong, Photon is a very 
 cool piece of work, but the nature of it's implementation 
 means that it suffers from the inherent limitations and 
 drawbacks of fibers, and does so in a highly obfuscated way.)


 I find critique of stackful coroutines really weak to be 
 honest. Implementing n:m or 1:n threads is kernel doesn’t 
 scale, but  combining async io with user-space fibers works 
 beautifully. The only problem I see is stack sizes, and even 
 there we can just reserve a lot more on x64 it’s not a problem 
 at all.

 Java after all these years is putting lots of effort to 
 support virtual threads,  that would be introduced along side 
 with normal threads.
 Go is highly popular and doing just fine without even having 
 normal threads.


 That strikes me as more of an opinion than objective fact. I 
 led a detailed discussion of this topic on Discord. The end 
 result was that the stack size issue ends up being catastrophic 
 in non-trivial workloads. Vibe went with a 16MB stack size for 
 precisely this reason, which means that to handle 65536 
 simultaneous connections, I need a server with *1TB* of RAM.


Full stop right there. Virtual memory was designed precisely to 
handle cases like this, small allocations that may sometimes 
balloon to large in size. Reservation is not commit and detection 
of going out of range is quite possible on all OSes.


 The reason for that is that due to performance concerns, we 
 turn off over-commit and thus allocating 16MB per stack means 
 that you are fully committing 16MB of physical RAM.


We can allocate virtual memory lazily even with overcommit 
disabled, in such a case you would use page fault handler to 
commit memory on as faulted basis.


 Go/Java/.NET, can all handle 10x that number of connections on 
 a server with 128GB of RAM, so that's the bar we have to meet.


No problem with _reserving_ terabytes and using fraction of that. 
If the only problem is that you see memory as commited I’d be 
willing to look into why that is the case.


 No other language suffers this problem, not even Go. The reason 
 is that all languages that successfully use Fibers, use 
 dynamically expanding stacks, but this means using a precise 
 stack-scanning moving GC.


Far as I can tell Go dropped segmented stacks I cannot tell if 
they copy the stack around.


 Something that D, so long as Walter is among the living, will 
 never have.

 Stackless coroutines also do not suffer this problem, which is 
 why .NET and Rust use them.


The effort to introduce stackless coroutines together with 
rewriting the world to use them is enormous. The only reason 
behind Photon going as syscall wrapper is that we do not have 
resources to rewrite every C client library in say vibe.d sockets.


 Being a system language D it allows anyone to use system APIs, 
 meaning it’s easy to step on the toes of DRT if it uses lots 
 of them.


 Can you explain what you mean? I am not sure how we'd step on 
 DRT's toes if applications use a lot of System API's?


Easy - using the same signal number for something internal or 
segfault handler.

—
Dmitry Olshansky
CEO   [Glow labs](https://glow-labs.pro)
https://olshansky.me
Jun 11
parent Derek Fawcus <dfawcus+dlang employees.org> writes:
On Tuesday, 11 June 2024 at 11:38:40 UTC, Dmitry Olshansky wrote:

 On Tuesday, 11 June 2024 at 09:25:15 UTC, Adam Wilson wrote:

 On Monday, 10 June 2024 at 09:36:14 UTC, Dmitry Olshansky 
 wrote:





 No other language suffers this problem, not even Go. The 
 reason is that all languages that successfully use Fibers, use 
 dynamically expanding stacks, but this means using a precise 
 stack-scanning moving GC.


 Far as I can tell Go dropped segmented stacks I cannot tell if 
 they copy the stack around.


FWIW:

(a reply to an old thread, but I did not see this noted in the 
thread)

While Go has a precise GC, the GC does not move any items once 
they're in the heap.

As to stack growth, once the segmented stacks were dropped, it 
switched to a scheme of doubling the size of stacks.  So a new 
stack is allocated, the contents of the old stack copied to it, 
and any pointers in to the stack (which can only be from 
elsewhere in the stack) are fixed up.  That depends upon the same 
precise info which the GC uses.

So the stack does move when it grows, and be fixed up. However 
that is not done by the GC.

The stack can point in to the heap, the heap can not point in to 
the stack.

DF
Oct 17
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Tuesday, 11 June 2024 at 09:25:15 UTC, Adam Wilson wrote:

 [...]


 That strikes me as more of an opinion than objective fact. I 
 led a detailed discussion of this topic on Discord. The end 
 result was that the stack size issue ends up being catastrophic 
 in non-trivial workloads. Vibe went with a 16MB stack size for 
 precisely this reason, which means that to handle 65536 
 simultaneous connections, I need a server with *1TB* of RAM. 
 The reason for that is that due to performance concerns, we 
 turn off over-commit and thus allocating 16MB per stack means 
 that you are fully committing 16MB of physical RAM. 
 Go/Java/.NET, can all handle 10x that number of connections on 
 a server with 128GB of RAM, so that's the bar we have to meet.

 No other language suffers this problem, not even Go. The reason 
 is that all languages that successfully use Fibers, use 
 dynamically expanding stacks, but this means using a precise 
 stack-scanning moving GC. Something that D, so long as Walter 
 is among the living, will never have.

 Stackless coroutines also do not suffer this problem, which is 
 why .NET and Rust use them.


I’ve measured the overheads of having millions of fibers with 
16mb of stack even with overcommit it’s way too much. We have to 
go stackless much as I do not like it.

—
Dmitry Olshansky
CEO   [Glow labs](https://glow-labs.pro)
https://olshansky.me
Jun 13
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/8/2024 8:10 PM, Adam Wilson wrote:

 On Friday, 7 June 2024 at 01:23:03 UTC, Walter Bright wrote:

 Is relying on the C runtime library really a problem? It's probably the most 
 debugged library in history, and it's small and lightweight.


 
 For example, let's say you want to do some asynchronous I/O. Forget the CRT,
it 
 just doesn't do that. So off you go to the system API's anyways. Or kludges
like 
 Photon. (Don't get me wrong, Photon is a very cool piece of work, but the
nature 
 of it's implementation means that it suffers from the inherent limitations and 
 drawbacks of fibers, and does so in a highly obfuscated way.)
 
 The point is more that if we want to do useful things in the modern world that 
 exists beyond the CRT, then we have to work around it, and if we have to work 
 around it anyways, why are we using it at all?
 
 If you go with the System API's the world is your oyster. Yes, there is more 
 work upfront, but the number of capabilities we would be able to enable is
immense.


I don't understand how the CRT impedes any of that, or why any of it needs to
be 
worked around. And D is intended to work with hybrid C/D code, so the CRT 
support needs to be there.

The CRT also does some startup initialization things that need doing, like 
collecting the command line arguments to present to the program.
Jun 10
next sibling parent reply aberba <karabutaworld gmail.com> writes:
On Tuesday, 11 June 2024 at 00:01:01 UTC, Walter Bright wrote:

 On 6/8/2024 8:10 PM, Adam Wilson wrote:

 [...]


 I don't understand how the CRT impedes any of that, or why any 
 of it needs to be worked around. And D is intended to work with 
 hybrid C/D code, so the CRT support needs to be there.

 The CRT also does some startup initialization things that need 
 doing, like collecting the command line arguments to present to 
 the program.


Why can't these things be in written D? Are we opposed to that or 
it's a matter of manpower (getting it done)?
Jun 11
next sibling parent Adam Wilson <flyboynw gmail.com> writes:
On Tuesday, 11 June 2024 at 08:40:32 UTC, aberba wrote:

 Why can't these things be in written D? Are we opposed to that 
 or it's a matter of manpower (getting it done)?


They can be done in D, and doing them in D would likely open us 
up to interesting use cases, like OSdev (since they no longer 
have to port CRT first), or acting as a bootstrap Runtime for 
modern languages that want to provide more than CRT.

However, we do have to be realistic about our manpower 
constraints. But if you're offering to lend a hand ... we'll take 
it!
Jun 11
prev sibling parent Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Tuesday, 11 June 2024 at 08:40:32 UTC, aberba wrote:


 Why can't these things be in written D? Are we opposed to that 
 or it's a matter of manpower (getting it done)?


Yes, this can be done without changing the code structure in any 
way: we have version(..) for this
Jun 12
prev sibling parent Adam Wilson <flyboynw gmail.com> writes:
On Tuesday, 11 June 2024 at 00:01:01 UTC, Walter Bright wrote:

 On 6/8/2024 8:10 PM, Adam Wilson wrote:

 For example, let's say you want to do some asynchronous I/O. 
 Forget the CRT, it just doesn't do that. So off you go to the 
 system API's anyways. Or kludges like Photon. (Don't get me 
 wrong, Photon is a very cool piece of work, but the nature of 
 it's implementation means that it suffers from the inherent 
 limitations and drawbacks of fibers, and does so in a highly 
 obfuscated way.)


 I don't understand how the CRT impedes any of that, or why any 
 of it needs to be worked around. And D is intended to work with 
 hybrid C/D code, so the CRT support needs to be there.


Strictly speaking it doesn't "impede" it, but it does introduce 
multiple code-paths into the code. Let's say you want to do some 
Async File I/O. You simply can't in CRT because it's just not a 
thing that CRT does. Since you *have* to write code using the 
system interfaces to do this, you are going to write the System 
API path, so why not just use the System API path and cut out the 
middle-man?

I am not suggesting that we remove support for CRT from the 
language. Only that we build DRT such that it minimizes or 
completely removes it's use. We've also discussed a design where 
we continue to provide a CRT enabled version of DRT to make 
porting to new platforms easier, but would use the System API 
code paths by default on systems where those have been 
implemented. More work, because Phobos would have to be aware 
that you're using it and appropriately remove the code that CRT 
cannot support, but potentially worth it.


 The CRT also does some startup initialization things that need 
 doing, like collecting the command line arguments to present to 
 the program.


On Windows you use `CommandLineToArgvW`, on Linux/BSD's you might 
actually use the CRT, I don't know what the equivalent is on 
MacOS.
Jun 11
prev sibling next sibling parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
i had to write a custom runtime, it was easy thanks to adam's POC 
repo, but i shouldn't have to want to do that, i did because i 
was unable to port druntime, it's too much work, since it does 
things it shouldn't be doing... why should i also port libc???

druntime was not built to be platform/architecture independent, 
that's the only problem it has, well maybe is also too big..


i don't like that GC is part of "core", it should me: "memory", 
memory has "GC and allocator api", please emphasis that GC is not 
the only way to deal with memory, it's one way of doing it 
alongside allocators for when performance matter, D has an 


for example)


the main the runtime uses should be  nogc nothrow, to ensure that 
the base is already portable for all kind of purpose, including 
embedded


the runtime shouldn't do too much, only the strict minimum, so it 
remains compatible with most platforms/architectures as much as 
possible



every big module should be split per platform, i like how it was 
done with `sections.d` (`sections_android.d` `sections_win64.d` 
etc)  i follow the same strategy for my own code


ideally the runtime should be a source file, not a system 
library, no need to version it, no need to ship it, D compiles 
code _VERY_ fast, it'll force everyone to not bloat it in the 
years to come ;)
Jun 12
parent Adam Wilson <flyboynw gmail.com> writes:
On Wednesday, 12 June 2024 at 08:10:53 UTC, ryuukk_ wrote:

 i had to write a custom runtime, it was easy thanks to adam's 
 POC repo, but i shouldn't have to want to do that, i did 
 because i was unable to port druntime, it's too much work, 
 since it does things it shouldn't be doing... why should i also 
 port libc???

 druntime was not built to be platform/architecture independent, 
 that's the only problem it has, well maybe is also too big..


 i don't like that GC is part of "core", it should me: "memory", 
 memory has "GC and allocator api", please emphasis that GC is 
 not the only way to deal with memory, it's one way of doing it 
 alongside allocators for when performance matter, D has an 

 to come up with stupid apis to write no GC code (stackalloc for 



At the moment I don't see any way for the GC to not be in the 
mini-runtime since the compiler hooks it and it's required for 
compiler features. We have such an API as you propose and all it 
got us was a massive slow-down in the GC to accommodate the 
three-level v-table. We are a Runtime based language, there is 
always going to be a minimum that you are going to have to port.


 the runtime shouldn't do too much, only the strict minimum, so 
 it remains compatible with most platforms/architectures as much 
 as possible


This is just not possible in practice. Runtimes are support code 
for the application, not the compiler, the compiler hooks the 
runtime to do things, but the purpose is to make compiler 
features for the application possible. Thus as the capabilities 
of Phobos grow, so too will the support code required to make 
those features function.

For example, Event Loops are already on the board as a "must 
have", and they are going into the runtime. Another one is that 
`std.math` absolutely needs to be moved into the runtime as the 
compiler hooks some of those methods (this one is going to happen 
for Phobos 3). There are more, but the point is that the Runtime 
gets bigger, not smaller from here.  So the question becomes, how 
do we manage the growth?


 every big module should be split per platform, i like how it 
 was done with `sections.d` (`sections_android.d` 
 `sections_win64.d` etc)  i follow the same strategy for my own 
 code


 ideally the runtime should be a source file, not a system 
 library, no need to version it, no need to ship it, D compiles 
 code _VERY_ fast, it'll force everyone to not bloat it in the 
 years to come ;)


I'm reasonably sure that making DRT into a source library is not 
feasible. Some things need to be compiled and linked in.
Jun 17
prev sibling parent Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Wednesday, 5 June 2024 at 23:58:14 UTC, Adam Wilson wrote:


 The most often given reason for keeping DRT as-is is that “we 
 only need to port DRT to a new platform, Phobos is platform 
 independent.” I submit that this is factually incorrect. There 
 are currently 31 files in Phobos with `version (WinXXXX)` 
 statements in them. 14 with `version (linux)`. And IIRC 
 everybody who has tried a port D to a new platform recently has 
 just not bothered to port Phobos at all.


By the way, I’ll answer as one of these persons: it will just be 
a waste of time!

Because how it need to be ported very much depends on the 
decision made about how we reorganize the druntime at first.
Jul 02