www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D with minimal runtime

reply Dibyendu Majumdar <d.majumdar gmail.com> writes:
Hi,

When writing compiler tools, a GC seems desirable as managing 
memory for the data structures used by compilers is a pain. I 
guess the better C option has no runtime, whereas normal D has 
the runtime with GC. I was wondering whether the runtime can be 
minimized to only support the GC. Or is that already the case, 
i.e. the D runtime is already the minimum needed for GC?

Thanks and Regards
Dibyendu
Jun 15 2020
next sibling parent Dibyendu Majumdar <d.majumdar gmail.com> writes:
On Monday, 15 June 2020 at 08:54:45 UTC, Dibyendu Majumdar wrote:

 When writing compiler tools, a GC seems desirable as managing 
 memory for the data structures used by compilers is a pain. I 
 guess the better C option has no runtime, whereas normal D has 
 the runtime with GC. I was wondering whether the runtime can be 
 minimized to only support the GC. Or is that already the case, 
 i.e. the D runtime is already the minimum needed for GC?


Just to clarify the context - the compiler needs to be built as a 
library that can be used in any project.
Jun 15 2020
prev sibling next sibling parent reply IGotD- <nise nise.com> writes:
On Monday, 15 June 2020 at 08:54:45 UTC, Dibyendu Majumdar wrote:

 Hi,

 When writing compiler tools, a GC seems desirable as managing 
 memory for the data structures used by compilers is a pain. I 
 guess the better C option has no runtime, whereas normal D has 
 the runtime with GC. I was wondering whether the runtime can be 
 minimized to only support the GC. Or is that already the case, 
 i.e. the D runtime is already the minimum needed for GC?

 Thanks and Regards
 Dibyendu


C does almost require a standard library which is clib. You can't 
do memcpy or memset without clib so C becomes kind of dependable 
on clib. However, you can use C without clib as well and D can 
that too with the -betterC switch.

I'm looking for a minimal druntime too. Right now it is all or 
nothing for druntime and it expect a full rich OS. There have 
been talks to partion druntime better so that you can incremental 
support, for example leave networking out if you want to. Also 
right now for all operating systems, druntime depends on clib 
itself. Nim has a compiler switch -os:any which means that the 
runtime will only depend on clib which makes it very portable. 
That would have been nice in druntime too.
Jun 15 2020
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/15/2020 3:01 AM, IGotD- wrote:

 I'm looking for a minimal druntime too. Right now it is all or nothing for 
 druntime and it expect a full rich OS. There have been talks to partion
druntime 
 better so that you can incremental support, for example leave networking out
if 
 you want to. Also right now for all operating systems, druntime depends on
clib 
 itself. Nim has a compiler switch -os:any which means that the runtime will
only 
 depend on clib which makes it very portable. That would have been nice in 
 druntime too.


Since druntime is a library, only the functions actually referenced are pulled 
out of the library when linking.
Jun 15 2020
parent reply IGotD- <nise nise.com> writes:
On Monday, 15 June 2020 at 10:07:10 UTC, Walter Bright wrote:

 Since druntime is a library, only the functions actually 
 referenced are pulled out of the library when linking.


Still you need to compile druntime and if it finds any missing 
symbol there, the build will fail. Also the in druntime files 
often include other files in druntime which doesn't make it that 
easy.

As we have discussed before, the best way for this in my opinion 
is to have an abstract interface in druntime for the OS 
functions. This way we could stub those functions that the 
programmer doesn't need.
Jun 15 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/15/2020 3:16 AM, IGotD- wrote:

 As we have discussed before, the best way for this in my opinion is to have an 
 abstract interface in druntime for the OS functions. This way we could stub 
 those functions that the programmer doesn't need.


If you include the line:

    extern (C) void printf() { }

somewhere in your code, printf() will *not* get linked in.

An extra interface layer is not necessary.
Jun 15 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/20 4:50 PM, Walter Bright wrote:

 On 6/15/2020 3:16 AM, IGotD- wrote:

 As we have discussed before, the best way for this in my opinion is to 
 have an abstract interface in druntime for the OS functions. This way 
 we could stub those functions that the programmer doesn't need.


 
 If you include the line:
 
     extern (C) void printf() { }
 
 somewhere in your code, printf() will *not* get linked in.
 
 An extra interface layer is not necessary.


Word. Any work on reducing druntime needs to be based on a good 
understanding of how linking works.
Jun 15 2020
parent reply dangbinghoo <dangbinghoo google-mail.com> writes:
On Tuesday, 16 June 2020 at 01:16:27 UTC, Andrei Alexandrescu 
wrote:

 On 6/15/20 4:50 PM, Walter Bright wrote:

 On 6/15/2020 3:16 AM, IGotD- wrote:

 As we have discussed before, the best way for this in my 
 opinion is to have an abstract interface in druntime for the 
 OS functions. This way we could stub those functions that the 
 programmer doesn't need.


 
 If you include the line:
 
     extern (C) void printf() { }
 
 somewhere in your code, printf() will *not* get linked in.
 
 An extra interface layer is not necessary.


 Word. Any work on reducing druntime needs to be based on a good 
 understanding of how linking works.


hi,

many pepole is requiring that the D runtime needs to be minimal. 
Yes, this "is a linking problem" just if you're always using 
"normal" or "rich" OS, things matter if you trying to using D in 
the microcontroller target such as STM32F4 or something similar, 
these platfrom has hundreds of KB ram and maybe 1MB of flash 
memory, it's powerfull enogh for running the D language.

if you trying to build a betterC programing on these 
platforms(only RTOS or no OS at all), it shows that D runtime is 
requring some thing that only Rich-OS have.

the rust and zig programming language is doing better in this 
situation, the D runtime can be made only need to requiring ANSI 
C stdlib (like nim 'any') and then the D runtime is more portable.


thanks!
----
binghoo dang
Jun 16 2020
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/16/20 10:09 PM, dangbinghoo wrote:

 the D runtime can be made only need to requiring ANSI C stdlib (like nim 
 'any') and then the D runtime is more portable.


That would be nice, and possible if most parts of druntime would be 
templated and others would only be pulled in if used. It boils down to 
finding the talented folks to make it happen.
Jun 16 2020
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 17 June 2020 at 02:29:44 UTC, Andrei Alexandrescu 
wrote:

 possible if most parts of druntime would be templated and 
 others would only be pulled in if used.


I actually don't think it is even necessary to template it. If 
you had independent modules written with minimal dependencies, 
there's several things that make it work automatically: dmd -i 
can compile as-needed. Linkers can leave things out of exes from 
static libs. Etc.

My arsd package as like 78 modules now. But I'm strict about them 
not importing each other unless absolutely necessary. You can use 
most of them individually without caring about the other 77 (but 
if you do reference it, it gets auto-imported, e.g. 
Document.fromUrl brings in http... but if you never call that, no 
need for the http module at all in your build)

There's been some decent work in dmd to support this for language 
features too, but from the other direction. If a feature isn't 
present in object.d, it won't reference it. That little glue 
layer might be templatized so it is available, but not used 
unless needed... then from there it just imports a traditional 
module.

Pretty sure good chunks of that would work.

druntime and phobos too have problems of intertwined imports that 
would take a lot of work to clean up though.
Jun 16 2020
prev sibling parent reply IGotD- <nise nise.com> writes:
On Wednesday, 17 June 2020 at 02:09:47 UTC, dangbinghoo wrote:

 many pepole is requiring that the D runtime needs to be 
 minimal. Yes, this "is a linking problem" just if you're always 
 using "normal" or "rich" OS, things matter if you trying to 
 using D in the microcontroller target such as STM32F4 or 
 something similar, these platfrom has hundreds of KB ram and 
 maybe 1MB of flash memory, it's powerfull enogh for running the 
 D language.

 if you trying to build a betterC programing on these 
 platforms(only RTOS or no OS at all), it shows that D runtime 
 is requring some thing that only Rich-OS have.

 the rust and zig programming language is doing better in this 
 situation, the D runtime can be made only need to requiring 
 ANSI C stdlib (like nim 'any') and then the D runtime is more 
 portable.


 thanks!
 ----
 binghoo dang


Indeed and it is not only about "not linking in what you don't 
need" as the druntime expects a lot of things from the OS. This 
includes reading elf section information, which shouldn't be 
needed in a minimal system as we don't even know if the 
programmer wants to use elf/pe. Maybe the programmer just want to 
use a binary blob. Typically RTOS in small platforms like STM32F4 
or similar often don't implement TLS. For these kind of systems 
it would be nice if druntime didn't have any TLS variables at all 
and then the programmer must promise not to use any TLS 
variables. Essentially where you obtain the TLS area for each 
thread must be stubbed and just report empty TLS area.

Full D isn't really adapted for limited OS/RTOS today and it has 
painted itself into a corner by implementing a lot of expected OS 
support in the druntime. There isn't any wrong with it, it's just 
that D isn't a systems programming language and the question is 
if D should go that route. I'd say that D missed that train and D 
shouldn't peruse that any further.
Jun 17 2020
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Wednesday, 17 June 2020 at 10:39:38 UTC, IGotD- wrote:

 Typically RTOS in small platforms like STM32F4 or similar often 
 don't implement TLS.


Don't worry - TLS implementation is trivial for each platform.
Jun 17 2020
parent IGotD- <nise nise.com> writes:
On Thursday, 18 June 2020 at 05:50:38 UTC, Denis Feklushkin wrote:

 Don't worry - TLS implementation is trivial for each platform.


For a static system which doesn't change, then yes it might be 
trivial. For a system that allows runtime loaded shared 
libraries, then no.
Jun 18 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/20 6:01 AM, IGotD- wrote:

 I'm looking for a minimal druntime too. Right now it is all or nothing 
 for druntime and it expect a full rich OS. There have been talks to 
 partion druntime better so that you can incremental support, for example 
 leave networking out if you want to. Also right now for all operating 
 systems, druntime depends on clib itself. Nim has a compiler switch 
 -os:any which means that the runtime will only depend on clib which 
 makes it very portable. That would have been nice in druntime too.


We really need to merge druntime and phobos in one. The entire 
distinction has long become nonsensical.

The new merged library ("hall" in honor of Asaph Hall who discovered 
Phobos and Deimos) would be built under a strict pay-as-you-go regime. 
That means an empty main uses virtually nothing, a main with a writeln() 
uses a little I/O, a main that allocates memory brings the GC in play 
(which auto-initializes on the first call), etc.

Templates and deduction must be heavily favored over old crummy C 
bindings. E.g. writeln() should actually link little else than a call to 
the fwrite() function.

We'd need at least 2 solid engineers for that.
Jun 15 2020
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Tuesday, 16 June 2020 at 01:08:16 UTC, Andrei Alexandrescu 
wrote:

 The new merged library ("hall" in honor of Asaph Hall who 
 discovered Phobos and Deimos) would be built under a strict 
 pay-as-you-go regime. That means an empty main uses virtually 
 nothing, a main with a writeln() uses a little I/O, a main that 
 allocates memory brings the GC in play (which auto-initializes 
 on the first call), etc.


How would that interact with architecture-based constraints (e.g. 
issues porting libraries in their entirety because of 
architecture-specific limitations, missing assembly 
implementations, etc.)?

Is it possible to ensure there is a well-defined "pay for what 
you can" separation of concerns, as well as "pay-as-you-go"?

P.S. I think you should call the library Asaph, it's more 
distinctive ;-)
Jun 18 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/18/20 8:10 AM, Joseph Rushton Wakeling wrote:

 On Tuesday, 16 June 2020 at 01:08:16 UTC, Andrei Alexandrescu wrote:

 The new merged library ("hall" in honor of Asaph Hall who discovered 
 Phobos and Deimos) would be built under a strict pay-as-you-go regime. 
 That means an empty main uses virtually nothing, a main with a 
 writeln() uses a little I/O, a main that allocates memory brings the 
 GC in play (which auto-initializes on the first call), etc.


 
 How would that interact with architecture-based constraints (e.g. issues 
 porting libraries in their entirety because of architecture-specific 
 limitations, missing assembly implementations, etc.)?


I assume porting a pay-as-you-go library would be easier to port than a 
monolithic one.


 Is it possible to ensure there is a well-defined "pay for what you can" 
 separation of concerns, as well as "pay-as-you-go"?


I don't know. That would be for the crack team that hasn't been 
assembled yet :o). What I can tell is that in a large project, whatever 
people pay attention to will work well. Whatever people don't pay 
attention to will work poorly. For example: in Phobos I paid little 
attention to things like object file size, linking in code, mutual 
dependencies, or separating memory allocation from other work. These 
aspects did not evolve well.


 P.S. I think you should call the library Asaph, it's more distinctive ;-)


Another decision for them!
Jun 18 2020
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Thursday, 18 June 2020 at 13:17:14 UTC, Andrei Alexandrescu 
wrote:

 I don't know. That would be for the crack team that hasn't been 
 assembled yet :o)


There are times I rather regret that I'm in a very busy job with 
lots to do ... :-)


 P.S. I think you should call the library Asaph, it's more 
 distinctive ;-)


 Another decision for them!


Sure, but it's the RIGHT decision ;-)
Jun 18 2020
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 15 June 2020 at 08:54:45 UTC, Dibyendu Majumdar wrote:

 I was wondering whether the runtime can be minimized to only 
 support the GC. Or is that already the case, i.e. the D runtime 
 is already the minimum needed for GC?


Eh, a good bulk of the d runtime is GC support, it would be hard 
to pull the gc out without most the rest of it.

In general, if you create a file called "object.d" in your work 
directory and compile it in, you can override druntime to nothing 
and build it back up from there just... that's a pain to get from 
nothing back up to GC point.
Jun 15 2020
next sibling parent reply IGotD- <nise nise.com> writes:
On Monday, 15 June 2020 at 12:21:08 UTC, Adam D. Ruppe wrote:

 Eh, a good bulk of the d runtime is GC support, it would be 
 hard to pull the gc out without most the rest of it.


I think he means the opposite, remove everything but GC. You 
bascially need GC in order to get the benefits of D but you don't 
necessarily need everything else.
Jun 15 2020
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 15 June 2020 at 16:45:21 UTC, IGotD- wrote:

 I think he means the opposite, remove everything but GC.


That's exactly what I meant... just I awkwardly worded it. I 
meant pull the GC out of full druntime and drop it into your 
minimal runtime so you have just it without the other stuff.


 You bascially need GC in order to get the benefits of D but you 
 don't necessarily need everything else.


eh GC is a big win with D, but you can get lots of D benefits 
without it too.
Jun 15 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/20 8:21 AM, Adam D. Ruppe wrote:

 On Monday, 15 June 2020 at 08:54:45 UTC, Dibyendu Majumdar wrote:

 I was wondering whether the runtime can be minimized to only support 
 the GC. Or is that already the case, i.e. the D runtime is already the 
 minimum needed for GC?


 
 Eh, a good bulk of the d runtime is GC support, it would be hard to pull 
 the gc out without most the rest of it.


How difficult would be to arrange things such that absolutely nothing in 
the GC makes it in the final ninary if not used?

My recollection is that we currently have a layering on top of the GC 
that is based on a misunderstanding of how linkers work, and also 
awkward and unnecessary.

Cutting that off and putting some good work into a self-loading GC 
(linked in if and only if used) would be amazing work.
Jun 15 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 16 June 2020 at 01:12:10 UTC, Andrei Alexandrescu 
wrote:

 How difficult would be to arrange things such that absolutely 
 nothing in the GC makes it in the final ninary if not used?


I don't know... I haven't looked at it all recently. I know the 
GC is used in part of the program initialization right now though 
(as are threads and other things) so that'd probably all have to 
change.


 My recollection is that we currently have a layering on top of 
 the GC that is based on a misunderstanding of how linkers work, 
 and also awkward and unnecessary.


the gc proxy thing is a mess yah.
Jun 16 2020
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/16/20 10:47 PM, Adam D. Ruppe wrote:

 On Tuesday, 16 June 2020 at 01:12:10 UTC, Andrei Alexandrescu wrote:

 How difficult would be to arrange things such that absolutely nothing 
 in the GC makes it in the final ninary if not used?


 
 I don't know... I haven't looked at it all recently. I know the GC is 
 used in part of the program initialization right now though (as are 
 threads and other things) so that'd probably all have to change.


There was some work done to remove shared static this() for the GC.
Jun 16 2020
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jun 17, 2020 at 02:47:50AM +0000, Adam D. Ruppe via Digitalmars-d wrote:

 On Tuesday, 16 June 2020 at 01:12:10 UTC, Andrei Alexandrescu wrote:

 How difficult would be to arrange things such that absolutely
 nothing in the GC makes it in the final ninary if not used?


 
 I don't know... I haven't looked at it all recently. I know the GC is
 used in part of the program initialization right now though (as are
 threads and other things) so that'd probably all have to change.


[...]

As of recent releases, GC initialization is actually lazy now, it will
not initialize anything until you actually use it.  Still one step away
from being totally optional, granted, but it shouldn't be *too*
difficult to make it so that, e.g., the GC code isn't even linked unless
there's a reference to one of its symbols somewhere.


T

-- 
You have to expect the unexpected. -- RL
Jun 16 2020
parent reply Jacob Carlborg <doob me.com> writes:
On 2020-06-17 06:08, H. S. Teoh wrote:


 As of recent releases, GC initialization is actually lazy now, it will
 not initialize anything until you actually use it.  Still one step away
 from being totally optional, granted, but it shouldn't be *too*
 difficult to make it so that, e.g., the GC code isn't even linked unless
 there's a reference to one of its symbols somewhere.


How would the GC know if it needs to initialize itself if you allocate 
the object at compile time, i.e.:

__gshared a = new Object; // module scope

Or is it expected that the above object fill never be freed?

What about the TLS data, isn't that used for the GC? This data is 
retrieved eagerly when a new thread is started. But since you'll always 
have at least one thread the data will always be retrieved?

-- 
/Jacob Carlborg
Jun 17 2020
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/17/20 5:06 AM, Jacob Carlborg wrote:

 On 2020-06-17 06:08, H. S. Teoh wrote:
 

 As of recent releases, GC initialization is actually lazy now, it will
 not initialize anything until you actually use it.  Still one step away
 from being totally optional, granted, but it shouldn't be *too*
 difficult to make it so that, e.g., the GC code isn't even linked unless
 there's a reference to one of its symbols somewhere.


 
 How would the GC know if it needs to initialize itself if you allocate 
 the object at compile time, i.e.:
 
 __gshared a = new Object; // module scope


Wouldn't that be allocated during compilation, and frozen at runtime 
(i.e. essentially static data)? That's my recollection of what happens.


 Or is it expected that the above object fill never be freed?


I don't think it's ever allocated actually.
Jun 17 2020
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 17 June 2020 at 12:48:03 UTC, Andrei Alexandrescu 
wrote:

 Wouldn't that be allocated during compilation, and frozen at 
 runtime (i.e. essentially static data)?


Yeah. You can modify the object (it is not const), but its memory 
address is statically allocated and will never change / be freed, 
even if you overwrite all the references. It is part of the exe 
file.

(this fact is actually useful for many things but it can be 
confusing)
Jun 17 2020
prev sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 17 Jun 2020 11:06:26 +0200 schrieb Jacob Carlborg:


 What about the TLS data, isn't that used for the GC? This data is
 retrieved eagerly when a new thread is started. But since you'll always
 have at least one thread the data will always be retrieved?


TLS memory is scanned by the GC, but it is managed / freed by the C 
library when a thread is stopped. Similar for static memory, which is as 
far as I know never freed. So as long as you don't have any heap memory, 
there's really no GC work to do.


-- 
Johannes
Jun 18 2020
prev sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 17 Jun 2020 02:47:50 +0000 schrieb Adam D. Ruppe:


 On Tuesday, 16 June 2020 at 01:12:10 UTC, Andrei Alexandrescu wrote:
 

 My recollection is that we currently have a layering on top of the GC
 that is based on a misunderstanding of how linkers work,
 and also awkward and unnecessary.


 
 the gc proxy thing is a mess yah.


Isn't the GC proxy only used for windows DLL support? IIRC we need it as 
we can't build druntime as a DLL right now.



-- 
Johannes
Jun 18 2020