www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DMD as cross-compiler

reply Walter Bright <newshound2 digitalmars.com> writes:
I realized that DMD is pretty close to being a cross-compiler, and won't need 
much work to go the rest of the way. I anticipate it could be very convenient 
when inconvenient to get the native DMD installed. It also should be useful for 
debugging problems with test suite failures :-/

Much of the static versioning in the backend is the result of converting it
from 
C with its #if statements. Being done with all that preprocessor legacy will be 
nice.

Working on a series of incremental PRs to make this happen.
Nov 11 2020
next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On Wednesday, 11 November 2020 at 10:49:54 UTC, Walter Bright 
wrote:

 I realized that DMD is pretty close to being a cross-compiler, 
 and won't need much work to go the rest of the way. I 
 anticipate it could be very convenient when inconvenient to get 
 the native DMD installed. It also should be useful for 
 debugging problems with test suite failures :-/

 Much of the static versioning in the backend is the result of 
 converting it from C with its #if statements. Being done with 
 all that preprocessor legacy will be nice.

 Working on a series of incremental PRs to make this happen.


Jonathon raised today that FreeBSD 11 -> FreeBSD 12 as been an 
issue for cross-compiling since the build scripts were replaced 
by a build.d source file.

It seems that cross-compiling between incompatible versions of 
libc on the same platform still needs to be sorted out.
Nov 11 2020
parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 11 November 2020 at 12:28:17 UTC, Iain Buclaw wrote:


 Jonathon raised today that FreeBSD 11 -> FreeBSD 12 as been an 
 issue for cross-compiling since the build scripts were replaced 
 by a build.d source file.


What's the issue with that? Use LDC and compile build.d to target 
FreeBSD 12. Sure, you need the SDK, but you should be able to 
copy that from a FreeBSD 12 machine.

--
/Jacob Carlborg
Nov 11 2020
parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On Thursday, 12 November 2020 at 06:45:37 UTC, Jacob Carlborg 
wrote:

 On Wednesday, 11 November 2020 at 12:28:17 UTC, Iain Buclaw 
 wrote:


 Jonathon raised today that FreeBSD 11 -> FreeBSD 12 as been an 
 issue for cross-compiling since the build scripts were 
 replaced by a build.d source file.


 What's the issue with that? Use LDC and compile build.d to 
 target FreeBSD 12. Sure, you need the SDK, but you should be 
 able to copy that from a FreeBSD 12 machine.


Judging from the state of druntime, LDC doesn't work on FreeBSD 
at all.

https://github.com/ldc-developers/druntime/blob/412467a452e2d12f561a2eace933dd44014af3c6/src/core/sys/freebsd/sys/mount.d#L291-L302

What's broken? pragma(mangle) string is invalid.
What needs to be fixed? There needs to be a new pragma(symver) 
added to support symbol versioning feature of ELF.

---

If it was me porting, I'd do it in the stages of:
1. Build FBSD11 compiler on FBSD11.
2. Cross-compile FBSD12 compiler on FBSD11 using compiler built 
in stage 1.
3. Copy FBSD12 compiler over to FBSD12.
4. Build FBSD12 compiler on FBSD12 using compiler built in stage 
2.

I think where Jonathon is going wrong is in skipping step 2 and 
jumping straight to three.  In the old make build system this 
just worked because (as I understand) there's a FBSD11 
compatibility layer, allowing old binaries to run on the new 
system.  But as that is no more, building build.d using the stage 
1 compiler on FBSD12 will result in a broken application (as it's 
linking against FBSD12 libc).
Nov 12 2020
prev sibling next sibling parent Andrea Fontana <nospam example.com> writes:
On Wednesday, 11 November 2020 at 10:49:54 UTC, Walter Bright 
wrote:

 I realized that DMD is pretty close to being a cross-compiler, 
 and won't need much work to go the rest of the way. I 
 anticipate it could be very convenient when inconvenient to get 
 the native DMD installed. It also should be useful for 
 debugging problems with test suite failures :-/

 Much of the static versioning in the backend is the result of 
 converting it from C with its #if statements. Being done with 
 all that preprocessor legacy will be nice.

 Working on a series of incremental PRs to make this happen.


Wow nice! But will it work out-of-the-box? I mean: what about 
linking with druntime, for example?

Andrea
Nov 11 2020
prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 11 November 2020 at 10:49:54 UTC, Walter Bright 
wrote:

 I realized that DMD is pretty close to being a cross-compiler, 
 and won't need much work to go the rest of the way. I 
 anticipate it could be very convenient when inconvenient to get 
 the native DMD installed. It also should be useful for 
 debugging problems with test suite failures :-/

 Much of the static versioning in the backend is the result of 
 converting it from C with its #if statements. Being done with 
 all that preprocessor legacy will be nice.

 Working on a series of incremental PRs to make this happen.


But what would be a target?
ARM?
MIPS?
Nov 11 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 11 November 2020 at 16:57:53 UTC, Stefan Koch wrote:

 But what would be a target?
 ARM?
 MIPS?


Windows, Linux, and Mac I'd imagine.

I tend to just run `wine dmd` to do linux-> windows but it might 
still be useful anyway.

this reminds me i still need to make my generic ldc x-compile 
setup program.
Nov 11 2020
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Nov 11, 2020 at 05:11:51PM +0000, Adam D. Ruppe via Digitalmars-d wrote:

 On Wednesday, 11 November 2020 at 16:57:53 UTC, Stefan Koch wrote:

 But what would be a target?
 ARM?
 MIPS?


 
 Windows, Linux, and Mac I'd imagine.
 
 I tend to just run `wine dmd` to do linux-> windows but it might still
 be useful anyway.


I prefer LDC for cross-compiling, as it can do both Windows and Android
cross-compile.



 this reminds me i still need to make my generic ldc x-compile setup
 program.


I just download and unpack both the Linux and Windows versions of LDC,
then add this to ldc2.conf (for Linux version):

	"(i686|x86_64)-.*-windows.msvc":
	{
	    switches = [
		"-defaultlib=phobos2-ldc,druntime-ldc",
		"-link-defaultlib-shared=false",
	    ];
	    lib-dirs = [
		"/path/to/ldc2-1.24.0-windows-x64/lib",
	    ];
	};

Then just compile with `-mtriple=x84_64-windows-msvc` and it Just
Works(tm).  It's been awesome.


T

-- 
VI = Visual Irritation
Nov 11 2020
prev sibling parent reply =?UTF-8?B?Um9iZXJ0IE0uIE3DvG5jaA==?= <robert.muench saphirion.com> writes:
On 11 Nov 2020 at 11:49:54 CET, "Walter Bright" <newshound2 digitalmars.com>
wrote:


 I realized that DMD is pretty close to being a cross-compiler, and won't need
 
 much work to go the rest of the way. I anticipate it could be very convenient
 
 when inconvenient to get the native DMD installed.


Digging into Go since some weeks, and the x-compiling of it is just super:

GOOS=windows GOARCH=amd64 go build

And you get a Windows EXE from OSX. That's it, zero config setup. The simpler
these things work, the better for the whole language.

-- 
Robert M. Münch
http://www.saphirion.com
smarter | better | faster
Nov 11 2020
parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 11 November 2020 at 21:09:14 UTC, Robert M. Münch 
wrote:


 Digging into Go since some weeks, and the x-compiling of it is 
 just super:

 GOOS=windows GOARCH=amd64 go build

 And you get a Windows EXE from OSX. That's it, zero config 
 setup. The simpler these things work, the better for the whole 
 language.


The reason why it works for Go is because they have their own 
full toolchain. That means, their own compiler (including 
backend), their own assembler, their own object format and their 
own linker. All of these are written to be cross-target. In 
addition to that, Go only interfaces with the kernel API. On 
Linux, that means direct system calls, they're not even going 
through the C wrappers. That's perfect for cross-compiling, since 
there's nothing to link against.

They used to do this on macOS as well, but had to back down, 
because on macOS the ABI is not the kernel, but the C wrappers 
for the system calls.

Zig is doing something similar, they're actually doing one 
better. Zig supports cross-compiling of C and C++ code as well (I 
don't think Go does this). Zig ships with Clang and the C 
standard library for various targets. But Zig uses their own 
frontend but the LLVM backend and the LLD linker. They do seem to 
have some trouble with linking some things on macOS. Also, on 
macOS they're cheating. Because they cannot ship the SDK, they 
solve the linking problem by allowing undefined symbols. I don't 
know if Go does the same.

Unfortunately it won't be that simple for D. D took the easy way 
out and relies both on the C standard library and other platform 
specific libraries implemented in C. While it's easy to 
cross-compile and object file, linking will not be as easy. 
There's LLD, which works on Windows and Linux, but it doesn't 
work for macOS. You also need the SDK to have something to link 
with. Targeting Windows, it might be this easy, using LDC, 
because it's already relying on LLD and the MinGW libraries.

--
/Jacob Carlborg
Nov 11 2020
next sibling parent reply Elronnd <elronnd elronnd.net> writes:
On Thursday, 12 November 2020 at 06:42:00 UTC, Jacob Carlborg 
wrote:

 Unfortunately it won't be that simple for D. D took the easy 
 way out and relies both on the C standard library and other 
 platform specific libraries implemented in C. While it's easy 
 to cross-compile and object file, linking will not be as easy.


FWIW I started working on some stubs for a pure-d libc a while 
back - https://github.com/moon-chilled/dlibc/.  If it were 
completed and integrated into druntime--or, at least, if enough 
of it were completed to support druntime+phobos--then it would 
make for a much nicer cross-compiling experience.



 Also, on macOS they're cheating. Because they cannot ship the 
 SDK, they solve the linking problem by allowing undefined 
 symbols. I don't know if Go does the same.


Perhaps a better solution would be to link to a stub dylib of 
libc?  (Assuming macos libc has a stable ABI.)



 There's LLD, which works on Windows and Linux, but it doesn't 
 work for macOS.


As of about a year and a half ago, its status was


 LLD for Mach-O has bitrotted significantly and isn't really 
 functional anymore (beyond the most basic programs). I'd 
 recommend sticking with ld64 as the linker.


(From 
http://lists.llvm.org/pipermail/cfe-dev/2019-March/061666.html)

so the situation may have improved in the mean time; looking at 
the logs 
(https://github.com/llvm/llvm-project/commits/master/lld/MachO) I 
see a fair amount of activity.  (In particular, commit 2e8e1bd 
adds support for stub dylibs, relevant to my above proposal.)
Nov 12 2020
parent reply Jacob Carlborg <doob me.com> writes:
On Thursday, 12 November 2020 at 18:58:56 UTC, Elronnd wrote:


 FWIW I started working on some stubs for a pure-d libc a while 
 back - https://github.com/moon-chilled/dlibc/.  If it were 
 completed and integrated into druntime--or, at least, if enough 
 of it were completed to support druntime+phobos--then it would 
 make for a much nicer cross-compiling experience.


That's nice. But I would prefer if druntime and Phobos did not 
depend on the C standard library, even if it's implemented in D. 
You're constrained by C and by the API of the C standard library. 
Implementing druntime and Phobos without the C standard library 
will give a big opportunity to do things better.


 Perhaps a better solution would be to link to a stub dylib of 
 libc?  (Assuming macos libc has a stable ABI.)


That's already what I'm working on [1].


 LLD for Mach-O has bitrotted significantly and isn't really 
 functional anymore (beyond the most basic programs). I'd 
 recommend sticking with ld64 as the linker.




Same here, working on that as well [2].

[1] 
https://github.com/d-cross-compiler/sdk-apple/releases/tag/v0.0.1
[2] 
https://github.com/d-cross-compiler/cctools-port/releases/tag/cctools-949.0.1-ld64-530

--
/Jacob Carlborg
Nov 12 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/12/2020 11:15 AM, Jacob Carlborg wrote:

 Implementing druntime and Phobos without the 
 C standard library will give a big opportunity to do things better.


How so? (Not that you're wrong, I'm just genuinely curious.)
Nov 12 2020
parent reply Jacob Carlborg <doob me.com> writes:
On Friday, 13 November 2020 at 03:33:05 UTC, Walter Bright wrote:

 On 11/12/2020 11:15 AM, Jacob Carlborg wrote:

 Implementing druntime and Phobos without the C standard 
 library will give a big opportunity to do things better.


 How so? (Not that you're wrong, I'm just genuinely curious.)


It might not necessary mean a better or different API for Phobos. 
But the lower level APIs could be better.

Classic examples are:

* Use arrays instead of pointers (or pointer length pair). 
`memcpy` would be:

void[] memcpy(void[] source, void[] destination); // I think it's 
more natural to pass the source first

* Better error handling. I think returning error codes or using 
`errno` is pretty bad error handling strategy. Even though 
exceptions might not be the right choice (unless something like 
this is used [1]), something like returning: `Result!(Value, 
Error)` might be better.

* No global symbols, i.e. not having to worry about conflicting 
names across multiple files

In short, all the reasons for which you created D in the first 
place ;)

Unfortunately there will still be some limitations due to how the 
kernel APIs use pointers and error codes/errno.

[1] 
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0709r0.pdf

--
/Jacob Carlborg
Nov 14 2020
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Saturday, 14 November 2020 at 19:19:02 UTC, Jacob Carlborg 
wrote:

 On Friday, 13 November 2020 at 03:33:05 UTC, Walter Bright 
 wrote:

 On 11/12/2020 11:15 AM, Jacob Carlborg wrote:

 Implementing druntime and Phobos without the C standard 
 library will give a big opportunity to do things better.


 How so? (Not that you're wrong, I'm just genuinely curious.)


 It might not necessary mean a better or different API for 
 Phobos. But the lower level APIs could be better.

 Classic examples are:

 * Use arrays instead of pointers (or pointer length pair). 
 `memcpy` would be:

 void[] memcpy(void[] source, void[] destination); // I think 
 it's more natural to pass the source first


Copying code will be same.


 * Better error handling. I think returning error codes or using 
 `errno` is pretty bad error handling strategy. Even though 
 exceptions might not be the right choice (unless something like 
 this is used [1]), something like returning: `Result!(Value, 
 Error)` might be better.


Errno can be covered by druntime like in: 
https://dlang.org/phobos/core_memory.html#.pureMalloc


 * No global symbols, i.e. not having to worry about conflicting 
 names across multiple files


You don't have to worry even now, even if there is a conflict, 
this can be resolved by linking settings (I propose to switch 
from DUB to Meson :-))

Really, libc is not (only) C library, but de-facto standard for 
accessing to OS syscalls. It is possible to drop it but it 
doesn't make any sense, it just adds extra work to maintain a 
huge compatibility layer for all supported OSes.

I once thought differently, but recently delved deeper into 
druntime and looked how it works. In total, I think, there are 
about ~15 calls to libc, not so many.
Nov 14 2020
parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Sunday, 15 November 2020 at 07:21:28 UTC, Denis Feklushkin 
wrote:

 [...]

 Really, libc is not (only) C library, but de-facto standard for 
 accessing to OS syscalls. It is possible to drop it but it 
 doesn't make any sense, it just adds extra work to maintain a 
 huge compatibility layer for all supported OSes.

 I once thought differently, but recently delved deeper into 
 druntime and looked how it works. In total, I think, there are 
 about ~15 calls to libc, not so many.


Only for the OSes where C was born on, UNIX and POSIX clones.

There are still several platforms around where this is not the 
case, because their main systems programming language isn't C, 
e.g. IBM and Unisys mainframes.

Embedded platforms that use their own toolchains without any kind 
of POSIX compatibility.

And desktop OSes like Windows, which follows the tradition of C 
and C++ compilers outside the UNIX world, where each compiler 
vendor ships their own ISO compliant library, and the OS syscalls 
are completly unrelated, e.g. ZeroMemory() instead of memset(). 
Amiga, Atari, OS/2, BeOS, Symbian, Brew, Mac OS pre OS X, Newton, 
PalmOS, are all examples of this tradition.
Nov 14 2020
parent reply Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Sunday, 15 November 2020 at 07:51:52 UTC, Paulo Pinto wrote:

 On Sunday, 15 November 2020 at 07:21:28 UTC, Denis Feklushkin 
 wrote:

 [...]

 Really, libc is not (only) C library, but de-facto standard 
 for accessing to OS syscalls. It is possible to drop it but it 
 doesn't make any sense, it just adds extra work to maintain a 
 huge compatibility layer for all supported OSes.

 I once thought differently, but recently delved deeper into 
 druntime and looked how it works. In total, I think, there are 
 about ~15 calls to libc, not so many.


 Only for the OSes where C was born on, UNIX and POSIX clones.

 There are still several platforms around where this is not the 
 case, because their main systems programming language isn't C, 
 e.g. IBM and Unisys mainframes.

 Embedded platforms that use their own toolchains without any 
 kind of POSIX compatibility.

 And desktop OSes like Windows, which follows the tradition of C 
 and C++ compilers outside the UNIX world, where each compiler 
 vendor ships their own ISO compliant library, and the OS 
 syscalls are completly unrelated, e.g. ZeroMemory() instead of 
 memset(). Amiga, Atari, OS/2, BeOS, Symbian, Brew, Mac OS pre 
 OS X, Newton, PalmOS, are all examples of this tradition.


Agree. Also threads are not part of the libc, but Posix: if 3-rd 
party libc provides C standard functions it does not necessarily 
provides pthread interface.

In this regard, I want to propose related idea to move this 
dependent from libc and threads and etc parts into a separate 
druntime package ("backend") so that those who wish can 
themselves implement  interfaces to these or another more 
appropriate OS calls.

Something like:
https://github.com/denizzzka/druntime/blob/non_posix/src/core/thread/osthread.d#L217
===>
https://github.com/denizzzka/d_c_arm_test/blob/master/d/freertos_druntime_backend/external/core/thread.d#L396

but without all these "version (DruntimeAbstractRt)" branches.

This will simplify druntime main code (get rid of many versions 
branches inside of the druntime) and make it possible to easily 
implement support for various not common operating systems such 
as HaikuOS or druntime for bare-metal.
Nov 15 2020
parent Kagamin <spam here.lot> writes:
On Sunday, 15 November 2020 at 08:12:01 UTC, Denis Feklushkin 
wrote:

 Agree. Also threads are not part of the libc, but Posix: if 
 3-rd party libc provides C standard functions it does not 
 necessarily provides pthread interface.


It's catching on 
https://en.cppreference.com/w/c/thread/thrd_create
Nov 16 2020
prev sibling parent =?UTF-8?B?Um9iZXJ0IE0uIE3DvG5jaA==?= <robert.muench saphirion.com> writes:
On 12 Nov 2020 at 07:42:00 CET, "Jacob Carlborg" <doob me.com> wrote:


 The reason why it works for Go is because they have their own 
 full toolchain. That means, their own compiler (including 
 backend), their own assembler, their own object format and their 
 own linker. All of these are written to be cross-target.


And being x-target is a key design goal, which is really an amazing
out-of-the-box experience.

Of course you have some (silent) different behavior for some Go std lib
stuff... which can be of surprise. The rust guys take a much more pedantic
approach here, which IMO is the cleaner way to do it.

However, Go's goal is getting stuff done. And it's really delivering on that.

-- 
Robert M. Münch
http://www.saphirion.com
smarter | better | faster
Nov 12 2020