www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.ldc - Cross compilation idea.

reply Rel <relmail rambler.ru> writes:
Lately I've been looking at Zig programming language. Besides 
other interesting features like error handling and automatically 
importing C headers, one feature made me really excited. Zig 
includes a minified version of musl's, glibc's and mingw's CRT, 
so it can cross compile stuff very easy. And for whatever reason 
it includes a C cross compiler as well (I guess they already had 
libclang bundled for importing C headers, so it was kinda easy to 
use it to compile C).

As far as I understand there are few things that forbids LDC from 
cross compiling as easy as Zig:
1) Cross linker. Zig uses LLVM's linker which is called LLD. Can 
LDC use LLD to target all suppored operating systems?
2) Cross libc. Zig bundles minified version of musl, glibc and 
mingw (I didn't quite understood if it can cross compile to 
macosx). Do you think LDC could do the same or reuse the work 
done by the Zig team?
3) Cross druntime and phobos. I guess LDC could distribute 
prebuilt versions of druntime and phobos for most used operating 
system. Or just build and cache it the first time the user tries 
to cross compile to new target the first time. What do you think?
Aug 01 2020
next sibling parent user1234 <user1234 12.de> writes:
On Saturday, 1 August 2020 at 10:26:12 UTC, Rel wrote:

 Lately I've been looking at Zig programming language. Besides 
 other interesting features like error handling and 
 automatically importing C headers, one feature made me really 
 excited. Zig includes a minified version of musl's, glibc's and 
 mingw's CRT, so it can cross compile stuff very easy. And for 
 whatever reason it includes a C cross compiler as well (I guess 
 they already had libclang bundled for importing C headers, so 
 it was kinda easy to use it to compile C).

 As far as I understand there are few things that forbids LDC 
 from cross compiling as easy as Zig:
 1) Cross linker. Zig uses LLVM's linker which is called LLD. 
 Can LDC use LLD to target all suppored operating systems?
 2) Cross libc. Zig bundles minified version of musl, glibc and 
 mingw (I didn't quite understood if it can cross compile to 
 macosx). Do you think LDC could do the same or reuse the work 
 done by the Zig team?
 3) Cross druntime and phobos. I guess LDC could distribute 
 prebuilt versions of druntime and phobos for most used 
 operating system. Or just build and cache it the first time the 
 user tries to cross compile to new target the first time. What 
 do you think?


IIRC the windows versions of the devel tools such as D-Scanner or 
DCD, available for download, are generated on CI services that 
run on Ubtuntu VMs and based on cross LDC abilities to 
cross-compile.

So this might be mostly "ok" already. Maybe not for musl but 
cross compilation works to some extent.
Aug 01 2020
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2020-08-01 12:26, Rel wrote:

 Lately I've been looking at Zig programming language. Besides other 
 interesting features like error handling and automatically importing C 
 headers, one feature made me really excited. Zig includes a minified 
 version of musl's, glibc's and mingw's CRT, so it can cross compile 
 stuff very easy. And for whatever reason it includes a C cross compiler 
 as well (I guess they already had libclang bundled for importing C 
 headers, so it was kinda easy to use it to compile C).


I've been thinking of the same thing myself. I've talked with kinke (via 
private email) about this. He wasn't sold on the idea. But I've started 
to work on a project for this [1].


 As far as I understand there are few things that forbids LDC from cross 
 compiling as easy as Zig:
 1) Cross linker. Zig uses LLVM's linker which is called LLD. Can LDC use 
 LLD to target all suppored operating systems?


There are some things that are missing for LLD to work for D 
applications for targeting Apple platforms.

My idea was to use the official Apple linker (ld64). Someone has already 
ported ld64 to other Posix platforms, including MinGW. I've setup a CI 
job to build ld64 for Linux [2].


 2) Cross libc. Zig bundles minified version of musl, glibc and mingw (I 
 didn't quite understood if it can cross compile to macosx). Do you think 
 LDC could do the same or reuse the work done by the Zig team?


It can cross-compiler to macOS, by using, in my opinion, an ugly hack. 
The Zig compiler adds a flag to the linker to ignore undefined symbols. 
Since the whole SDK is dynamically linked on Apple platforms, the only 
thing you need the SDK for, is to verify that the symbols the 
application is using exist. It cannot cross-compiler C/C++ code to macOS 
because it doesn't include the header files.

My solution to this problem was to generate SDK in a CI job. These days 
the libraries in the SDK don't contain any code, since everything is 
dynamically linked. Instead, the SDK consists of .tbd files, which are 
basically YAML files. It more or less only contains the symbol names. 
Apple provides a tool to generate these files based on the actual 
library files. I've built a tool [3] that automatically generates the 
SDK. Then I have a separate CI job that uses that tool to generate the 
SDK [4] and publish it on GitHub. I'm not sure what the Xcode license 
say about this. But I created the files myself and not copied any 
existing files.


 3) Cross druntime and phobos. I guess LDC could distribute prebuilt 
 versions of druntime and phobos for most used operating system. Or just 
 build and cache it the first time the user tries to cross compile to new 
 target the first time. What do you think?


It could, but my idea was to have more or less an archive for each 
target available online. The first time the target is used it will 
automatically download the archive, unpack it and install it so it can 
be used.

[1] https://github.com/d-cross-compiler

[2] 
https://github.com/d-cross-compiler/cctools-port/releases/tag/cctools-949.0.1-ld64-530

[3] https://github.com/d-cross-compiler/apple-sdk-generator

[4] https://github.com/d-cross-compiler/sdk-apple/releases/tag/v0.0.1

-- 
/Jacob Carlborg
Aug 01 2020
prev sibling next sibling parent IGotD- <nise nise.com> writes:
On Saturday, 1 August 2020 at 10:26:12 UTC, Rel wrote:

 As far as I understand there are few things that forbids LDC 
 from cross compiling as easy as Zig:
 1) Cross linker. Zig uses LLVM's linker which is called LLD. 
 Can LDC use LLD to target all suppored operating systems?


Potentially yes. You can use GNU linker as well.


 2) Cross libc. Zig bundles minified version of musl, glibc and 
 mingw (I didn't quite understood if it can cross compile to 
 macosx). Do you think LDC could do the same or reuse the work 
 done by the Zig team?


druntime already supports musl and probably glibc as it is 
usually default for Linux. I don't think we should bundle various 
C-libs in the D distribution as it would just baggage the D 
distribution and far from all would use it. Also I don't see the 
benefit of a "minimized" version. The C-library should be 
installed as part of the system with the full version. For 
embedded targets things starts to become more complicated as it's 
full of special cases.


 3) Cross druntime and phobos. I guess LDC could distribute 
 prebuilt versions of druntime and phobos for most used 
 operating system. Or just build and cache it the first time the 
 user tries to cross compile to new target the first time. What 
 do you think?


LDC has made a compile druntime/phobos tool and this works quite 
well for the supported targets.


I regulary cross compile using LDC but using the GNU linker 
because LLD is a bit immature and this worked well for me without 
much hazzle. The big problem is the structure of the 
druntime/phobos so that it can be adapted to new systems more 
easily. This has been discussed in this forum previously. I see 
no problems supporting more types of C-libraries but 
unfortunately all of them have their own quirks so it is not just 
a plug and play operation. Before that I think we must get rid of

version(this) { } else version(that) {}

in druntime because as we support more targets it is going to 
become a merge hell.
Aug 02 2020
prev sibling parent kinke <noone nowhere.com> writes:
On Saturday, 1 August 2020 at 10:26:12 UTC, Rel wrote:

 1) Cross linker. Zig uses LLVM's linker which is called LLD. 
 Can LDC use LLD to target all suppored operating systems?


Almost, as already mentioned (proper Apple targets support is 
underway AFAIK, and our custom Android TLS emulation doesn't work 
with LLD out of the box either). LLD is integrated in official 
LDC builds (-link-internally), but Posix systems require quite a 
lot of extra flags when invoking the linker directly instead of a 
preconfigured C compiler as linker driver.


 2) Cross libc. Zig bundles minified version of musl, glibc and 
 mingw (I didn't quite understood if it can cross compile to 
 macosx). Do you think LDC could do the same or reuse the work 
 done by the Zig team?


We have been doing so for a while with the Windows package, which 
includes MinGW-based libs and makes cross-compilation to Windows 
a trivial thing on all hosts. For Posix targets, I clearly prefer 
a C cross-toolchain (à la `apt-get install 
gcc-aarch64-linux-gnu`).


 3) Cross druntime and phobos. I guess LDC could distribute 
 prebuilt versions of druntime and phobos for most used 
 operating system.


That's what we do, see 
https://wiki.dlang.org/Cross-compiling_with_LDC. You just need to 
download the according package, extract the libs and extend the 
ldc2.conf file. Automating these last steps (doable manually in 
1-2 mins once you've already done it once) has been discussed 
(multiple times) before.
Aug 02 2020