www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - What stops DMD from cross-compiling?

reply Rel <relmail rambler.ru> writes:
So, okey, bare with me here. As I once told here before the only 
one thing I love about Golang is the ability to easily 
cross-compile code from any supported OS targeting any supported 
OS. So I was thinking what actually stops DMD from doing the same 
thing? DMD has own backends targeting X86 and x64 already, it 
seems to work well with LLD (at least for me) that was integrated 
not so long ago, and LLD has the ability to cross-link.

So I'm not a compiler expert here, and I'm not familiar with the 
DMD code base, but from a user perspective the one huge 
showstopper for bringing the same level of cross-compilation to 
DMD like Golang has seems to be the dependency on C-library. As 
of 2.079 we can write code that is not dependent on the druntime 
and libc, so technically we can reimplement the needed parts of 
cruntime in D and make it the part of druntime itself. What do 
you think about it?

Also we will need some libraries that would generate bindings 
from extern functions to actual OS library code, like for example 
kernel32.lib on Windows. The library doesn't have actual code, it 
just binds the symbol to the executable import table. Pascal 
programming language doesn't seem to have the same notion, when 
you declare extern function you just declare it with the actual 
library name (like kernel32.dll on windows) and the compiler 
generates the proper imports. Can the behavior like this be 
implemented for DMD? How we can solve this problem otherwise?

And any other thought on the topic is welcomed, I like talking 
about compilers and stuff with some wise and experienced people. 
Thanks!
Apr 27 2018
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2018-04-27 12:56, Rel wrote:
 So, okey, bare with me here. As I once told here before the only one 
 thing I love about Golang is the ability to easily cross-compile code 
 from any supported OS targeting any supported OS. So I was thinking what 
 actually stops DMD from doing the same thing? DMD has own backends 
 targeting X86 and x64 already, it seems to work well with LLD (at least 
 for me) that was integrated not so long ago, and LLD has the ability to 
 cross-link.
 
 So I'm not a compiler expert here, and I'm not familiar with the DMD 
 code base, but from a user perspective the one huge showstopper for 
 bringing the same level of cross-compilation to DMD like Golang has 
 seems to be the dependency on C-library. 
DMD can cross-compile between 32-bit and 64-bit on the same platform. To targeting a different platform than the host the code in DMD needs to be reorganized a bit. When compiling the compiler it will only include support for targeting the same platform as you're compiling on. Currently it's using #ifdefs to decide which target should be supported. This occurs the compile time of the compiler. This needs to occur at runtime instead. It doesn't depend on anything expect for the standard C library so it should be fairly straight forward to fix.
 As of 2.079 we can write code 
 that is not dependent on the druntime and libc, so technically we can 
 reimplement the needed parts of cruntime in D and make it the part of 
 druntime itself. What do you think about it?
That would work, but it requires a lot of effort. It's not only depending on the C standard library, it also depends on other system functionality that are not part of the kernel. For example, the thread local storage implementation depends on the platform. If we're only using the kernel that would be needed to be implemented as well. It's possible to use the libraries provided by the target platform when cross-compiling. I've done that with LDC and created two Dockerfiles, one targeting Windows [1] and one targeting macOS [2]. Note, the SDKs for macOS and Windows are downloaded from Dropbox accounts. [1] https://github.com/jacob-carlborg/docker-ldc-windows/blob/master/Dockerfile [2] https://github.com/jacob-carlborg/docker-ldc-darwin/blob/master/Dockerfile -- /Jacob Carlborg
Apr 27 2018
parent reply Rel <relmail rambler.ru> writes:
On Friday, 27 April 2018 at 15:31:37 UTC, Jacob Carlborg wrote:
 DMD can cross-compile between 32-bit and 64-bit on the same 
 platform. To targeting a different platform than the host the 
 code in DMD needs to be reorganized a bit. When compiling the 
 compiler it will only include support for targeting the same 
 platform as you're compiling on. Currently it's using #ifdefs 
 to decide which target should be supported. This occurs the 
 compile time of the compiler. This needs to occur at runtime 
 instead. It doesn't depend on anything expect for the standard 
 C library so it should be fairly straight forward to fix.
Well, that's a good news.
 That would work, but it requires a lot of effort. It's not only 
 depending on the C standard library, it also depends on other 
 system functionality that are not part of the kernel. For 
 example, the thread local storage implementation depends on the 
 platform. If we're only using the kernel that would be needed 
 to be implemented as well.
Something like memcpy and similar stuff is easy to implement. If we are talking about Linux stuff like open, malloc and etc can be implemented either by using syscalls or by generating the binding to the libc.so of some minimal version that we decide to support (like Pascal compiler do). If we are talking about Windows all of the needed functions can be reimplemented using API provided by kernel32.dll and friends.
 It's possible to use the libraries provided by the target 
 platform when cross-compiling. I've done that with LDC and 
 created two Dockerfiles, one targeting Windows [1] and one 
 targeting macOS [2]. Note, the SDKs for macOS and Windows are 
 downloaded from Dropbox accounts.
Downloading some SDK's from dropbox accounts is the thing I'd like to avoid, I'd like the Dlang compiler to be a self-contained toolchain. This doesn't mean something, just a personal preference.
Apr 28 2018
parent Jacob Carlborg <doob me.com> writes:
On 2018-04-28 10:49, Rel wrote:

 Something like memcpy and similar stuff is easy to implement.
 If we are talking about Linux stuff like open, malloc and etc
 can be implemented either by using syscalls or by generating
 the binding to the libc.so of some minimal version that we
 decide to support (like Pascal compiler do). If we are talking
 about Windows all of the needed functions can be reimplemented
 using API provided by kernel32.dll and friends.
For macOS I'm thinking functions that are implemented in the dynamic loader [1] or other platform specific functions [2]. For macOS the thread local implementation is in the dynamic loader. It will setup TLS variables before C main function is called. That would need to be reimplemented.
 Downloading some SDK's from dropbox accounts is the thing I'd
 like to avoid, 
If you have access to a machine running the target system you can copy/extract the SDK yourself. If you don't have there's Travis CI which has support for macOS and AppVeyor which supports Windows which you can copy the SDK from. When it comes to the SDK for macOS we might be able to ship our own SDK. Most of the libraries are dynamic libraries. The SDK doesn't actually contain any real libraries, only YAML files containing some metadata and the symbols the library defines. We could generate those YAML files ourselves based on the real libraries. I don't know what the SDK EULA says about this. But since we created the files I think we should be able to distribute them.
 I'd like the Dlang compiler to be a self-contained
 toolchain.
That would be ideal yes. [1] https://github.com/dlang/druntime/blob/master/src/core/sys/darwin/mach/dyld.d [2] https://github.com/dlang/druntime/blob/master/src/core/sys/darwin/mach/getsect.d -- /Jacob Carlborg
Apr 29 2018
prev sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Friday, 27 April 2018 at 10:56:50 UTC, Rel wrote:
 So I was thinking what actually stops DMD from doing the same 
 thing?
IIRC: 1. Linking. DMD uses platform linkers, and the only thing it can "link" by itself is a library (with -lib). 2. Import libraries, for Windows COFF targets. (OMF import libraries are in the Windows distribution.) 3. Cross-platform C compilation Platform headers (some parts of Phobos/Druntime are in C), with all its dependencies, i.e. C compiler support, platform and libc headers... 4. When we link to the libc statically (i.e. on Windows), that is also needed. 5. CTFE needs to emulate the target architecture, which creates some issues with things like pointer size and floating-point calculations. Probably the closest thing to cross-platform building would be through LDC, as LLVM already includes everything for C cross compilation.
Apr 28 2018
parent Jacob Carlborg <doob me.com> writes:
On 2018-04-28 12:02, Vladimir Panteleev wrote:

 Probably the closest thing to cross-platform building would be through 
 LDC, as LLVM already includes everything for C cross compilation.
Except for a C standard library. -- /Jacob Carlborg
Apr 29 2018