www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.ldc - Add targets for release binaries? (1.3+)

reply Johan Engelen <j j.nl> writes:
Would it make sense to release 1.3 with more targets enabled than 
the host?
X86, ARM, AArch64 ?

- Johan
Apr 22 2017
next sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen wrote:

 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?


I don't really think it makes sense for end users, as we would 
need to ship the respective standard libraries and a way to 
select between cross-compilation targets as well.

That being said, purely from a "marketing" perspective it might 
be nice to increase visibility for ARM support in LDC. (Users 
might read the -help output or just try to specify a target 
triple.)

The primary motivation behind shipping host-only releases was 
binary size - have you checked recently how the two build compare?

  -- David
Apr 22 2017
next sibling parent Johan Engelen <j j.nl> writes:
On Saturday, 22 April 2017 at 11:55:39 UTC, David Nadlinger wrote:

 On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen wrote:

 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?


 I don't really think it makes sense for end users, as we would 
 need to ship the respective standard libraries and a way to 
 select between cross-compilation targets as well.


The extra standard libraries could be separate downloads (hosted 
by whoever), but there is also `-betterC`.
We could include a script that builds Phobos for a certain triple 
(I was already thinking about including such a script for 
building Phobos with LTO), but for druntime we don't include all 
the source...


 The primary motivation behind shipping host-only releases was 
 binary size - have you checked recently how the two build 
 compare?


Just tested: adding ARM and AArch64 to X86 on Mac adds 5 MB (LTO 
build).
Apr 22 2017
prev sibling parent reply Adrian Matoga <dlang.spam matoga.info> writes:
On Saturday, 22 April 2017 at 11:55:39 UTC, David Nadlinger wrote:

 On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen wrote:

 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?


 I don't really think it makes sense for end users, as we would 
 need to ship the respective standard libraries and a way to 
 select between cross-compilation targets as well.


Moreover, "ARM" alone doesn't mean anything in this context. The 
sheer number of platforms using different ABIs, architecture 
versions, library versions and configurations makes it virtually 
impossible to even build druntime and Phobos in a way that they 
could later be at least linked correctly for all ARM-based 
platforms.

That being said, having binary packages that work out-of-the-box 
for a few popular targets, such as Raspbian, Android and OS X, 
wouldn't be a bad thing.
Apr 28 2017
parent reply David Nadlinger via digitalmars-d-ldc <digitalmars-d-ldc puremagic.com> writes:
On 28 Apr 2017, at 9:23, Adrian Matoga via digitalmars-d-ldc wrote:

 Moreover, "ARM" alone doesn't mean anything in this context. The sheer 
 number of platforms using different ABIs, architecture versions, 
 library versions and configurations makes it virtually impossible to 
 even build druntime and Phobos in a way that they could later be at 
 least linked correctly for all ARM-based platforms.

 That being said, having binary packages that work out-of-the-box for a 
 few popular targets, such as Raspbian, Android and OS X, wouldn't be a 
 bad thing.


True. But as you mention, there are a couple of popular targets. For 
example, Debian/Ubuntu/Fedora/Arch/… all have an armhf distribution 
which targets ARMv7 without NEON. A binary package for that would 
already cover a huge portion of people just looking to try out LDC on 
their ARM dev board. (RP1 is of course its own thing, unfortunately.)

  — David
Apr 30 2017
parent reply Adrian Matoga <dlang.spam matoga.info> writes:
On Sunday, 30 April 2017 at 15:44:50 UTC, David Nadlinger wrote:

 On 28 Apr 2017, at 9:23, Adrian Matoga via digitalmars-d-ldc 
 wrote:

 Moreover, "ARM" alone doesn't mean anything in this context. 
 The sheer number of platforms using different ABIs, 
 architecture versions, library versions and configurations 
 makes it virtually impossible to even build druntime and 
 Phobos in a way that they could later be at least linked 
 correctly for all ARM-based platforms.

 That being said, having binary packages that work 
 out-of-the-box for a few popular targets, such as Raspbian, 
 Android and OS X, wouldn't be a bad thing.


 True. But as you mention, there are a couple of popular 
 targets. For example, Debian/Ubuntu/Fedora/Arch/… all have an 
 armhf distribution which targets ARMv7 without NEON. A binary 
 package for that would already cover a huge portion of people 
 just looking to try out LDC on their ARM dev board. (RP1 is of 
 course its own thing, unfortunately.)


Yeah, GDC has a binary release for cross-compiling to armhf, 
which works well for RPi (including v1) out of the box. It's 
really a huge time saver.

What I miss the most in LDC is the ability to build and use 
druntime/Phobos for different targets (including subtargets, e.g. 
different ARM architecture versions and ABIs) with the same 
compiler. I'd like to add it myself, but I don't really know 
where to start. Perhaps it's an idea to be discussed at this 
DConf's hackathon.
May 05 2017
parent reply David Nadlinger via digitalmars-d-ldc <digitalmars-d-ldc puremagic.com> writes:
On 5 May 2017, at 9:52, Adrian Matoga via digitalmars-d-ldc wrote:

 What I miss the most in LDC is the ability to build and use 
 druntime/Phobos for different targets (including subtargets, e.g. 
 different ARM architecture versions and ABIs) with the same compiler. 
 I'd like to add it myself, but I don't really know where to start. 
 Perhaps it's an idea to be discussed at this DConf's hackathon.


The biggest issue to solve here is the rest of the toolchain (linker, 
system libraries, …). LDC's part can easily be done by adding a small 
piece of code to the config file handling such that different sections 
are matched depending on the chosen target (where you can then set 
library lookup paths for druntime/Phobos, the GCC excecutable to use for 
cross linking, etc.).

Unfortunately I'm not at DConf this year, but I'd be happy to help out 
with any questions you might have implementing something like that.

  — David
May 05 2017
next sibling parent reply Johan Engelen <j j.nl> writes:
On Friday, 5 May 2017 at 12:49:32 UTC, David Nadlinger wrote:

 On 5 May 2017, at 9:52, Adrian Matoga via digitalmars-d-ldc 
 wrote:

 What I miss the most in LDC is the ability to build and use 
 druntime/Phobos for different targets (including subtargets, 
 e.g. different ARM architecture versions and ABIs) with the 
 same compiler.




Perhaps what would be nice is a script that we can ship with LDC, 
that then compiles Phobos/druntime with specific compiler 
options: target triple, LTO flag (!), ...
LDC will be able to link libraries in the future, so the script 
will not have to deal with some of the platform issues.

-Johan
May 05 2017
next sibling parent reply Adrian Matoga <dlang.spam matoga.info> writes:
On Friday, 5 May 2017 at 13:43:31 UTC, Johan Engelen wrote:

 On Friday, 5 May 2017 at 12:49:32 UTC, David Nadlinger wrote:

 On 5 May 2017, at 9:52, Adrian Matoga via digitalmars-d-ldc 
 wrote:

 What I miss the most in LDC is the ability to build and use 
 druntime/Phobos for different targets (including subtargets, 
 e.g. different ARM architecture versions and ABIs) with the 
 same compiler.




 Perhaps what would be nice is a script that we can ship with 
 LDC, that then compiles Phobos/druntime with specific compiler 
 options: target triple, LTO flag (!), ...
 LDC will be able to link libraries in the future, so the script 
 will not have to deal with some of the platform issues.

 -Johan


Yes, and I believe the "script" is doable in a few lines inserted 
in CMakeLists we already have. I need to improve my CMakese to 
prove it.

For my target, I only had to change few things in 
runtime/CMakeLists.txt: set CMAKE_C_COMPILER, and add -triple and 
--sysroot.
I believe in most cases it'll be just a matter of iterating over 
a list of targets, each having its specific flags and paths, and 
for each of them rebuilding druntime and Phobos in a separate 
directory.

The other part of it is using the LDC you've built this way 
later. You should be able to conveniently specify which of these 
targets you want. Technically, you can already use -mtriple, 
-gcc, -defaultlib etc., but it could be made easier.
May 05 2017
parent kinke <noone nowhere.com> writes:
On Friday, 5 May 2017 at 15:59:21 UTC, Adrian Matoga wrote:

 Yes, and I believe the "script" is doable in a few lines 
 inserted in CMakeLists we already have. I need to improve my 
 CMakese to prove it.


Our CMake scripts are a holy mess. An approach patching the 
runtime's CMakeLists.txt in the few required places, re-running 
CMake (with `-DCMAKE_C_COMPILER=... -DD_FLAGS=-w;-mtriple=...` 
etc.) and then building the new runtime libs (obviously using the 
already built LDC), all of this for each desired cross-target, 
might be a simpler approach.


 The other part of it is using the LDC you've built this way 
 later. You should be able to conveniently specify which of 
 these targets you want. Technically, you can already use 
 -mtriple, -gcc, -defaultlib etc., but it could be made easier.


See my post above; all you need is an extended config file and 
then specifying `-mtriple=...` for the LDC command line(s).
May 05 2017
prev sibling parent David Nadlinger via digitalmars-d-ldc <digitalmars-d-ldc puremagic.com> writes:
On 5 May 2017, at 14:43, Johan Engelen via digitalmars-d-ldc wrote:

 LDC will be able to link libraries in the future, so the script will 
 not have to deal with some of the platform issues.


I believe you'd still need to have the target toolchain around at build 
time for shared libraries.

  — David
May 05 2017
prev sibling parent reply kinke <noone nowhere.com> writes:
On Friday, 5 May 2017 at 12:49:32 UTC, David Nadlinger wrote:

 LDC's part can easily be done by adding a small piece of code 
 to the config file handling such that different sections are 
 matched depending on the chosen target (where you can then set 
 library lookup paths for druntime/Phobos, the GCC excecutable 
 to use for cross linking, etc.).


Rainer (IIRC) already implemented that a year ago or so.
Just add additional sections, named exactly like the target 
triple, to the config file, e.g.:

`
default:
{
     switches = [
         "-I%%ldcbinarypath%%/../import/ldc",
         "-I%%ldcbinarypath%%/../import",
         "-L-L%%ldcbinarypath%%/../lib64",
         "-defaultlib=phobos2-ldc,druntime-ldc",
         "-debuglib=phobos2-ldc-debug,druntime-ldc-debug"
     ];
};
armv7-none-linux-android:
{
     switches = [
         "-I%%ldcbinarypath%%/../import/ldc",
         "-I%%ldcbinarypath%%/../import",
         "-gcc=MY_CROSS_LINKING_GCC",
         "-L-LMY_ARM_LIBS_DIR",
         "-defaultlib=phobos2-ldc,druntime-ldc",
         "-debuglib=phobos2-ldc-debug,druntime-ldc-debug"
     ];
};
`

We could spice it up by allowing wildcards in the section names 
at some point, but cross-compiling druntime/Phobos, installing an 
appropriate cross-gcc and modifying the LDC config file is really 
all what's needed.
May 05 2017
parent reply Adrian Matoga <dlang.spam matoga.info> writes:
On Friday, 5 May 2017 at 15:52:44 UTC, kinke wrote:

 On Friday, 5 May 2017 at 12:49:32 UTC, David Nadlinger wrote:

 LDC's part can easily be done by adding a small piece of code 
 to the config file handling such that different sections are 
 matched depending on the chosen target (where you can then set 
 library lookup paths for druntime/Phobos, the GCC excecutable 
 to use for cross linking, etc.).


 Rainer (IIRC) already implemented that a year ago or so.
 Just add additional sections, named exactly like the target 
 triple, to the config file, e.g.:

 `
 default:
 {
     switches = [
         "-I%%ldcbinarypath%%/../import/ldc",
         "-I%%ldcbinarypath%%/../import",
         "-L-L%%ldcbinarypath%%/../lib64",
         "-defaultlib=phobos2-ldc,druntime-ldc",
         "-debuglib=phobos2-ldc-debug,druntime-ldc-debug"
     ];
 };
 armv7-none-linux-android:
 {
     switches = [
         "-I%%ldcbinarypath%%/../import/ldc",
         "-I%%ldcbinarypath%%/../import",
         "-gcc=MY_CROSS_LINKING_GCC",
         "-L-LMY_ARM_LIBS_DIR",
         "-defaultlib=phobos2-ldc,druntime-ldc",
         "-debuglib=phobos2-ldc-debug,druntime-ldc-debug"
     ];
 };
 `

 We could spice it up by allowing wildcards in the section names 
 at some point, but cross-compiling druntime/Phobos, installing 
 an appropriate cross-gcc and modifying the LDC config file is 
 really all what's needed.


Is this included in the releases? Or at least in master?
I'm trying it out with 1.2.0 and LDC doesn't seem to pick up 
anything but "default", whatever -mtriple I pass to it.
May 05 2017
parent reply kinke <noone nowhere.com> writes:
On Friday, 5 May 2017 at 16:27:10 UTC, Adrian Matoga wrote:

 Is this included in the releases? Or at least in master?
 I'm trying it out with 1.2.0 and LDC doesn't seem to pick up 
 anything but "default", whatever -mtriple I pass to it.


This is how our Win64 multilib package handles linking Win32 and 
Win64 binaries, so it's in master for at least a year or so. I 
just tested our prebuilt 1.2 Linux x64 package just to make sure 
and it works.
By adding `-v` to the LDC command line, you see the used config 
file and, in parentheses, the section being looked up (falling 
back to `default` if it's missing), e.g. (when specifying `-m32` 
or `-mtriple=i686-unknown-linux-gnu`):

binary    /home/martin/ldc2-1.2.0-linux-x86_64/bin/ldc2
version   1.2.0 (DMD v2.072.2, LLVM 4.0.0)
config    /home/martin/ldc2-1.2.0-linux-x86_64/etc/ldc2.conf 
(i686-unknown-linux-gnu)
...
May 05 2017
parent reply Adrian Matoga <dlang.spam matoga.info> writes:
On Friday, 5 May 2017 at 16:43:01 UTC, kinke wrote:

 On Friday, 5 May 2017 at 16:27:10 UTC, Adrian Matoga wrote:

 Is this included in the releases? Or at least in master?
 I'm trying it out with 1.2.0 and LDC doesn't seem to pick up 
 anything but "default", whatever -mtriple I pass to it.


 This is how our Win64 multilib package handles linking Win32 
 and Win64 binaries, so it's in master for at least a year or 
 so. I just tested our prebuilt 1.2 Linux x64 package just to 
 make sure and it works.
 By adding `-v` to the LDC command line, you see the used config 
 file and, in parentheses, the section being looked up (falling 
 back to `default` if it's missing), e.g. (when specifying 
 `-m32` or `-mtriple=i686-unknown-linux-gnu`):

 binary    /home/martin/ldc2-1.2.0-linux-x86_64/bin/ldc2
 version   1.2.0 (DMD v2.072.2, LLVM 4.0.0)
 config    /home/martin/ldc2-1.2.0-linux-x86_64/etc/ldc2.conf 
 (i686-unknown-linux-gnu)
 ...


Thank you. The "config" line is very informative. It seems that 
LDC doesn't use the specified triple verbatim, but may modify it, 
e.g. "arm-linux-gnueabi" becomes "arm--linux-gnueabi".

Taking this into account, it works and it looks like it's a part 
of the solution I'm looking for.
May 05 2017
parent reply kinke <noone nowhere.com> writes:
On Friday, 5 May 2017 at 16:56:57 UTC, Adrian Matoga wrote:

 It seems that LDC doesn't use the specified triple verbatim, 
 but may modify it, e.g. "arm-linux-gnueabi" becomes 
 "arm--linux-gnueabi".


That 2nd component of the 4-component 'triple' bothers me too. 
I've only seen the values `pc`, `unknown`, `none` and now, thanks 
to you ;), even '' for it so far.
May 05 2017
next sibling parent Adrian Matoga <dlang.spam matoga.info> writes:
On Friday, 5 May 2017 at 18:49:46 UTC, kinke wrote:

 On Friday, 5 May 2017 at 16:56:57 UTC, Adrian Matoga wrote:

 It seems that LDC doesn't use the specified triple verbatim, 
 but may modify it, e.g. "arm-linux-gnueabi" becomes 
 "arm--linux-gnueabi".


 That 2nd component of the 4-component 'triple' bothers me too. 
 I've only seen the values `pc`, `unknown`, `none` and now, 
 thanks to you ;), even '' for it so far.


It seems that any number of components is accepted as a 'triple', 
and if there's more than four, some of them are moved from the 
inside to the back of it. :)

I'd like to have a couple of targets side by side that would all 
use "arm-linux-gnueabi" (or maybe "arm-tizen-linux-gnueabi"?), 
but would differ in their sysroot paths.
I could add extra components to the triple to distinguish these 
targets and it'd work with the current LDC, but it seems a bit 
hackish to me. What do you think?
May 05 2017
prev sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Friday, 5 May 2017 at 18:49:46 UTC, kinke wrote:

 On Friday, 5 May 2017 at 16:56:57 UTC, Adrian Matoga wrote:

 It seems that LDC doesn't use the specified triple verbatim, 
 but may modify it, e.g. "arm-linux-gnueabi" becomes 
 "arm--linux-gnueabi".


 That 2nd component of the 4-component 'triple' bothers me too. 
 I've only seen the values `pc`, `unknown`, `none` and now, 
 thanks to you ;), even '' for it so far.


the second component of the triple is the vendor, so 
hardware-vendor-os/environment[-abi]. but yeah a blank '' is 
weird. Most OSX triples are $hardware-apple-$os. All the NVPTX 
triples are $hardware-nvidia-(cuda|opencl).
May 14 2017
prev sibling next sibling parent reply kinke <noone nowhere.com> writes:
On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen wrote:

 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?

 - Johan


FWIW, the MSVC packages already feature a bunch of other targets 
incl. ARM and PowerPC 
[https://github.com/ldc-developers/ldc-scripts/blob/master/ldc2-msvc/RELEASE.proj#L67].
Apr 22 2017
parent reply Johan Engelen <j j.nl> writes:
On Saturday, 22 April 2017 at 12:51:23 UTC, kinke wrote:

 On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen wrote:

 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?

 - Johan


 FWIW, the MSVC packages already feature a bunch of other 
 targets incl. ARM and PowerPC


Ah, so let's use the same list for the ldc-scripts script?
Apr 22 2017
parent kinke <noone nowhere.com> writes:
On Saturday, 22 April 2017 at 14:53:46 UTC, Johan Engelen wrote:

 On Saturday, 22 April 2017 at 12:51:23 UTC, kinke wrote:

 On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen 
 wrote:

 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?

 - Johan


 FWIW, the MSVC packages already feature a bunch of other 
 targets incl. ARM and PowerPC


 Ah, so let's use the same list for the ldc-scripts script?


I don't have a strong opinion on this (and I didn't create that 
list), but a couple of MBytes (uncompressed) seem totally worth 
it to me (at least for x86 hosts), even if mainly for advertising 
the cross-compilation capabilities and target variety via `ldc2 
--version`. With 1.3 able to generate entire static libs for all 
supported targets, it'd make some more sense I think.
Apr 22 2017
prev sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Saturday, 22 April 2017 at 10:21:05 UTC, Johan Engelen wrote:

 Would it make sense to release 1.3 with more targets enabled 
 than the host?
 X86, ARM, AArch64 ?

 - Johan


Not for 1.3, but once the compiler dcompute stuff is finished 
it'd be great to have NVPTX (and SPIRV once I get it into LLVM 
and we release with a version that has it). The increased 
visibility would be very god to have.
Apr 22 2017