www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - The D ecosystem in Debian with free-as-in-freedom DMD

reply Matthias Klumpp <mak debian.org> writes:
Hi there!
These are probably questions directed mostly at Walter and others 
shaping D's goals, but this could be of general interest for many 
people, so to the forum it goes :-)

DMD is completely free software now and we can legally distribute 
it in Debian main - yay! This is an awesome achievement and will 
make D adoption in Linux distributions much easier (in fact, Red 
Hat is working on getting good support into Fedora too already).

There are a two issues though that we will be facing in Debian 
soon, and I would like to get some opinion and maybe perspective 
on from the D community on them.

Naturally, when the reference compiler is available in Debian, we 
would compile everything with that, as it is the development 
focus and the thing many people test with.
We do, however, have quite a bit of bioinformatics and other D 
software in the archive where performance matters - so our users 
and the developers of that software (like BioD, potentially Mir, 
maybe even Vibe.d) will want the fastest performance and will ask 
us to compile the libraries with LDC or GDC.

If we do that, we will run into the D ABI trap: Libraries 
compiled with compiler X can not be used from software compiled 
with D compiler Y. There is actually no ABI stability guarantee 
even between DMD releases.
This will make integrating D a huge pain. Recompiling the 
dependency-chain of a software from source when compiling a 
package using the "right" compiler and statically adding the code 
is forbidden by distro policy. Having static libraries in the 
dependencies doesn't solve the issue. Compiling each library with 
all D compilers is highly impractical and not really feasible.
So, how should we proceed here? We could make it "DMD is the only 
thing on the highway" compiling everything with DMD with zero 
exceptions, which would leave us with only DMD-internal ABI 
breakage and bad D code performance for some libraries. We could 
also continue using LDC for everything, but that comes with it's 
own issues (I am hitting quite a bunch of LDC bugs and upstream 
projects usually test with DMD or use features which are only in 
the latest DMD).

The other issue is architecture support. In Debian we are 
strongly encouraged to support as many architectures as possible, 
to the point of having to justify why arch X is not supported.
LDC runs on at least armhf and ppc64el additionally to 
amd64/ia32, while DMD says it's specifically only for ia32/amd64.
This means we might end up compiling stuff with different D 
compilers on different architectures, or we will need to drop 
architectures and request arch-specific package removals for 
things that currently build on architectures not supported by 
DMD, which will trigger resistance.

So, in summary:
   1) Is there some perspective on D getting a defined ABI that 
works with all major D compilers?
   2) What would the D community recommend on how to deal with the 
ABI issues currently? A Linux distribution is a bunch of tightly 
integrated software, and changing one piece in an incompatible 
way (e.g. by building it with LDC instead of DMD) will have 
consequences.
   3) Will DMD support more architectures in the near future? How 
should the architecture issue be handled?

I am interested in some feedback here, since I currently can't 
see a good way to address these issues.

Also: If you want to help out Debian's D team, feel free to ping 
me or any other D team member (we are very short handed and are 
only two active people right now). See https://wiki.debian.org/D 
(caution, wiki page is very outdated, last touched in 2012)
Apr 10 2017
next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:

 Recompiling the dependency-chain of a software from source when 
 compiling a package using the "right" compiler and statically 
 adding the code is forbidden by distro policy.


This is the part that I do not understand.

Who came up with those policies and decided that they apply to D? 
Because I really don't think they should.

A lot of D code is generic. Very extremely few exceptions, it is 
a given that any time a library is used, at least some part of 
the library's code is going to be compiled not when the library 
is "built", but when programs using the library are. The notion 
of D libraries preserving ABI compatibility of built shared 
libraries is pretty much unheard of (it would have to be an 
explicit design goal), with Phobos and everything else going for 
source-compatibility only. Heck, there are packages out there 
that do not even produce useful object files (such as header-only 
C++ libraries).

As far as I can see, the main obstacle of D in Debian is the 
extreme dogmatism of the distribution's policies. D is not C 
(where these rules make much more sense), so I really don't think 
the same rules should apply. Can we treat it more like an 
interpreted language instead?
Apr 10 2017
next sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 12:40:33 UTC, Vladimir Panteleev 
wrote:

 On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:

 Recompiling the dependency-chain of a software from source 
 when compiling a package using the "right" compiler and 
 statically adding the code is forbidden by distro policy.


 This is the part that I do not understand.

 Who came up with those policies and decided that they apply to 
 D? Because I really don't think they should.


They are the result of years of experience in building complex 
systems and keeping them secure.
If you have a dependency chain "X -> Y -> Z" (-> meaning "depends 
on"), and you find a security bug in Z, you the security team 
will just need to fix the bug in Z to resolve it in the whole 
distribution.
But if the code which has this issue is compiled into all of the 
packages that depend on them, you will need to rebuild the full 
dependency chain to actually fix the security issue, which is not 
only time intensive but also a huge maintenance effort.
In this simple example it doesn't look like much, but those 
dependency chains can grow massively large and complicated, and 
the only way to keep the large software stack maintainable and 
secure is by splitting pieces cleanly.

Embedded code copies are allowed in rare events, but in these 
cases the security team needs to be aware of them.
Sometimes, the licenses also explicitly prevent embedded code 
copies.

Aside from these issues, splitting things cleanly also makes 
general package maintenance much easier, and adds flexibility for 
our users who can mix and match parts of the distribution as they 
like and combine them with their own code.

You need to see here that D is not the center of the world and we 
will need to make it work nicely with the rest of the system. The 
technical policies work for everything else, so there is nothing 
that really justifies an exception for D here (if 10% of Debian's 
code was written in D and the Debian D team was really large we 
could maybe get one, but not the way it is now).
And tbh, I think finding a good solution here is entirely 
possible.
Apr 10 2017
next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Monday, 10 April 2017 at 12:59:37 UTC, Matthias Klumpp wrote:

 Who came up with those policies and decided that they apply to 
 D? Because I really don't think they should.


 They are the result of years of experience in building complex 
 systems and keeping them secure.
 If you have a dependency chain "X -> Y -> Z" (-> meaning 
 "depends on"), and you find a security bug in Z, you the 
 security team will just need to fix the bug in Z to resolve it 
 in the whole distribution.
 But if the code which has this issue is compiled into all of 
 the packages that depend on them, you will need to rebuild the 
 full dependency chain to actually fix the security issue, which 
 is not only time intensive but also a huge maintenance effort.
 In this simple example it doesn't look like much, but those 
 dependency chains can grow massively large and complicated, and 
 the only way to keep the large software stack maintainable and 
 secure is by splitting pieces cleanly.

 Embedded code copies are allowed in rare events, but in these 
 cases the security team needs to be aware of them.
 Sometimes, the licenses also explicitly prevent embedded code 
 copies.

 Aside from these issues, splitting things cleanly also makes 
 general package maintenance much easier, and adds flexibility 
 for our users who can mix and match parts of the distribution 
 as they like and combine them with their own code.


No, I understand all of this. What I'm saying that in the case of 
D, these rules, though making sense, will just not work. You 
can't replace a piece of code in a template instantiation in a 
compiled program, shared libraries and stable ABI or not.


 You need to see here that D is not the center of the world and 
 we will need to make it work nicely with the rest of the system.


The opposite is also true: requiring a stable shared library API 
of every packaged D library is just as unreasonable. In fact, to 
make these rules useful and applicable to all D programs, you'd 
have to completely forbid templates in the library's public 
interface, which would immediately exclude Phobos for one.
Apr 10 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 13:07:22 UTC, Vladimir Panteleev 
wrote:

 On Monday, 10 April 2017 at 12:59:37 UTC, Matthias Klumpp wrote:

 Who came up with those policies and decided that they apply 
 to D? Because I really don't think they should.


 [...]
 You need to see here that D is not the center of the world and 
 we will need to make it work nicely with the rest of the 
 system.


 The opposite is also true: requiring a stable shared library 
 API of every packaged D library is just as unreasonable. In 
 fact, to make these rules useful and applicable to all D 
 programs, you'd have to completely forbid templates in the 
 library's public interface, which would immediately exclude 
 Phobos for one.


There is a really easy way to fix this: SONAMEs. Whenever you 
change something in the library breaking ABI or API, you bump 
it's SOVERSION, which will force the distribution to perform a 
transition and rebuild the dependency chain. If you give 
absolutely zero stability guarantees, you just set the SOVERSION 
equal to the project's version and trigger a transition every 
time (incredibly annoying, but, well, okay).

This has worked nicely for every language. If you don't have 
templates in your API or don't change the templates between 
releases, you can survive with one library for a long time.

This is working really great on the level of individual 
libraries, but if the whole language is ABI-unstable, the issues 
are much bigger and harder to track.

Btw, at time we are just ignore the ABI issues, and surprisingly 
nothing broke yet, indicating that ABI breakage isn't very common 
or not affecting commonly used interfaces much.
Apr 10 2017
next sibling parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 April 2017 at 15:20, Matthias Klumpp via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 10 April 2017 at 13:07:22 UTC, Vladimir Panteleev wrote:

 On Monday, 10 April 2017 at 12:59:37 UTC, Matthias Klumpp wrote:

 Who came up with those policies and decided that they apply to D?
 Because I really don't think they should.


 [...]
 You need to see here that D is not the center of the world and we will
 need to make it work nicely with the rest of the system.



 The opposite is also true: requiring a stable shared library API of every
 packaged D library is just as unreasonable. In fact, to make these rules
 useful and applicable to all D programs, you'd have to completely forbid
 templates in the library's public interface, which would immediately exclude
 Phobos for one.



 There is a really easy way to fix this: SONAMEs. Whenever you change
 something in the library breaking ABI or API, you bump it's SOVERSION, which
 will force the distribution to perform a transition and rebuild the
 dependency chain. If you give absolutely zero stability guarantees, you just
 set the SOVERSION equal to the project's version and trigger a transition
 every time (incredibly annoying, but, well, okay).


I believe GDC in Debian is done correctly on this part.  Last time
someone else looked, it seemed like LDC and DMD make use of SOVERSION,
but do so in an incorrect manner.


 This has worked nicely for every language. If you don't have templates in
 your API or don't change the templates between releases, you can survive
 with one library for a long time.

 This is working really great on the level of individual libraries, but if
 the whole language is ABI-unstable, the issues are much bigger and harder to
 track.

 Btw, at time we are just ignore the ABI issues, and surprisingly nothing
 broke yet, indicating that ABI breakage isn't very common or not affecting
 commonly used interfaces much.


Everyone should follow GDC's ABI, rather than trying to mimic DMD
calling convention. ;-)
Apr 10 2017
next sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 16:12:35 UTC, Iain Buclaw wrote:

 [...]
 Everyone should follow GDC's ABI, rather than trying to mimic 
 DMD calling convention. ;-)


GDC is working very well, and using it would actually be the 
natural choice for us as GCC is the default compiler.
However, there are a few problems with that:
  1) GDC doesn't compile a large amount of D code currently out 
there due to having an outdated standard library and runtime. 
While we would like to use it, it doesn't help if it just can't 
compile the things.

  2) GDC isn't part of the official GCC. This has quite wide 
reaching implications, most importantly other distros not 
offering it - in Fedora / Red Hat, this is the reason why GDC 
isn't available. This means fewer people test with GDC and we're 
basically compiling upstream projects with a compiler they never 
ever tested, potentially creating new issues that we can't easily 
forward upstream. This is less of an issue with LDC as LDC is 
more widely available in other distributions.
Apr 10 2017
parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 April 2017 at 19:48, Matthias Klumpp via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 10 April 2017 at 16:12:35 UTC, Iain Buclaw wrote:

 [...]
 Everyone should follow GDC's ABI, rather than trying to mimic DMD calling
 convention. ;-)



 GDC is working very well, and using it would actually be the natural choice
 for us as GCC is the default compiler.
 However, there are a few problems with that:
  1) GDC doesn't compile a large amount of D code currently out there due to
 having an outdated standard library and runtime. While we would like to use
 it, it doesn't help if it just can't compile the things.


Master sports Phobos 2.071.  Someone will have to see whether latter
versions can be built using it.



  2) GDC isn't part of the official GCC. This has quite wide reaching
 implications, most importantly other distros not offering it - in Fedora /
 Red Hat, this is the reason why GDC isn't available. This means fewer people
 test with GDC and we're basically compiling upstream projects with a
 compiler they never ever tested, potentially creating new issues that we
 can't easily forward upstream. This is less of an issue with LDC as LDC is
 more widely available in other distributions.


This is an unfortunate distribution problem, things would be different
if GCC were more like a library.  It's not like anyone is helping me
with the push.  I am ultimately the one who is doing the tens of
thousands of lines of code changes, and writing up the thousands of
lines of documentation just to make it lean and presentable.  ;-)
Apr 10 2017
next sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Monday, 10 April 2017 at 20:43:06 UTC, Iain Buclaw wrote:

 Master sports Phobos 2.071.  Someone will have to see whether 
 latter versions can be built using it.


… and some weird Frankensteinian mix of several frontend 
versions, I take it, maybe enough to build Phobos, but not 
necessarily compatible for user code? Or did you port all the 
changes since 2.068.2 back to C++?

  — David
Apr 10 2017
parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 April 2017 at 23:52, David Nadlinger via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 10 April 2017 at 20:43:06 UTC, Iain Buclaw wrote:

 Master sports Phobos 2.071.  Someone will have to see whether latter
 versions can be built using it.



 … and some weird Frankensteinian mix of several frontend versions, I take
 it, maybe enough to build Phobos, but not necessarily compatible for user
 code? Or did you port all the changes since 2.068.2 back to C++?

  — David


All the regression fixes and none of the bugs!

The current situation is that it should be link-compatible with
current upstream/stable.  Enough so that when someone has the time to
test, it should just be a case switching the sources and building the
D version.
Apr 10 2017
next sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 22:36:39 UTC, Iain Buclaw wrote:

 On 10 April 2017 at 23:52, David Nadlinger via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:

 On Monday, 10 April 2017 at 20:43:06 UTC, Iain Buclaw wrote:

 Master sports Phobos 2.071.  Someone will have to see whether 
 latter versions can be built using it.



 … and some weird Frankensteinian mix of several frontend 
 versions, I take it, maybe enough to build Phobos, but not 
 necessarily compatible for user code? Or did you port all the 
 changes since 2.068.2 back to C++?

  — David


 All the regression fixes and none of the bugs!

 The current situation is that it should be link-compatible with 
 current upstream/stable.  Enough so that when someone has the 
 time to test, it should just be a case switching the sources 
 and building the D version.


First of all, thank you for your tremendous work on GDC! Fellow 
developers and me were also pretty stunned by you maintaining a 
quite large amount of different GDC versions in parallel without 
a huge team - that's some impressive work!

What is the thing that's blocking GDCs GCC inclusion? Just 
manpower? Also, you were talking about "bugs" on several 
occasions, what's the thing with that? Is it GCC or general 
Phobos bugs? It would probably be awesome to have a summary 
blogpost or similar on the state of GDC, that could potentially 
also attract volunteers.
Anyway, all a bit off-topic :-)
Apr 11 2017
next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 11 April 2017 at 12:11:10 UTC, Matthias Klumpp wrote:

 On Monday, 10 April 2017 at 22:36:39 UTC, Iain Buclaw wrote:

 All the regression fixes and none of the bugs!

 The current situation is that it should be link-compatible 
 with current upstream/stable.  Enough so that when someone has 
 the time to test, it should just be a case switching the 
 sources and building the D version.


 First of all, thank you for your tremendous work on GDC! Fellow 
 developers and me were also pretty stunned by you maintaining a 
 quite large amount of different GDC versions in parallel 
 without a huge team - that's some impressive work!

 What is the thing that's blocking GDCs GCC inclusion? Just 
 manpower? Also, you were talking about "bugs" on several 
 occasions, what's the thing with that? Is it GCC or general 
 Phobos bugs? It would probably be awesome to have a summary 
 blogpost or similar on the state of GDC, that could potentially 
 also attract volunteers.
 Anyway, all a bit off-topic :-)


I assume the bugs would be DMDFE bugs.

Iain is giving  talk at DConf on GDC 
(http://dconf.org/2017/talks/buclaw.html) which should help.
Apr 11 2017
prev sibling parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 April 2017 at 14:11, Matthias Klumpp via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 10 April 2017 at 22:36:39 UTC, Iain Buclaw wrote:

 On 10 April 2017 at 23:52, David Nadlinger via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Monday, 10 April 2017 at 20:43:06 UTC, Iain Buclaw wrote:

 Master sports Phobos 2.071.  Someone will have to see whether latter
 versions can be built using it.




 … and some weird Frankensteinian mix of several frontend versions, I take
 it, maybe enough to build Phobos, but not necessarily compatible for user
 code? Or did you port all the changes since 2.068.2 back to C++?

  — David



 All the regression fixes and none of the bugs!

 The current situation is that it should be link-compatible with current
 upstream/stable.  Enough so that when someone has the time to test, it
 should just be a case switching the sources and building the D version.



 First of all, thank you for your tremendous work on GDC! Fellow developers
 and me were also pretty stunned by you maintaining a quite large amount of
 different GDC versions in parallel without a huge team - that's some
 impressive work!


It was worse when we were using `#if D_GCC_VER == 33' macros to
support all versions in the same branch. ;-)

Having a really good CI helps too for when the periodic merges are done.



 What is the thing that's blocking GDCs GCC inclusion? Just manpower? Also,
 you were talking about "bugs" on several occasions, what's the thing with
 that? Is it GCC or general Phobos bugs? It would probably be awesome to have
 a summary blogpost or similar on the state of GDC, that could potentially
 also attract volunteers.
 Anyway, all a bit off-topic :-)


Manpower has got something to do with it when you have two fast ships
sailing and you have one foot in each.

I think I only mentioned bugs once here, and was talking strictly
about the D front-end, which deals with the parsing and semantic
analysis of D sources.  Every release there are listed new features,
regressions fixed, etc.  The regressions fixed have a higher precedent
for me at least.
Apr 11 2017
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Monday, 10 April 2017 at 22:36:39 UTC, Iain Buclaw wrote:

 All the regression fixes and none of the bugs!


That's an interesting approach (spoken with British language 
sensibilities).

Is anybody using GDC on a big "modern" D2 codebase 
(metaprogramming-heavy, …) right now? From my experience with 
Thrift, Weka, and so on, even regression fixes have a habit of 
breaking that sort of code in weird and wonderful ways. And 
that's not even mentioning other improvements/fixes/additions 
people might be relying on.

Right now, the frontend version serves as a convenient aid for 
users to navigate those issues (apart, of course, from us trying 
to improve things by making semantic analysis more deterministic, 
etc). If something works on DMD 2.072.2, then users can also 
expect it to work on the corresponding LDC version as well. I'm 
not sure you are helping anybody by introducing another set of 
weird in-between versions.

  — David
Apr 11 2017
parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 April 2017 at 20:13, David Nadlinger via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 10 April 2017 at 22:36:39 UTC, Iain Buclaw wrote:

 All the regression fixes and none of the bugs!



 That's an interesting approach (spoken with British language sensibilities).

 Is anybody using GDC on a big "modern" D2 codebase (metaprogramming-heavy,
 …) right now? From my experience with Thrift, Weka, and so on, even
 regression fixes have a habit of breaking that sort of code in weird and
 wonderful ways. And that's not even mentioning other
 improvements/fixes/additions people might be relying on.

 Right now, the frontend version serves as a convenient aid for users to
 navigate those issues (apart, of course, from us trying to improve things by
 making semantic analysis more deterministic, etc). If something works on DMD
 2.072.2, then users can also expect it to work on the corresponding LDC
 version as well. I'm not sure you are helping anybody by introducing another
 set of weird in-between versions.

  — David


And DMD didn't help by switching codebase to D without first ironing
out all implementation regressions and interoperability.  Every
release between 2.069 and 2.074 is a broken release as far I'm
concerned.  The most recent is getting closer, but there are likely
still many things missing that need to be exposed to C++ (mostly
::create functions).
Apr 11 2017
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2017 1:43 PM, Iain Buclaw via Digitalmars-d wrote:

 This is an unfortunate distribution problem, things would be different
 if GCC were more like a library.  It's not like anyone is helping me
 with the push.  I am ultimately the one who is doing the tens of
 thousands of lines of code changes, and writing up the thousands of
 lines of documentation just to make it lean and presentable.  ;-)


Iain, thank you for doing this herculean task all by yourself. I know that you 
need help - anyone want to step up?

I hope that making D fully open source will have a side effect of motivating 
people to help you get gdc into the main gcc distribution.
Apr 10 2017
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On Monday, 10 April 2017 at 16:12:35 UTC, Iain Buclaw wrote:

 Last time someone else looked, it seemed like LDC and DMD make 
 use of SOVERSION, but do so in an incorrect manner.


You know what exactly is the problem? Any suggestion what to use 
instead?

It's currently using libphobos2.so.0.74, where using major 
version 0 to mean unstable might be a misapplication of semver 
(http://semver.org/#spec-item-4) for SONAME.
Our point releases might also contain small ABI 
incompatibilities, so they aren't really eligible as patch 
version.

https://github.com/dlang/phobos/blob/c648097621db225aa59e07d775f54dbaa847c68a/posix.mak#L151
Apr 12 2017
next sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 12 Apr 2017 07:42:42 +0000
schrieb Martin Nowak <code dawg.eu>:


 On Monday, 10 April 2017 at 16:12:35 UTC, Iain Buclaw wrote:

 Last time someone else looked, it seemed like LDC and DMD make 
 use of SOVERSION, but do so in an incorrect manner.  


 
 You know what exactly is the problem? Any suggestion what to use 
 instead?
 
 It's currently using libphobos2.so.0.74, where using major 
 version 0 to mean unstable might be a misapplication of semver 
 (http://semver.org/#spec-item-4) for SONAME.


I still haven't found some definitive documentation about this, but it
seems linux shared library working essentially works like this:

There's a major and a minor number. There's sometimes a patch version,
but there's no conclusive documentation. Reading the ldconfig source
code you can have as many version levels as you want [3] (though
essentially this is treated like one minor version, it only affects
comparing minor versions). But there are different ldconfig
implementations (linux, bsd) so I don't know if multi-level
minor versions are portable.

* Major version reflects ABI level. A new major version can break ABI
  or add new stuff to ABI.
* Minor versions can only extend the ABI of the major version but
  should not break any ABI.
* Micro / Patch version is mostly unused in ldconfig. It only affects
  comparing minor versions, i.e. when you have libphobos.so.74.0.1 and
  libphobos.so.74.0.2 ldconfig will link
  libphobos.so.74=>libphobos.so.74.0.2

Filename format: lib[name].so.[major].[minor][.patch]
Soname format: always lib[name].so.[major]

It is possible to install and use multiple major versions. Every major
version will always use the last installed minor version. (The
distribution will manage symlinks for libfoo.[major] to
libfoo.[major].[minor] for the largest minor version). Additionally a
libfoo.so is installed (for linking / development only, might even be
in -dev packages) to point to the latest libfoo.[major] symlink.

The libfoo.[major] symlink is used when linking with -lfoo. The
dependency encoded in the executable will use the soname though, so it
will encode libfoo.[major]. If you install a new major library version,
all existing executables will continue to use the major version
hardcoded in the executable. If you update a major version to a new
minor version, all executables using the major version soname will use
the new minor version.

This means:
* Increase major every time you break ABI
* Increase some minor level every time you only extend ABI

So DMD should not keep the major version fixed as 0 (every time you
update libphobos you break all existing binaries, as you break the ABI
of libphobos.so.0).

In GDC we use libtool which encodes like this [1][2]:
libgphobos.so.[major].0.[release] e.g. libgphobos.74.0.2 This is not
100% safe if a minor release breaks ABI though.  

BTW: Interestingly even with these complicated rules you can end up in
situation where versioning does not work: If you link application APP
against libfoo.so on a system with libfoo.so.1.2 the soname will only
encode libfoo.so.1. Now ship the library to a system with libfoo.so.1.1
and you may have missing symbols due to extended ABI in libfoo.so.1.2.
TLDR: Downgrading libraries is not safe with this versioning approach.



[1] https://autotools.io/libtool/version.html
[2]
https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html

ldconfig is responsible for maintaining the symlinks, here's how it
determines which version of a library is newer:

[3]
https://github.com/lattera/glibc/blob/master/elf/ldconfig.c#L939
https://github.com/lattera/glibc/blob/master/elf/dl-cache.c#L138
Apr 12 2017
prev sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 12 Apr 2017 07:42:42 +0000
schrieb Martin Nowak <code dawg.eu>:


 Our point releases might also contain small ABI 
 incompatibilities, so they aren't really eligible as patch 
 version.


I've actually been hit by this in one point release on Gentoo,
where I used dynamic linking for Dlang as soon as it was
possible. Programs would suddenly complain about a mismatch
between the size of some compiler generated item after a point
release update of dmd. Just posting this to give the
discussion a bit more substance. :)

-- 
Marco
May 02 2017
prev sibling next sibling parent reply Johan Engelen <j j.nl> writes:
On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:

 
 Btw, at time we are just ignore the ABI issues, and 
 surprisingly nothing broke yet, indicating that ABI breakage 
 isn't very common or not affecting commonly used interfaces 
 much.


One big ABI change was in 2.071: 
https://issues.dlang.org/show_bug.cgi?id=15644. And it involved 
interfaces. ;-)
Nothing broke because of that?

cheers,
   Johan
Apr 10 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 16:58:05 UTC, Johan Engelen wrote:

 On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:

 
 Btw, at time we are just ignore the ABI issues, and 
 surprisingly nothing broke yet, indicating that ABI breakage 
 isn't very common or not affecting commonly used interfaces 
 much.


 One big ABI change was in 2.071: 
 https://issues.dlang.org/show_bug.cgi?id=15644. And it involved 
 interfaces. ;-)
 Nothing broke because of that?


I am reading release notes, so we rebuilt dependencies of LDC - I 
have no idea about GDC-depending D code though. But since no bugs 
were reported, I assume no issues are present :-)
Apr 10 2017
next sibling parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 April 2017 at 19:50, Matthias Klumpp via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 10 April 2017 at 16:58:05 UTC, Johan Engelen wrote:

 On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:

 Btw, at time we are just ignore the ABI issues, and surprisingly nothing
 broke yet, indicating that ABI breakage isn't very common or not affecting
 commonly used interfaces much.



 One big ABI change was in 2.071:
 https://issues.dlang.org/show_bug.cgi?id=15644. And it involved interfaces.
 ;-)
 Nothing broke because of that?



 I am reading release notes, so we rebuilt dependencies of LDC - I have no
 idea about GDC-depending D code though. But since no bugs were reported, I
 assume no issues are present :-)


That change is in gdc-7.x - so still in Debian unstable.
Apr 10 2017
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Monday, 10 April 2017 at 17:50:08 UTC, Matthias Klumpp wrote:

 I am reading release notes, so we rebuilt dependencies of LDC -


(I assume you mean reverse dependencies.)


 […] But since no bugs were reported, I assume no issues are 
 present :-)


So do we need to put a reminder about the ABI being unstable into 
set of every release notes to make sure we won't get angry bug 
reports once users actually build their own D code against your 
packages? ;)

  — David
Apr 10 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 22:15:53 UTC, David Nadlinger wrote:

 On Monday, 10 April 2017 at 17:50:08 UTC, Matthias Klumpp wrote:

 I am reading release notes, so we rebuilt dependencies of LDC -


 (I assume you mean reverse dependencies.)


 […] But since no bugs were reported, I assume no issues are 
 present :-)


 So do we need to put a reminder about the ABI being unstable 
 into set of every release notes to make sure we won't get angry 
 bug reports once users actually build their own D code against 
 your packages? ;)


Nah, there are several options here, one would simply be to tell 
people not to use the distro packages with anything but the 
default D compiler used in the respective Debian release.
Go apparently tells people not to use Debian-shipped go code in 
their own projects at all.
Apr 11 2017
next sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 12:03 +0000, Matthias Klumpp via Digitalmars-d
wrote:

=20


[=E2=80=A6]

 Nah, there are several options here, one would simply be to tell=C2=A0
 people not to use the distro packages with anything but the=C2=A0
 default D compiler used in the respective Debian release.
 Go apparently tells people not to use Debian-shipped go code in=C2=A0
 their own projects at all.


The vendoring systems that Go folk have invented are effectively
mandatory for projects that want reproducible builds, and using
platform specific code is not feasible. It suprises me that Debian and
Fedora are going flat out trying to package Go stuff.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Apr 11 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 12:42:13 UTC, Russel Winder wrote:

 On Tue, 2017-04-11 at 12:03 +0000, Matthias Klumpp via 
 Digitalmars-d wrote:

 


 […]

 Nah, there are several options here, one would simply be to 
 tell
 people not to use the distro packages with anything but the
 default D compiler used in the respective Debian release.
 Go apparently tells people not to use Debian-shipped go code in
 their own projects at all.


 The vendoring systems that Go folk have invented are 
 effectively mandatory for projects that want reproducible 
 builds, and using platform specific code is not feasible. It 
 suprises me that Debian and Fedora are going flat out trying to 
 package Go stuff.


That's false. Debian is leading the effort on reproducible builds 
that many other projects (including Fedora) have joined, and a 
large chunk of packages is already reproducible[1].
It's actually quite the opposite: Build systems downloading 
random stuff from the internet make the system more likely to 
produce different build results.

But in any case, the primary use for Debian packages is to be 
used by the distribution.

[1]: 
https://tests.reproducible-builds.org/debian/reproducible.html
Apr 11 2017
parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 12:48 +0000, Matthias Klumpp via Digitalmars-d
wrote:
[=E2=80=A6]

 That's false. Debian is leading the effort on reproducible builds=C2=A0
 that many other projects (including Fedora) have joined, and a=C2=A0
 large chunk of packages is already reproducible[1].
 It's actually quite the opposite: Build systems downloading=C2=A0
 random stuff from the internet make the system more likely to=C2=A0
 produce different build results.


You are closer the that community that I am and so have better
knowledge there. My comment was really reflecting the cant about the OS
packages.


 But in any case, the primary use for Debian packages is to be=C2=A0
 used by the distribution.


Where Debian and Fedora provide applications written in Go they will be
statically linked. So is the use case that everything used to create
the application has to be packaged?


 [1]:=C2=A0
 https://tests.reproducible-builds.org/debian/reproducible.html


--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Apr 11 2017
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Tuesday, 11 April 2017 at 12:03:27 UTC, Matthias Klumpp wrote:

 On Monday, 10 April 2017 at 22:15:53 UTC, David Nadlinger wrote:

 So do we need to put a reminder about the ABI being unstable 
 into set of every release notes to make sure we won't get 
 angry bug reports once users actually build their own D code 
 against your packages? ;)


 Nah, there are several options here, one would simply be to 
 tell people not to use the distro packages with anything but 
 the default D compiler used in the respective Debian release.


So as long as one sticks to packages in the official apt repos, 
all the libraries are guaranteed to be built with the distributed 
compiler as well?

When you mentioned that you'd read the release notes regarding 
the ABI change, I got the impression that you had to manually 
rebuild the world for that to happen – hence my tongue-in-cheek 
remark about reminding you to do this in the release notes.

Otherwise, you might get lucky as far as the distributed 
applications are concerned (i.e. happen not to hit any ABI 
issues), but users might still be hosed when it comes to their 
own code.

  — David
Apr 11 2017
parent Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 18:14:40 UTC, David Nadlinger wrote:

 On Tuesday, 11 April 2017 at 12:03:27 UTC, Matthias Klumpp 
 wrote:

 On Monday, 10 April 2017 at 22:15:53 UTC, David Nadlinger 
 wrote:

 So do we need to put a reminder about the ABI being unstable 
 into set of every release notes to make sure we won't get 
 angry bug reports once users actually build their own D code 
 against your packages? ;)


 Nah, there are several options here, one would simply be to 
 tell people not to use the distro packages with anything but 
 the default D compiler used in the respective Debian release.


 So as long as one sticks to packages in the official apt repos, 
 all the libraries are guaranteed to be built with the 
 distributed compiler as well?


Yes. Unfortunately there will be three of them which aren't 
compatible with each other, so we will kind of have to settle 
with one as default.


 When you mentioned that you'd read the release notes regarding 
 the ABI change, I got the impression that you had to manually 
 rebuild the world for that to happen – hence my tongue-in-cheek 
 remark about reminding you to do this in the release notes.


Well, it's a matter of telling the build admins or making a 
proper transition package (doesn't exit yet for D), but yeah, 
technically we'd need to rebuild all D stuff on ABI changes.
Apr 11 2017
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:

 This has worked nicely for every language. If you don't have 
 templates in your API or don't change the templates between 
 releases, you can survive with one library for a long time.


But the vast majority of D libraries _do_ have templates 
(starting with Phobos).  How should this situation be dealt with?

How does Debian deal with, e.g., fixes to the templated code in 
Boost, which impact on other packages built using those 
header-only libraries?
Apr 10 2017
parent Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 22:26:46 UTC, Joseph Rushton Wakeling 
wrote:

 On Monday, 10 April 2017 at 13:20:00 UTC, Matthias Klumpp wrote:

 This has worked nicely for every language. If you don't have 
 templates in your API or don't change the templates between 
 releases, you can survive with one library for a long time.


 But the vast majority of D libraries _do_ have templates 
 (starting with Phobos).  How should this situation be dealt 
 with?

 How does Debian deal with, e.g., fixes to the templated code in 
 Boost, which impact on other packages built using those 
 header-only libraries?


Boost's soversion is changed on every release, and the version is 
included in it's -dev package as well. That's why we have 
libboost1.62-dev: 
https://packages.debian.org/de/sid/libboost1.62-dev (and possibly 
more).
There is also a boost-defaults package setting the current 
default Boos version for packages to depend on.
If a new Boost comes out, it's soversion and -dev package name 
changes, triggering a package transition and subsequently a full 
rebuild of all stuff depending on Boost.

Doing something like this with D libraries would obviously be 
possible as well.
Apr 11 2017
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2017 5:59 AM, Matthias Klumpp wrote:

 You need to see here that D is not the center of the world and we will need to
 make it work nicely with the rest of the system. The technical policies work
for
 everything else, so there is nothing that really justifies an exception for D
 here (if 10% of Debian's code was written in D and the Debian D team was really
 large we could maybe get one, but not the way it is now).
 And tbh, I think finding a good solution here is entirely possible.


I think it is possible, too, and thank you for your efforts helping us do this.

The ABI differences between the compilers is unfortunate, and is largely the 
result of historical accident.

The first problem is that my idea originally was for D to have its own function 
calling convention, which would free us to innovate to have a more efficient 
calling convention than the C one. This hasn't panned out in practice, and 
ldc/gdc decided to sensibly stick with the C ABI. At some point, we should just 
crowbar dmd to generate the C ABI, but this has its own problems - it'll break 
code that uses the inline assembler. No obvious solution there.

The next problem is that dmd occasionally changes the interface to the D 
runtime. Or more accurately, with about every release. This has not been an 
issue historically for us, as the two have always been a matched set. I'm a lot 
less sure how to deal with this. I also do not know how the gdc/lds druntime 
interfaces differ.
Apr 10 2017
next sibling parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 April 2017 at 01:27, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 The next problem is that dmd occasionally changes the interface to the D
 runtime. Or more accurately, with about every release. This has not been an
 issue historically for us, as the two have always been a matched set. I'm a
 lot less sure how to deal with this. I also do not know how the gdc/lds
 druntime interfaces differ.



If you are referring to things such as _d_throw, _d_throwc,
_d_throwdwarf, others... In most cases, GDC has always done things in
the way that works for it, and as a result has almost never changed
ABI of library functions that it is tied to.  So we just have the one
_d_throw function for throwing dwarf EH, while DMD has gone through
several iterations over the years before finding themselves doing
things the same way as GDC - by which point you've given the function
a different name.  ;-)
Apr 10 2017
prev sibling next sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Monday, 10 April 2017 at 23:27:35 UTC, Walter Bright wrote:

 The next problem is that dmd occasionally changes the interface 
 to the D runtime. […] I also do not know how the gdc/lds 
 druntime
 interfaces differ.


Just to make this very clear to everybody reading this thread: 
It's not even just that, but also the fact that we guarantee 
API-, but not ABI-stability for Phobos. Every time we continue to 
improve the pure/nothrow/ nogc situation by marking up some more 
code, we are breaking the ABI, because the mangled names of the 
involved symbols change. The ongoing work on `scope` also breaks 
the ABI when enabled.

Yes, there are also some differences between the DMD/GDC/LDC 
runtime interfaces, but this is a comparatively very minor issue 
that doesn't make sense worrying too much about as long as ABI 
compatibility between releases isn't going to happen for other 
reasons such as the above. [1]

  — David


[1] In fact, it looks like – for example with DMD moving to 
libunwind-based EH as well – the issue is slowly resolving itself 
anyway and at some point we'll merely have to sit down for a week 
and iron out the last few kinks.
Apr 10 2017
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2017 4:43 PM, David Nadlinger wrote:

 [1] In fact, it looks like – for example with DMD moving to libunwind-based
EH
 as well – the issue is slowly resolving itself anyway and at some point we'll
 merely have to sit down for a week and iron out the last few kinks.


dmd is not moving to a libunwind-based EH, it already has moved to it!

But as an example of ABI change, the latest EH  nogc proposal changes the
layout 
of the Throwable object, and adds some code to the _d_throwdwarf() function.

https://github.com/dlang/druntime/pull/1804
Apr 10 2017
parent reply Jacob Carlborg <doob me.com> writes:
On 2017-04-11 02:20, Walter Bright wrote:

 On 4/10/2017 4:43 PM, David Nadlinger wrote:

 [1] In fact, it looks like – for example with DMD moving to
 libunwind-based EH
 as well – the issue is slowly resolving itself anyway and at some
 point we'll
 merely have to sit down for a week and iron out the last few kinks.


 dmd is not moving to a libunwind-based EH, it already has moved to it!

 But as an example of ABI change, the latest EH  nogc proposal changes
 the layout of the Throwable object, and adds some code to the
 _d_throwdwarf() function.

 https://github.com/dlang/druntime/pull/1804


It is possible to solve. For example, in Objective-C they have a 
non-fragile ABI where you can freely add instance variables to base 
classes and the offset will be adjusted automatically at load time for 
the subclasses [1]. The methods are already handled with dynamic message 
dispatch.

[1] 
http://www.sealiesoftware.com/blog/archive/2009/01/27/objc_explain_Non-fragile_ivars.html

-- 
/Jacob Carlborg
Apr 11 2017
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/11/2017 6:46 AM, Jacob Carlborg wrote:

 http://www.sealiesoftware.com/blog/archive/2009/01/27/objc_explain_Non-fragile_ivars.html


Looks like it solves it by adding another layer of indirection:

"the performance overhead of the extra ivar offset variable is small."
Apr 11 2017
parent Jacob Carlborg <doob me.com> writes:
On 2017-04-11 19:34, Walter Bright wrote:


 Looks like it solves it by adding another layer of indirection:

 "the performance overhead of the extra ivar offset variable is small."


Yes, there are always tradeoffs. But as you've said, exceptions are 
already slow ;)

-- 
/Jacob Carlborg
Apr 11 2017
prev sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 23:43:04 UTC, David Nadlinger wrote:

 On Monday, 10 April 2017 at 23:27:35 UTC, Walter Bright wrote:

 The next problem is that dmd occasionally changes the 
 interface to the D runtime. […] I also do not know how the 
 gdc/lds druntime
 interfaces differ.


 Just to make this very clear to everybody reading this thread: 
 It's not even just that, but also the fact that we guarantee 
 API-, but not ABI-stability for Phobos. Every time we continue 
 to improve the pure/nothrow/ nogc situation by marking up some 
 more code, we are breaking the ABI, because the mangled names 
 of the involved symbols change. The ongoing work on `scope` 
 also breaks the ABI when enabled.


If you could change the SOVERSION with every one of these 
changes, or simply just tie it to the respective Phobos release, 
distributions would automatically do the right thing and compile 
all D code using Phobos against the new version.
This might give Phobos a large soversion or and ugly one like 
"2.074", but it would work.
(LDC's Phobos and GDC's Phobos already have a soversion set in 
Debian...)
Apr 11 2017
parent reply David Nadlinger <code klickverbot.at> writes:
On Tuesday, 11 April 2017 at 12:38:01 UTC, Matthias Klumpp wrote:

 If you could change the SOVERSION with every one of these 
 changes, or simply just tie it to the respective Phobos 
 release, distributions would automatically do the right thing 
 and compile all D code using Phobos against the new version.


As you mention, this is already done in LDC; not just the Debian 
packages, but also upstream. The soname will always be 
`libphobos2-ldc.so.74` or what have you.

(Thinking about it, it should probably include the patch version 
as well, although we haven't had a situation so far where we 
would have wanted to release multiple LDC versions for 
ABI-incompatible patch releases.)

  — David
Apr 11 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 15:31:46 UTC, David Nadlinger wrote:

 On Tuesday, 11 April 2017 at 12:38:01 UTC, Matthias Klumpp 
 wrote:

 If you could change the SOVERSION with every one of these 
 changes, or simply just tie it to the respective Phobos 
 release, distributions would automatically do the right thing 
 and compile all D code using Phobos against the new version.


 As you mention, this is already done in LDC; not just the 
 Debian packages, but also upstream. The soname will always be 
 `libphobos2-ldc.so.74` or what have you.

 (Thinking about it, it should probably include the patch 
 version as well, although we haven't had a situation so far 
 where we would have wanted to release multiple LDC versions for 
 ABI-incompatible patch releases.)


Phobos is versioned properly in both GDC and LDC, and as long as 
that continues, no problems exist at all :-)
https://packages.debian.org/search?suite=stretch&keywords=phobos

This would - as said - also work for other D shared libraries, 
unless the D ABI in general breaks or someone tries to build a 
program with GDC and uses a library that was built with LDC or 
DMD (or any other possible compiler combination).
Apr 11 2017
parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 April 2017 at 17:48, Matthias Klumpp via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Tuesday, 11 April 2017 at 15:31:46 UTC, David Nadlinger wrote:

 On Tuesday, 11 April 2017 at 12:38:01 UTC, Matthias Klumpp wrote:

 If you could change the SOVERSION with every one of these changes, or
 simply just tie it to the respective Phobos release, distributions would
 automatically do the right thing and compile all D code using Phobos against
 the new version.



 As you mention, this is already done in LDC; not just the Debian packages,
 but also upstream. The soname will always be `libphobos2-ldc.so.74` or what
 have you.

 (Thinking about it, it should probably include the patch version as well,
 although we haven't had a situation so far where we would have wanted to
 release multiple LDC versions for ABI-incompatible patch releases.)



 Phobos is versioned properly in both GDC and LDC, and as long as that
 continues, no problems exist at all :-)
 https://packages.debian.org/search?suite=stretch&keywords=phobos

 This would - as said - also work for other D shared libraries, unless the D
 ABI in general breaks or someone tries to build a program with GDC and uses
 a library that was built with LDC or DMD (or any other possible compiler
 combination).


I thought I might try a "will it blend" experiment:

$ cat foo.d
module foo;

int times2(int a)
{
   return a * 2;
}

void throwIt(Throwable t)
{
    throw t;
}

$ cat test.d
import foo;

void main()
{
    import std.stdio;
    writeln("2 x 2 = ", times2(2));

    Throwable t = new Exception("Die!");
    try
    {
        throwIt(t);
    }
    catch (Exception e)
    {
        writeln("Caught Exception for: ", e.msg);
    }
}


1. Naive:

$ dmd -g -shared foo.d
$ ldd foo.so
    linux-vdso.so.1 =>  (0x00007ffc1a293000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f0f78bed000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0f789e5000)
    libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f0f787ce000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0f78405000)
    /lib64/ld-linux-x86-64.so.2 (0x0000557aa8c5f000)
$ gdc -g test.d foo.so -shared-libphobos
$ ldd a.out
    linux-vdso.so.1 =>  (0x00007fff020b9000)
    foo.so => ./foo.so (0x00007f262d66d000)
    libgphobos.so.71 => //usr/lib64/libgphobos.so.71 (0x00007f262cfd7000)
    libgdruntime.so.71 => /usr/lib64/libgdruntime.so.71 (0x00007f262ccde000)
    libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f262cac8000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f262c6d6000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f262c3cc000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f262c1af000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f262bfab000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f262bda2000)
    /lib64/ld-linux-x86-64.so.2 (0x0000564d0bc19000)
$ ./a.out
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7bc01ae in gc_disable () from ./foo.so

Ouch, DMD's statically linked libphobos has overridden GDC's gc_disable().


2. Mixing shared libs.

$ dmd -g -defaultlib=phobos2 -shared foo.d
$ ldd foo.so
    linux-vdso.so.1 =>  (0x00007ffc84dfb000)
--> libphobos2.so.0.74 => /usr/lib/x86_64-linux-gnu/libphobos2.so.0.74
(0x00007f7574f53000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7574b8a000)
--> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7574985000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f7574768000)
--> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f757445f000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7574256000)
    libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f7574040000)
    /lib64/ld-linux-x86-64.so.2 (0x000055c715c3e000)
$ gdc -g test.d foo.so -shared-libphobos
$ ldd a.out
    linux-vdso.so.1 =>  (0x00007ffd9e3eb000)
    foo.so => ./foo.so (0x00007fd09d9cc000)
    libgphobos.so.71 => /usr/lib64/libgphobos.so.71 (0x00007fd09d336000)
    libgdruntime.so.71 => /usr/lib64/libgdruntime.so.71 (0x00007fd09d03d000)
    libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007fd09ce27000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd09ca35000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd09c72b000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fd09c50e000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd09c30a000)
--> libphobos2.so.0.74 => /usr/lib/x86_64-linux-gnu/libphobos2.so.0.74
(0x00007fd09bae2000)
    /lib64/ld-linux-x86-64.so.2 (0x000056135abd3000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd09b8da000)
$ ./a.out
Fatal Error while loading '/usr/lib/x86_64-linux-gnu/libphobos2.so.0.74':
    The module 'std.array' is already defined in '/usr/lib64/libgphobos.so.71'.

This seems bad, but it's actually a good sign.  GDC's copy of
rt.sections.checkModuleCollisions() has loaded DMD's shared libphobos
and has correctly determined that in there exists a module that shares
the same name.


3. Telling DMD not to link libphobos

$ dmd -g -defaultlib= -shared foo.d
$ ldd foo.so
    linux-vdso.so.1 =>  (0x00007ffdc2155000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f44ede1d000)
    /lib64/ld-linux-x86-64.so.2 (0x000055e14d836000)
$ gdc -g test.d foo.so -shared-libphobos
foo.so: undefined reference to `_d_throwdwarf'
collect2: error: ld returned 1 exit status

This is expected, GDC's libgdruntime doesn't have this.  So I added it
as a stub:

extern(C) void _d_throwdwarf(Throwable o)
{
   throw o;
}


$ gdc -g test.d foo.so -shared-libphobos
$ ldd a.out
    linux-vdso.so.1 =>  (0x00007fffb05ee000)
    foo.so => ./foo.so (0x00007fe818644000)
    libgphobos.so.71 => /usr/lib64/libgphobos.so.71 (0x00007fe817fae000)
    libgdruntime.so.71 => /usr/lib64/libgdruntime.so.71 (0x00007fe817cb5000)
    libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007fe817a9f000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe8176ad000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe8173a3000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fe817186000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe816f82000)
    /lib64/ld-linux-x86-64.so.2 (0x00005644b677f000)
$ ./a.out
2 x 2 = 4
Caught Exception for: Die!


Success!  There may be some hope, but you currently need a
compatibility library for linking DMD compiled objects with GDC
programs.  And you must cut out Phobos out of the third party D
library so that the library for the compiler building the application
holds the one and only reference to symbols.
Apr 11 2017
prev sibling parent Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 23:27:35 UTC, Walter Bright wrote:

 On 4/10/2017 5:59 AM, Matthias Klumpp wrote:

 You need to see here that D is not the center of the world and 
 we will need to
 make it work nicely with the rest of the system. The technical 
 policies work for
 everything else, so there is nothing that really justifies an 
 exception for D
 here (if 10% of Debian's code was written in D and the Debian 
 D team was really
 large we could maybe get one, but not the way it is now).
 And tbh, I think finding a good solution here is entirely 
 possible.


 I think it is possible, too, and thank you for your efforts 
 helping us do this.

 The ABI differences between the compilers is unfortunate, and 
 is largely the result of historical accident.

 The first problem is that my idea originally was for D to have 
 its own function calling convention, which would free us to 
 innovate to have a more efficient calling convention than the C 
 one. This hasn't panned out in practice, and ldc/gdc decided to 
 sensibly stick with the C ABI. At some point, we should just 
 crowbar dmd to generate the C ABI, but this has its own 
 problems - it'll break code that uses the inline assembler. No 
 obvious solution there.

 The next problem is that dmd occasionally changes the interface 
 to the D runtime. Or more accurately, with about every release. 
 This has not been an issue historically for us, as the two have 
 always been a matched set. I'm a lot less sure how to deal with 
 this. I also do not know how the gdc/lds druntime interfaces 
 differ.


Thank you very much for this context! It's really good to know 
why things are the way they are to properly understand the 
problem (I am no compiler developer and ABIs are not my expertise 
- as outsider to this I see pages like 
https://dlang.org/spec/abi.html though and wonder whether the 
incompatibilities are seen as an issue in the D community and 
whether there is a chance to address them before putting work 
into setting up infrastructure to rebuild the world on compiler 
updates).
Apr 11 2017
prev sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 12:40:33 UTC, Vladimir Panteleev 
wrote:

 [...]
 Can we treat it more like an interpreted language instead?


An interpreted language would interpret the code on the target 
system at runtime. This is not what D does, so we can't really 
treat it like we treat Python (where it is possible no neatly 
separate Python modules into separate packages - transitions 
triggering rebuild cascades only exist when we jump to the next 
major CPython version, which is something distributions are well 
prepared for. Transitions are only an issue if they happen 
constantly).
At time, D is treated like C++, since it has much of the same 
challenges and we know how to deal with C++ - additionally to 
C++, D unfortunately though also has the unique issues outlined 
above, which complicate things.

I also want to stress that having a single C++ library like Boost 
compiled into stuff and rolling dependency transitions when its 
API/ABI changes with a major release is less of a problem than 
having the entire language give zero stability and 
interoperability guarantees on anything that is compiled with it.
Apr 10 2017
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2017 6:08 AM, Matthias Klumpp wrote:

 I also want to stress that having a single C++ library like Boost compiled into
 stuff and rolling dependency transitions when its API/ABI changes with a major
 release is less of a problem than having the entire language give zero
stability
 and interoperability guarantees on anything that is compiled with it.


How is the g++/clang++ issue handled?
Apr 10 2017
next sibling parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 April 2017 at 01:33, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 4/10/2017 6:08 AM, Matthias Klumpp wrote:

 I also want to stress that having a single C++ library like Boost compiled
 into
 stuff and rolling dependency transitions when its API/ABI changes with a
 major
 release is less of a problem than having the entire language give zero
 stability
 and interoperability guarantees on anything that is compiled with it.



 How is the g++/clang++ issue handled?


I suspect that clang++ keeps ABI compatibility with g++ for both
mangling and calling convention, except when functions are not
callable from outside the compilation unit (static functions,
templates).

Not sure about libstdc++, I suspect they are either compatible for
certain things.
Apr 10 2017
prev sibling parent Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 23:33:17 UTC, Walter Bright wrote:

 On 4/10/2017 6:08 AM, Matthias Klumpp wrote:

 I also want to stress that having a single C++ library like 
 Boost compiled into
 stuff and rolling dependency transitions when its API/ABI 
 changes with a major
 release is less of a problem than having the entire language 
 give zero stability
 and interoperability guarantees on anything that is compiled 
 with it.


 How is the g++/clang++ issue handled?


The C ABI is 100% compatible, the C++ ABI is "mostly" compatible, 
there is some deliberate breakage from the Clang guys though.
The issue isn't actually handled in Debian as all our code is 
always compiled with GCC, I am not aware of anything defaulting 
to Clang (although it might exist, but definitely not for library 
packages).

With only one dominant compiler, things are way easier ^^
Apr 11 2017
prev sibling next sibling parent reply qznc <qznc web.de> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:

 So, in summary:
   1) Is there some perspective on D getting a defined ABI that 
 works with all major D compilers?
   2) What would the D community recommend on how to deal with 
 the ABI issues currently? A Linux distribution is a bunch of 
 tightly integrated software, and changing one piece in an 
 incompatible way (e.g. by building it with LDC instead of DMD) 
 will have consequences.
   3) Will DMD support more architectures in the near future? 
 How should the architecture issue be handled?

 I am interested in some feedback here, since I currently can't 
 see a good way to address these issues.


How do Debian and C++ go along? There is no ABI compatibility 
between GCC and Clang afaik.

I general, I would favor to build stuff with LDC instead of DMD, 
because of performance.
Apr 10 2017
parent Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 12:59:58 UTC, qznc wrote:

 [...]
 How do Debian and C++ go along? There is no ABI compatibility 
 between GCC and Clang afaik.


Clang offers compatibility for most basic features. There are 
some ABI compatibility issues though and you find them reported 
in the Clang/libc++ bugtrackers, and it's a pain (but the 
Clang/LLVM guys think they can do some ABI better/faster than 
what GCC offers, so some breakage is deliberate).
In Debian, GCC compiles everything as the system's default 
compiler, so at least inside the distribution we don't have to 
worry about potential incompatibilities. Since GCC also supports 
an enormous amount of architectures and has strong optimization, 
the case is different there.

In terms of "what happens when users use the OSes C++ libraries 
and compile with Clang instead of GCC" the situation is similar 
though: They might run into ABI issues (rarer though than with 
D). For the distro itself the problem doesn't exist though.
Apr 10 2017
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:

 Hi there!
 [...]
 If we do that, we will run into the D ABI trap: Libraries 
 compiled with compiler X can not be used from software compiled 
 with D compiler Y. There is actually no ABI stability guarantee 
 even between DMD releases.
 This will make integrating D a huge pain. Recompiling the 
 dependency-chain of a software from source when compiling a 
 package using the "right" compiler and statically adding the 
 code is forbidden by distro policy. Having static libraries in 
 the dependencies doesn't solve the issue. Compiling each 
 library with all D compilers is highly impractical and not 
 really feasible.
 So, how should we proceed here? We could make it "DMD is the 
 only thing on the highway" compiling everything with DMD with 
 zero exceptions, which would leave us with only DMD-internal 
 ABI breakage and bad D code performance for some libraries.


I am working on a project that will require to be used almost 
exclusively with LDC (targeting the GPU backends of LLVM), and 
while technically only the mangling needs to match using any 
other complier will likely be a major workflow impediment. 
Perhaps it is reasonable to make them compile the stdlibs for 
LDC, but if it becomes as popular as I hope (dominating it's 
niché) it may be worthwhile considering shipping prebuilts.
Apr 10 2017
prev sibling next sibling parent reply Gerald <gerald.b.nunn gmail.com> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:

 There are a two issues though that we will be facing in Debian 
 soon, and I would like to get some opinion and maybe 
 perspective on from the D community on them.


First I would like to say thank you for all the work you did in 
getting Tilix packaged for Debian, it is very much appreciated. 
While I'm not an expert in this topic I'll throw in my two cents 
anyway :)

So to the topic at end, I tend to agree with Vladimir in that I 
just don't see it as feasible to make every dependency a shared 
library. Personally I would suggest that only libphobos and 
libdruntime be managed as shared libraries by a distro. 
Otherwise, any other dependencies used in DUB or elsewhere would 
be statically compiled.

If in time a new D library ends up becoming a keystone foundation 
for many packages it could be considered for inclusion as a 
shared library. Otherwise, trying to manage every DUB dependency 
as a potential shared library is a huge amount of work and I 
don't feel most of them are mature or well maintained enough to 
support this approach.

If you just concentrate on compiler, libphobos and libdruntime, 
you can have separate packages for each compiler toolchain which 
was Arch does. It has a libphobos package for DMD and a 
liblphobos for LDC. This then enables developers to specify the 
tool chain they prefer without interference.

In addition to the question about D and C++. what do distros 
typically do for Rust and Cargo dependencies, or Java and Maven? 
Wouldn't that be a similar paradigm to D and DUB dependencies?
Apr 10 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 14:21:43 UTC, Gerald wrote:

 On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:

 There are a two issues though that we will be facing in Debian 
 soon, and I would like to get some opinion and maybe 
 perspective on from the D community on them.


 First I would like to say thank you for all the work you did in 
 getting Tilix packaged for Debian, it is very much appreciated. 
 While I'm not an expert in this topic I'll throw in my two 
 cents anyway :)


Thanks!


 So to the topic at end, I tend to agree with Vladimir in that I 
 just don't see it as feasible to make every dependency a shared 
 library. Personally I would suggest that only libphobos and 
 libdruntime be managed as shared libraries by a distro. 
 Otherwise, any other dependencies used in DUB or elsewhere 
 would be statically compiled.


This would raise the maintenance effort for D software, and 
require policy exceptions, which is something we won't do. It's 
more likely that we'll see less D in the distro than going down 
this route.
DUB is unusable for Debian packaging anyway, and alternatives 
like Automake or Meson handle shared libraries really well, so in 
itself there is no reason not to do it.


 If in time a new D library ends up becoming a keystone 
 foundation for many packages it could be considered for 
 inclusion as a shared library. Otherwise, trying to manage 
 every DUB dependency as a potential shared library is a huge 
 amount of work and I don't feel most of them are mature or well 
 maintained enough to support this approach.


We'll take care of that, it's what we do as a distro and we have 
a fair amount of experience in handling these kinds of upstream 
projects.
The goal is to reduce the maintenance cost for things once they 
are *in* the distro, any prior work to get it into the 
distribution is very well invested.
That's why I have been writing a lot of Makefiles and Meson build 
definitions lately.


 If you just concentrate on compiler, libphobos and libdruntime, 
 you can have separate packages for each compiler toolchain 
 which was Arch does.


That's a necessity already and we do the same. But with an 
unstable ABI the standard library is also affected, which would 
trigger us to do Haskell-style versioning (which mangles 
dependencies to depend on a virtual package containing a hash of 
the GHC version), and that not only sucks but also requires quite 
a lot of manpower.


 It has a libphobos package for DMD and a liblphobos for LDC. 
 This then enables developers to specify the tool chain they 
 prefer without interference.


Yes, but that's not feasible for use *in the distribution itself* 
as we can't just pick and choose the right compiler per package 
if something up in the dependency chain was compiled with the 
"wrong" compiler.


 In addition to the question about D and C++. what do distros 
 typically do for Rust and Cargo dependencies, or Java and 
 Maven? Wouldn't that be a similar paradigm to D and DUB 
 dependencies?


Maven dependencies are also separated out into smaller packages 
and maven (unlike dub) plays relatively well with Debian. It also 
has no ABI issues and interface stability can be tracked. Rust 
only has one compiler which strongly optimizes, so we don't have 
the problem of choosing the right one. Cargo is/was an issue but 
it's being worked on and seems to work well now: 
https://wiki.debian.org/Teams/RustPackaging/Cargo
(not that I am not involved in Haskell, Rust or Java packaging).

Just because it makes sense: We do have an upstream guide showing 
some best practices for upstream projects which help us to 
maintain software for long periods. It also contains 
language-specific advice and some general info which is valid for 
all languages:
https://wiki.debian.org/UpstreamGuide
Apr 10 2017
parent reply David Nadlinger <code klickverbot.at> writes:
On Monday, 10 April 2017 at 17:27:28 UTC, Matthias Klumpp wrote:

 That's why I have been writing a lot of Makefiles and Meson 
 build definitions lately.


It seems like doing so without having a closer look at the 
realities of D software (no stable ABI, etc.) might not have been 
the best use of your time. How would Dub be such fundamentally 
unfixable that making the official language tools play nice with 
distro packaging (and the other way round) wouldn't be preferable 
over manually re-writing build systems all over the place?


 But with an unstable ABI the standard library is also affected, 
 which would trigger us to do Haskell-style versioning (which 
 mangles dependencies to depend on a virtual package containing 
 a hash of the GHC version), and that not only sucks but also 
 requires quite a lot of manpower.


So it sounds like there is a solution already for other 
languages. Could you elaborate some more on the problems with it? 
I suppose there is some wiki page documenting the process 
somewhere?


 Rust only has one compiler which strongly optimizes, so we 
 don't have the problem of choosing the right one. Cargo is/was 
 an issue but it's being worked on and seems to work well now: 
 https://wiki.debian.org/Teams/RustPackaging/Cargo


Rust doesn't have a stable ABI either, and it doesn't look like 
there is any movement in that direction (not that I think that 
there should be). That the people driving the effort might not be 
aware of it yet doesn't mean it isn't an issue for them.

IIRC OCaml is also very much a statically linked affair. And how 
does Debian distribute Go binaries? Is there any issue with those 
being linked statically? If not, let's just distribute D 
libraries as source and compile/link them statically when 
building binaries, and problem solved.

Some of the compiler developers, myself included, understand the 
issues involving ABI stability and distro packaging quite well 
(although the latter admittedly only on a general level). In 
fact, one of my earliest open source memories is of some work in 
the trenches ensuring ABI stability of some bits of KDE across 
releases. Yet we are still going to tell you that the D ABI is 
going to remain unstable for the foreseeable future. This is not 
something that just requires a man-week or month to "fix" in the 
compiler, but would impact many other areas as well, for example 
language evolution.

If you somehow got the impression that this is just due to D 
developers "not getting it", just have a look at the other recent 
compiled languages. Go and Rust don't fare any differently, and 
even Swift, with all its development manpower, doesn't have a 
stable ABI yet [1]. And I believe header-only C++ template 
libraries have been mentioned already as well.

I can only speak for myself, of course, but I certainly see the 
strategic importance of integration into the Linux distribution 
ecosystem for D, and I'm very happy to work with packagers 
wherever possible. However, you also need to acknowledge the 
properties of the ecosystem *you* are working with. If you see a 
big stretch of difficult terrain in front of you, closing your 
eyes won't make it go away; you'll only lose time you could spend 
working around it. ABI instability is something you'll have to 
work around one way or the other.

  — David


[1] Actually, the Swift guys tried to pin it down leading up to 
version 4, but in the end decided it was too much work to get 
done in time and possibly not a wise idea at this point 
altogether.
Apr 10 2017
next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Monday, April 10, 2017 23:08:17 David Nadlinger via Digitalmars-d wrote:

 IIRC OCaml is also very much a statically linked affair. And how
 does Debian distribute Go binaries? Is there any issue with those
 being linked statically? If not, let's just distribute D
 libraries as source and compile/link them statically when
 building binaries, and problem solved.


Honestly, I don't see how it really makes much sense to use shared libraries
with D except in cases where you have no choice. The lack of ABI
compatibility makes them almost useless.

Also, what are we even looking to distribute in debian? I would have thought
that the normal thing to do would be to build with dub, in which case,
having the compiler and dub be debian packages makes sense but not really
anything else. If you're looking to package an application that was written
in D, then that becomes another question, but then if you just statically
link it, the ABI compatibility problem goes away as does any need to package
any D library dependencies.

- Jonathan M Davis
Apr 10 2017
next sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 00:47:34 UTC, Jonathan M Davis wrote:

 On Monday, April 10, 2017 23:08:17 David Nadlinger via [...]
 Also, what are we even looking to distribute in debian? I would 
 have thought that the normal thing to do would be to build with 
 dub, in which case, having the compiler and dub be debian 
 packages makes sense but not really anything else. If you're 
 looking to package an application that was written in D, then 
 that becomes another question, but then if you just statically 
 link it, the ABI compatibility problem goes away as does any 
 need to package any D library dependencies.


You will have static-library packages which have the exact same 
ABI issues shared libraries have.
And yeah, this is obviously about stuff being built with D 
compilers in the distro, such as Tilix, BioD, AppStream Generator 
and all future things which might emerge and be useful to have in 
the OS.
Apr 11 2017
parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, April 11, 2017 12:40:34 Matthias Klumpp via Digitalmars-d wrote:

 On Tuesday, 11 April 2017 at 00:47:34 UTC, Jonathan M Davis wrote:

 On Monday, April 10, 2017 23:08:17 David Nadlinger via [...]
 Also, what are we even looking to distribute in debian? I would
 have thought that the normal thing to do would be to build with
 dub, in which case, having the compiler and dub be debian
 packages makes sense but not really anything else. If you're
 looking to package an application that was written in D, then
 that becomes another question, but then if you just statically
 link it, the ABI compatibility problem goes away as does any
 need to package any D library dependencies.


 You will have static-library packages which have the exact same
 ABI issues shared libraries have.
 And yeah, this is obviously about stuff being built with D
 compilers in the distro, such as Tilix, BioD, AppStream Generator
 and all future things which might emerge and be useful to have in
 the OS.


My point was that there really isn't a reason to package D libraries in the
distro. ABI compatibility makes including D libraries in a distro
problematic, and dub takes care dependencies for you, making it completely
unnecessary to include any D libraries in a distro. Now, there may be a
reason to include a D _program_ in a distro, but with a D program, you can
just build it with dub and statically link in its D libraries so that they
don't need to be part of the distro. And if we go that route, you pretty
much avoid this whole mess.

- Jonathan M Davis
Apr 11 2017
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-04-11 02:47, Jonathan M Davis via Digitalmars-d wrote:


 Honestly, I don't see how it really makes much sense to use shared libraries
 with D except in cases where you have no choice. The lack of ABI
 compatibility makes them almost useless.

 Also, what are we even looking to distribute in debian? I would have thought
 that the normal thing to do would be to build with dub, in which case,
 having the compiler and dub be debian packages makes sense but not really
 anything else. If you're looking to package an application that was written
 in D, then that becomes another question, but then if you just statically
 link it, the ABI compatibility problem goes away as does any need to package
 any D library dependencies.


I agree, I don't see any point in distributing libraries. just 
applications. But I do know some people will refuse to install anything 
that doesn't come through the system package manager.

-- 
/Jacob Carlborg
Apr 11 2017
parent Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 13:43:38 UTC, Jacob Carlborg wrote:

 On 2017-04-11 02:47, Jonathan M Davis via Digitalmars-d wrote:


 Honestly, I don't see how it really makes much sense to use 
 shared libraries
 with D except in cases where you have no choice. The lack of 
 ABI
 compatibility makes them almost useless.

 Also, what are we even looking to distribute in debian? I 
 would have thought
 that the normal thing to do would be to build with dub, in 
 which case,
 having the compiler and dub be debian packages makes sense but 
 not really
 anything else. If you're looking to package an application 
 that was written
 in D, then that becomes another question, but then if you just 
 statically
 link it, the ABI compatibility problem goes away as does any 
 need to package
 any D library dependencies.


 I agree, I don't see any point in distributing libraries. just 
 applications. But I do know some people will refuse to install 
 anything that doesn't come through the system package manager.


Every single bit of software that is available in the 
distribution needs to be packaged in it so you can replicate its 
build using only what is available in the distro. Fetching things 
from the internet is not allowed.
(That's actually a hard no in the policy, while other things like 
the use of static linking are rather "you shouldn't do it if you 
can avoid it")

The fundamental thing a distribution does is integrating software 
and creating a consistent whole out of many moving parts. In 
order to do that, you absolutely can not rely on site-specific 
package managers like pip, dub, npm, etc. as they are not built 
for that purpose and only see their own ecosystem.
You could still build with them though in case they integrate 
well with the distro.
Apr 11 2017
prev sibling parent Enrico Weigelt, metux IT consult <enrico.weigelt gr13.net> writes:
On Tuesday, 11 April 2017 at 00:47:34 UTC, Jonathan M Davis wrote:

Hi folks,


 Also, what are we even looking to distribute in debian? I would 
 have thought that the normal thing to do would be to build with 
 dub, in which case, having the compiler and dub be debian 
 packages makes sense but not really anything else.


Are you only targeting manually operated inhouse applications
where operator = developer ?

I haven't done any serious D programming yet - the language itself
looks quite interesting to me, eg. for embedded systems.

Unfortunately, the current situation makes it pretty unusable to 
me.


 but then if you just statically link it, the ABI compatibility 
 problem goes away as does any need to package any D library 
 dependencies.


Technically correct, but then the maintainers of individual 
packages
have to keep an eye on each single package in the dependency 
chain,
eg. on security issues and rebuild in those cases - basically 
defeating
one of the major goals of the concept of distros.


--mtx
May 19 2017
prev sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 23:08:17 UTC, David Nadlinger wrote:

 On Monday, 10 April 2017 at 17:27:28 UTC, Matthias Klumpp wrote:

 That's why I have been writing a lot of Makefiles and Meson 
 build definitions lately.


 It seems like doing so without having a closer look at the 
 realities of D software (no stable ABI, etc.) might not have 
 been the best use of your time. How would Dub be such 
 fundamentally unfixable that making the official language tools 
 play nice with distro packaging (and the other way round) 
 wouldn't be preferable over manually re-writing build systems 
 all over the place?


Dub is not unfixable, but there are many more issues with it 
which make it very very hard to use in a distribution context - I 
tried that and it didn't work well.
I am treating D in Debian like I would treat C++, which is quite 
fair given the similar featuresets and challenges.


 But with an unstable ABI the standard library is also 
 affected, which would trigger us to do Haskell-style 
 versioning (which mangles dependencies to depend on a virtual 
 package containing a hash of the GHC version), and that not 
 only sucks but also requires quite a lot of manpower.


 So it sounds like there is a solution already for other 
 languages. Could you elaborate some more on the problems with 
 it? I suppose there is some wiki page documenting the process 
 somewhere?


Haskell and OCaml permanently rebuild the whole stack on every 
new compiler release, which is why they have permanent transition 
trackers[1], so they basically continuously rebuild. I want to 
avoid this at all cost for D, as this is very very maintenance 
intensive and painful, and will require much more people to work 
on it than D has available in Debian.

[1]: https://release.debian.org/transitions/


 Rust only has one compiler which strongly optimizes, so we 
 don't have the problem of choosing the right one. Cargo is/was 
 an issue but it's being worked on and seems to work well now: 
 https://wiki.debian.org/Teams/RustPackaging/Cargo


 Rust doesn't have a stable ABI either, and it doesn't look like 
 there is any movement in that direction (not that I think that 
 there should be). That the people driving the effort might not 
 be aware of it yet doesn't mean it isn't an issue for them.


It's being worked on[2], but it's not a super-high priority.
There doesn't seem to be a definitive answer on how Rust is 
handled in Debian yet (but to know for certain, I would need to 
ask the Rust team).

[2]: https://github.com/rust-lang/rfcs/issues/600


 IIRC OCaml is also very much a statically linked affair. And 
 how does Debian distribute Go binaries? Is there any issue with 
 those being linked statically? If not, let's just distribute D 
 libraries as source and compile/link them statically when 
 building binaries, and problem solved.


Surprisingly it looks like many Go packages are indeed provided 
as source-installations.
Doing something like this with D would require Makefiles and Dub 
to pick up sources from system locations properly which isn't 
really done yet...


 Some of the compiler developers, myself included, understand 
 the issues involving ABI stability and distro packaging quite 
 well (although the latter admittedly only on a general level). 
 In fact, one of my earliest open source memories is of some 
 work in the trenches ensuring ABI stability of some bits of KDE 
 across releases. Yet we are still going to tell you that the D 
 ABI is going to remain unstable for the foreseeable future. 
 This is not something that just requires a man-week or month to 
 "fix" in the compiler, but would impact many other areas as 
 well, for example language evolution.

 If you somehow got the impression that this is just due to D 
 developers "not getting it", just have a look at the other 
 recent compiled languages. Go and Rust don't fare any 
 differently, and even Swift, with all its development manpower, 
 doesn't have a stable ABI yet [1]. And I believe header-only 
 C++ template libraries have been mentioned already as well.

 I can only speak for myself, of course, but I certainly see the 
 strategic importance of integration into the Linux distribution 
 ecosystem for D, and I'm very happy to work with packagers 
 wherever possible. However, you also need to acknowledge the 
 properties of the ecosystem *you* are working with. If you see 
 a big stretch of difficult terrain in front of you, closing 
 your eyes won't make it go away; you'll only lose time you 
 could spend working around it. ABI instability is something 
 you'll have to work around one way or the other.


That's the whole point of this thread, I want to find the best 
solution to deal with this issue and also one the D community can 
live with.
I am not set on any particular solution yet, at the moment I see 
the problem and I am thinking about how to deal with it in the 
best possible way (ideally not doing what OCaml/Haskell are 
doing, although pulling a "soft Haskell" where we rebuild 
everything when the D ABI changes would be way easier).
Apr 11 2017
next sibling parent reply Gerald <gerald.b.nunn gmail.com> writes:
Looking into Rust and Cargo Debian packaging, it looks like they 
are looking to go with source packages to work around the ABI 
issue:

"Because Rust doesn't have a stable shared-library ABI, and we 
don't want to rebuild every Rust library package for every 
architecture every time we upload a new version of rustc, we 
package library crates as source code, installed into 
/usr/share/cargo/registry/cratename-version/. Thus, the Debian 
"binary" package for a library crate contains the same source as 
the source package."

https://internals.rust-lang.org/t/debian-rust-packaging-policy-draft/4453
Apr 11 2017
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 12:38 +0000, Gerald via Digitalmars-d wrote:

 Looking into Rust and Cargo Debian packaging, it looks like they=C2=A0
 are looking to go with source packages to work around the ABI=C2=A0
 issue:
=20
 "Because Rust doesn't have a stable shared-library ABI, and we=C2=A0
 don't want to rebuild every Rust library package for every=C2=A0
 architecture every time we upload a new version of rustc, we=C2=A0
 package library crates as source code, installed into=C2=A0
 /usr/share/cargo/registry/cratename-version/. Thus, the Debian=C2=A0
 "binary" package for a library crate contains the same source as=C2=A0
 the source package."
=20
 https://internals.rust-lang.org/t/debian-rust-packaging-policy-draft/
 4453


So why bother, no Rust people will use it, they just use Cargo and it
gets stuff for them. Why bother packaging anything when people will not
use it? All you need to have a working Rust system on Debian is curl
and bash.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Apr 11 2017
parent Gerald <gerald.b.nunn gmail.com> writes:
On Tuesday, 11 April 2017 at 12:47:13 UTC, Russel Winder wrote:

 So why bother, no Rust people will use it, they just use Cargo 
 and it gets stuff for them. Why bother packaging anything when 
 people will not use it? All you need to have a working Rust 
 system on Debian is curl and bash.


Users of an application written in Rust are not developers, they 
just want to install an app using whatever packaging mechanism 
the distro provides. Also, I would expect that the source library 
packages for Rust are likely just a build dependency and users 
are not building Rust binaries on their system. This is just a 
way for the Debian packagers to build packages and track 
dependencies using the tools and processes they already have in 
place.

Having said that, I don't disagree with you as I think it's a bit 
silly to replicate what cargo, maven and dub already provide OOTB 
and essentially duplicate their existing repositories. However, 
if Rust is going in this direction it can make sense for D to 
follow and just make the necessary improvements in Dub to support 
the process.
Apr 11 2017
prev sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, April 11, 2017 12:25:09 Matthias Klumpp via Digitalmars-d wrote:

 Haskell and OCaml permanently rebuild the whole stack on every
 new compiler release, which is why they have permanent transition
 trackers[1], so they basically continuously rebuild. I want to
 avoid this at all cost for D, as this is very very maintenance
 intensive and painful, and will require much more people to work
 on it than D has available in Debian.


That's basically what's required with D. It is not ABI compatible across
releases, and while ABI compatibility might be nice, it really isn't
reasonable with D - especially with how attributes work and how template
heavy D code is. If the D compiler gets upgraded, then anything that gets
built needs everything that it depends on to have been built with that exact
compiler version.

But if we just use dub - which _is_ the official packaging and build tool -
then we avoid these issues. Ideally, the compiler and dub would be part of
the distro, but libraries don't need to be. And it sounds like that's
basically how the Go and Rust folks want to function as well. So, it would
make sense for these languages to simply not have their libraries be
included in distros. The build tools are plenty.

- Jonathan M Davis
Apr 11 2017
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-04-11 14:56, Jonathan M Davis via Digitalmars-d wrote:


 That's basically what's required with D. It is not ABI compatible across
 releases, and while ABI compatibility might be nice, it really isn't
 reasonable with D - especially with how attributes work and how template
 heavy D code is. If the D compiler gets upgraded, then anything that gets
 built needs everything that it depends on to have been built with that exact
 compiler version.

 But if we just use dub - which _is_ the official packaging and build tool -
 then we avoid these issues. Ideally, the compiler and dub would be part of
 the distro, but libraries don't need to be. And it sounds like that's
 basically how the Go and Rust folks want to function as well. So, it would
 make sense for these languages to simply not have their libraries be
 included in distros. The build tools are plenty.


I agree, but I think that they want to avoid having packages that 
require dependencies outside of the Debian tree to build. That is, a 
user end application written in D, that is built using Dub. To solve 
that and still use Dub they would need to package the dependent 
libraries as source packages and somehow point Dub to the library in the 
Debian tree and not code.dlang.org. I don't see why that wouldn't be 
possible and if anything needs to be changed in Dub for that, I think 
that's reasonable.

-- 
/Jacob Carlborg
Apr 11 2017
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 11/04/2017 3:01 PM, Jacob Carlborg wrote:

 On 2017-04-11 14:56, Jonathan M Davis via Digitalmars-d wrote:


 That's basically what's required with D. It is not ABI compatible across
 releases, and while ABI compatibility might be nice, it really isn't
 reasonable with D - especially with how attributes work and how template
 heavy D code is. If the D compiler gets upgraded, then anything that gets
 built needs everything that it depends on to have been built with that
 exact
 compiler version.

 But if we just use dub - which _is_ the official packaging and build
 tool -
 then we avoid these issues. Ideally, the compiler and dub would be
 part of
 the distro, but libraries don't need to be. And it sounds like that's
 basically how the Go and Rust folks want to function as well. So, it
 would
 make sense for these languages to simply not have their libraries be
 included in distros. The build tools are plenty.


 I agree, but I think that they want to avoid having packages that
 require dependencies outside of the Debian tree to build. That is, a
 user end application written in D, that is built using Dub. To solve
 that and still use Dub they would need to package the dependent
 libraries as source packages and somehow point Dub to the library in the
 Debian tree and not code.dlang.org. I don't see why that wouldn't be
 possible and if anything needs to be changed in Dub for that, I think
 that's reasonable.


/usr/share/source/D/package-name-version

Add a search path like that to Dub and create source only library 
packages and that is pretty much all the distribution we need for 
libraries I reckon.

Destroy!
Apr 11 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 14:04:44 UTC, rikki cattermole wrote:

 [...]
 /usr/share/source/D/package-name-version

 Add a search path like that to Dub and create source only 
 library packages and that is pretty much all the distribution 
 we need for libraries I reckon.


It's more likely that the path would be `/usr/include/d` because 
that's the place the D sources seem to usually reside, especially 
if they are also used as "headers" for shared libraries.

At time I am playing around with the idea of using pkg-config[1] 
files to enlist the sources a D library consists of.
By doing that, we would have a very build-system agnostic way of 
doing storing that information that can be used by Automake 
(native), Dub, Meson and even plain Makefiles. It's also very 
widely used (although it is commonly not used to list sources).

This would also jive really well with the existing D packages in 
Debian and we could - if we go down the source-only route - still 
decide to precompile a few selected libraries, for example in 
case they are very time consuming to compile or really large.

This will *not* solve the issues with Phobos breakage though, as 
Phobos is a shared library. But doing that would pretty much work 
around all other problems, except for the cost of having 
statically linked binaries, but that seems to be inevitable 
anyway given the state of things.

(Disclaimer: This is not a D team policy draft yet, just an 
opinion at the moment that might be refined or changed entirely)

[1]: https://www.freedesktop.org/wiki/Software/pkg-config/
Apr 11 2017
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 11/04/2017 3:21 PM, Matthias Klumpp wrote:

 On Tuesday, 11 April 2017 at 14:04:44 UTC, rikki cattermole wrote:

 [...]
 /usr/share/source/D/package-name-version

 Add a search path like that to Dub and create source only library
 packages and that is pretty much all the distribution we need for
 libraries I reckon.


 It's more likely that the path would be `/usr/include/d` because that's
 the place the D sources seem to usually reside, especially if they are
 also used as "headers" for shared libraries.

 At time I am playing around with the idea of using pkg-config[1] files
 to enlist the sources a D library consists of.
 By doing that, we would have a very build-system agnostic way of doing
 storing that information that can be used by Automake (native), Dub,
 Meson and even plain Makefiles. It's also very widely used (although it
 is commonly not used to list sources).

 This would also jive really well with the existing D packages in Debian
 and we could - if we go down the source-only route - still decide to
 precompile a few selected libraries, for example in case they are very
 time consuming to compile or really large.

 This will *not* solve the issues with Phobos breakage though, as Phobos
 is a shared library. But doing that would pretty much work around all
 other problems, except for the cost of having statically linked
 binaries, but that seems to be inevitable anyway given the state of things.

 (Disclaimer: This is not a D team policy draft yet, just an opinion at
 the moment that might be refined or changed entirely)

 [1]: https://www.freedesktop.org/wiki/Software/pkg-config/


The problem with /usr/include/d is that is where .di files would be 
located not .d. This would also match up with the c/c++ usage of it.
Apr 11 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 14:26:37 UTC, rikki cattermole wrote:

 [...]
 The problem with /usr/include/d is that is where .di files 
 would be located not .d. This would also match up with the 
 c/c++ usage of it.


When I asked about this a while back, I was told to just install 
the sources into the include D as "almost nobody uses .di files 
except for proprietary libraries" (and do those even exist?).
But in any case, any path would be fine with me as long as people 
can settle on using it - `/usr/share/d` would be available ^^
Apr 11 2017
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 11/04/2017 3:33 PM, Matthias Klumpp wrote:

 On Tuesday, 11 April 2017 at 14:26:37 UTC, rikki cattermole wrote:

 [...]
 The problem with /usr/include/d is that is where .di files would be
 located not .d. This would also match up with the c/c++ usage of it.


 When I asked about this a while back, I was told to just install the
 sources into the include D as "almost nobody uses .di files except for
 proprietary libraries" (and do those even exist?).
 But in any case, any path would be fine with me as long as people can
 settle on using it - `/usr/share/d` would be available ^^


Satoshi is developing one and is having a load of trouble in doing so.
That says everything, we got a lot of work to do.
Apr 11 2017
prev sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, April 11, 2017 14:33:01 Matthias Klumpp via Digitalmars-d wrote:

 On Tuesday, 11 April 2017 at 14:26:37 UTC, rikki cattermole wrote:

 [...]
 The problem with /usr/include/d is that is where .di files
 would be located not .d. This would also match up with the
 c/c++ usage of it.


 When I asked about this a while back, I was told to just install
 the sources into the include D as "almost nobody uses .di files
 except for proprietary libraries" (and do those even exist?).
 But in any case, any path would be fine with me as long as people
 can settle on using it - `/usr/share/d` would be available ^^


Putting .di files in the include directory makes sense when compared to C++,
but it's definitely true that they're used quite rarely. They gain you very
little and cost you a lot (e.g. no CTFE). But unless someone were looking to
put both .di and .d files for the same module on the same system, it
wouldn't really be an issue to put them both in the include directory. I
would expect open source librares to use .di files very rarely though.

- Jonathan M Davis
Apr 11 2017
parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 11 Apr 2017 07:44:45 -0700
schrieb Jonathan M Davis via Digitalmars-d
<digitalmars-d puremagic.com>:


 On Tuesday, April 11, 2017 14:33:01 Matthias Klumpp via Digitalmars-d
 wrote:

 On Tuesday, 11 April 2017 at 14:26:37 UTC, rikki cattermole wrote:  

 [...]
 The problem with /usr/include/d is that is where .di files
 would be located not .d. This would also match up with the
 c/c++ usage of it.  


 When I asked about this a while back, I was told to just install
 the sources into the include D as "almost nobody uses .di files
 except for proprietary libraries" (and do those even exist?).
 But in any case, any path would be fine with me as long as people
 can settle on using it - `/usr/share/d` would be available ^^  


 
 Putting .di files in the include directory makes sense when compared
 to C++, but it's definitely true that they're used quite rarely. They
 gain you very little and cost you a lot (e.g. no CTFE). But unless
 someone were looking to put both .di and .d files for the same module
 on the same system, it wouldn't really be an issue to put them both
 in the include directory. I would expect open source librares to
 use .di files very rarely though.
 
 - Jonathan M Davis
 


I'd think of .d files as a superset of .di files. There's no reason to
install both (assuming same library version) and having the .d files
will produce better cross-module inlining so these are preferred.

Of course for proprietary libraries .di files are required. But
nevertheless .di and .d can be mixed in /usr/include/d.

The only downside is that inlining from /usr/include could cause
licensing problems (e.g. LGPL allows only linking, not sure how
inlining would be affected. But it seems they have an Exception for
inlining)


-- Johannes
Apr 11 2017
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 11 April 2017 at 21:08:05 UTC, Johannes Pfau wrote:

 I'd think of .d files as a superset of .di files.


Note that there is zero difference between them. The compiler 
treats both *exactly* the same way (it just considers the 
extensions to be aliases of each other).

It is just a naming convention.
Apr 11 2017
prev sibling next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, April 11, 2017 14:21:57 Matthias Klumpp via Digitalmars-d wrote:

 This will *not* solve the issues with Phobos breakage though, as
 Phobos is a shared library.


It could always just be distributed as a static library. There arguably
isn't much point in distributing it as a shared library anyway -
particularly when it's not ABI compatible across versions. I actively avoid
having phobos as a shared library on my systems, because it just causes
versioning problems when programs are built against it. All of those
problems go away with a static library. And so much of Phobos is templated
anyway that there isn't even much to share.

- Jonathan M Davis
Apr 11 2017
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 11 Apr 2017 07:40:12 -0700
schrieb Jonathan M Davis via Digitalmars-d
<digitalmars-d puremagic.com>:


 It could always just be distributed as a static library. There arguably
 isn't much point in distributing it as a shared library anyway -
 particularly when it's not ABI compatible across versions. I actively avoid
 having phobos as a shared library on my systems, because it just causes
 versioning problems when programs are built against it. All of those
 problems go away with a static library. And so much of Phobos is templated
 anyway that there isn't even much to share.
 
 - Jonathan M Davis


I see what you're doing there, but your last point is wishful
thinking. Dynamically linked binaries can share megabytes of
code. Even Phobos - although heavily templated - has proven to
be very amenable to sharing. For example, a "Hello world!"
program using `writeln()` has these sizes when compiled with
`dmd -O -release -inline`:

     static Phobos2 : 806968 bytes
    dynamic Phobos2 :  18552 bytes

That's about 770 KiB to share or 97.7% of its total size!
Awesome!

Another package we have on Gentoo now is the tiling terminal
emulator Tilix, compiled with its default `dmd -O`:

     static Phobos2+GtkD : 14828272 bytes
    dynamic Phobos2+GtkD :  6126464 bytes

So here we get ~8.3 megabytes in binary size reduction due to
sharing. (Though 6 MiB for a terminal is still a lot - are
we slim yet? :P )

Some forms of proprietary code and GPL code need clean
shared library interfaces, free from templates and attribute
inference. I mentioned this before: As far as feasible have
full attribute inference for non-exported functions and
reliance on only explicit attributes for exported functions,
so that code changes can't break the public API (due to
mangling changes).

When Dlang's ABI changes, release a major version and make a
public announcement so that everyone can schedule the system
update to a convenient time as is done with GCC C++ ABI
changes.

-- 
Marco
May 02 2017
next sibling parent David Nadlinger <code klickverbot.at> writes:
On Tuesday, 2 May 2017 at 19:34:44 UTC, Marco Leise wrote:

      static Phobos2 : 806968 bytes
     dynamic Phobos2 :  18552 bytes

 That's about 770 KiB to share or 97.7% of its total size! 
 Awesome!


By the way, using LDC: 402736 bytes for the static build (Linux 
x86_64). ;)

  — David
May 02 2017
prev sibling parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Tuesday, 2 May 2017 at 19:34:44 UTC, Marco Leise wrote:

 I see what you're doing there, but your last point is wishful 
 thinking. Dynamically linked binaries can share megabytes of 
 code. Even Phobos - although heavily templated - has proven to 
 be very amenable to sharing. For example, a "Hello world!" 
 program using `writeln()` has these sizes when compiled with 
 `dmd -O -release -inline`:

      static Phobos2 : 806968 bytes
     dynamic Phobos2 :  18552 bytes

 That's about 770 KiB to share or 97.7% of its total size! 
 Awesome!


Is all of that active code, or is some of that (statically 
knowable) never getting executed (as in could've been removed at 
compile/link time)?
May 02 2017
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 02 May 2017 20:53:50 +0000
schrieb Moritz Maxeiner <moritz ucworks.org>:


 On Tuesday, 2 May 2017 at 19:34:44 UTC, Marco Leise wrote:

 I see what you're doing there, but your last point is wishful 
 thinking. Dynamically linked binaries can share megabytes of 
 code. Even Phobos - although heavily templated - has proven to 
 be very amenable to sharing. For example, a "Hello world!" 
 program using `writeln()` has these sizes when compiled with 
 `dmd -O -release -inline`:

      static Phobos2 : 806968 bytes
     dynamic Phobos2 :  18552 bytes

 That's about 770 KiB to share or 97.7% of its total size! 
 Awesome!  


 
 Is all of that active code, or is some of that (statically 
 knowable) never getting executed (as in could've been removed at 
 compile/link time)?


I guess David gave you the answer. So it's just 95.4% of its
total size. :p
By the way, is the fully dynamic linking version possible with
ldc2 now as well ?

-- 
Marco
May 02 2017
parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Tuesday, 2 May 2017 at 23:27:28 UTC, Marco Leise wrote:

 Am Tue, 02 May 2017 20:53:50 +0000
 schrieb Moritz Maxeiner <moritz ucworks.org>:


 On Tuesday, 2 May 2017 at 19:34:44 UTC, Marco Leise wrote:

 I see what you're doing there, but your last point is 
 wishful thinking. Dynamically linked binaries can share 
 megabytes of code. Even Phobos - although heavily templated 
 - has proven to be very amenable to sharing. For example, a 
 "Hello world!" program using `writeln()` has these sizes 
 when compiled with `dmd -O -release -inline`:

      static Phobos2 : 806968 bytes
     dynamic Phobos2 :  18552 bytes

 That's about 770 KiB to share or 97.7% of its total size! 
 Awesome!


 
 Is all of that active code, or is some of that (statically 
 knowable) never getting executed (as in could've been removed 
 at compile/link time)?


 I guess David gave you the answer. So it's just 95.4% of its
 total size. :p


Under the assumption that ldc2 produces no dead code in the 
output; is that a reasonable assumption (I'm not sure)?


 By the way, is the fully dynamic linking version possible with
 ldc2 now as well ?


I did have a modified ebuild to try that out a while back and it 
seemed to work fine in my limited testing scope. Since I quite 
often change installed d compilers and don't want my programs 
(like tilix) to stop working (or have old d compiler versions 
being kept installed *just* because some programs were built with 
them), I generally link against phobos statically, anyway, so I 
my tests weren't exhaustive. The cmake flag to to use in the 
ebuild would be BUILD_SHARED [1].

[1] 
https://github.com/ldc-developers/ldc/blob/v1.2.0/CMakeLists.txt#L522
May 02 2017
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 03 May 2017 01:02:38 +0000
schrieb Moritz Maxeiner <moritz ucworks.org>:


 On Tuesday, 2 May 2017 at 23:27:28 UTC, Marco Leise wrote:

 Am Tue, 02 May 2017 20:53:50 +0000
 schrieb Moritz Maxeiner <moritz ucworks.org>:
  

 On Tuesday, 2 May 2017 at 19:34:44 UTC, Marco Leise wrote:  

 I see what you're doing there, but your last point is 
 wishful thinking. Dynamically linked binaries can share 
 megabytes of code. Even Phobos - although heavily templated 
 - has proven to be very amenable to sharing. For example, a 
 "Hello world!" program using `writeln()` has these sizes 
 when compiled with `dmd -O -release -inline`:

      static Phobos2 : 806968 bytes
     dynamic Phobos2 :  18552 bytes

 That's about 770 KiB to share or 97.7% of its total size! 
 Awesome!  


 
 Is all of that active code, or is some of that (statically 
 knowable) never getting executed (as in could've been removed 
 at compile/link time)?  


 I guess David gave you the answer. So it's just 95.4% of its
 total size. :p  


 
 Under the assumption that ldc2 produces no dead code in the 
 output; is that a reasonable assumption (I'm not sure)?
 

 By the way, is the fully dynamic linking version possible with
 ldc2 now as well ?  


 
 I did have a modified ebuild to try that out a while back and it 
 seemed to work fine in my limited testing scope. Since I quite 
 often change installed d compilers and don't want my programs 
 (like tilix) to stop working (or have old d compiler versions 
 being kept installed *just* because some programs were built with 
 them), I generally link against phobos statically, anyway, so I 
 my tests weren't exhaustive. The cmake flag to to use in the 
 ebuild would be BUILD_SHARED [1].
 
 [1] 
 https://github.com/ldc-developers/ldc/blob/v1.2.0/CMakeLists.txt#L522


I remember seeing this line and thinking "aww .. no way to
build both versions?". I.e. enable the "static-libs" USE flag
to _also_ build static libraries accompanying the shared ones.
I'll create an issue asking about that.

-- 
Marco
May 03 2017
prev sibling next sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 14:21 +0000, Matthias Klumpp via Digitalmars-d
wrote:

=20


[=E2=80=A6]

 At time I am playing around with the idea of using pkg-config[1]=C2=A0
 files to enlist the sources a D library consists of.
 By doing that, we would have a very build-system agnostic way of=C2=A0
 doing storing that information that can be used by Automake=C2=A0
 (native), Dub, Meson and even plain Makefiles. It's also very=C2=A0
 widely used (although it is commonly not used to list sources).


And SCons.

What about CMake? (Which is the CLion build system using Make
underneath, but they are rapidly moving to supporting Ninja.)

BTW, I'd like to reiterate that the D support in Meson is great, thanks
for getting that  in place. Meson and CMake are the only systems that
can install stuff sensibly.=20


 [=E2=80=A6]


--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Apr 11 2017
parent Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 14:43:15 UTC, Russel Winder wrote:

 On Tue, 2017-04-11 at 14:21 +0000, Matthias Klumpp via 
 Digitalmars-d wrote:

 


 […]

 At time I am playing around with the idea of using 
 pkg-config[1]
 files to enlist the sources a D library consists of.
 By doing that, we would have a very build-system agnostic way 
 of
 doing storing that information that can be used by Automake
 (native), Dub, Meson and even plain Makefiles. It's also very
 widely used (although it is commonly not used to list sources).


 And SCons.

 What about CMake? (Which is the CLion build system using Make 
 underneath, but they are rapidly moving to supporting Ninja.)


CMake supports this as well: 
https://cmake.org/cmake/help/v3.0/module/FindPkgConfig.html

The current idea is in case a library "foobar" would be packaged, 
to have a "foobar-src.pc" pkgconfig file (additionally to a 
potentially also existing "foobar.pc" file), containing something 
like this:

```
prefix=/usr/local
exec_prefix=${prefix}
includedir=${prefix}/include/d/foo

Name: foobar
Description: The foo library (sources)
Version: 1.0.0
Cflags: -I${includedir} -d-version=blahblub
Sources: ${includedir}/alpha.d ${includedir}/beta.d 
${includedir}/gamma.d
```

Build systems would then need to explicitly fetch the Sources 
field and add its expanded values to the project's sources.
(Using Cflags for this would be messy, as the build system might 
want to deal with flags and sources separately).

Alternatively dub could also define a layout and we write plugins 
for each build-system to make it work.

This will be really annoying with large libraries like GtkD 
though, which will require substantially longer to build. Maybe 
it's worth keeping some libraries precompiled (thereby locking 
all their reverse-dependencies to whatever D compiler was used to 
compile the library).

One problem with the pkg-config approach is that to support the 
precompiled case too, build systems would need to search for both 
"foobar" and "foobar-src" and only fail dependency detection if 
both of them are missing.

At least with CMake/Meson that's easy to do though (although it's 
a bit messy).
Apr 11 2017
prev sibling next sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 07:40 -0700, Jonathan M Davis via Digitalmars-d
wrote:

 On Tuesday, April 11, 2017 14:21:57 Matthias Klumpp via Digitalmars-d=20
 wrote:

 This will *not* solve the issues with Phobos breakage though, as
 Phobos is a shared library.


=20
 It could always just be distributed as a static library. There
 arguably
 isn't much point in distributing it as a shared library anyway -
 particularly when it's not ABI compatible across versions. I actively
 avoid
 having phobos as a shared library on my systems, because it just
 causes
 versioning problems when programs are built against it. All of those
 problems go away with a static library. And so much of Phobos is
 templated
 anyway that there isn't even much to share.
=20
 - Jonathan M Davis


Having played a bit with GtkD, you always want this as a shared library
for development.=20

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Apr 11 2017
next sibling parent Matthias Klumpp <mak debian.org> writes:
On Tuesday, 11 April 2017 at 14:49:03 UTC, Russel Winder wrote:

 [...]
 Having played a bit with GtkD, you always want this as a shared 
 library for development.


Yeah, GtkD is pretty massive and takes quite a large amount of 
time to compile... Redoing that for each software depending on it 
is pretty wasteful.
Apr 11 2017
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Tuesday, 11 April 2017 at 14:49:03 UTC, Russel Winder wrote:

 Having played a bit with GtkD, you always want this as a shared 
 library for development.


Why would a shared library be preferable to a static library for 
that (which might still be re-used between different projects, 
etc.)?

  — David
Apr 11 2017
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 17:02 +0000, David Nadlinger via Digitalmars-d
wrote:

 On Tuesday, 11 April 2017 at 14:49:03 UTC, Russel Winder wrote:

 Having played a bit with GtkD, you always want this as a shared=C2=A0
 library for development.


=20
 Why would a shared library be preferable to a static library for=C2=A0
 that (which might still be re-used between different projects,=C2=A0
 etc.)?
=20


I have only the data that compiling and linking a GtkD application
against a shared library is a lot shorter than against a static
library. I'll get proper data, but tomorrow =E2=80=93 that upstairs have a
water leak and our kitchen ceiling is about to fall in (again, third
time) means I am not competent just now.

Hassle me Thursday if I have presented data.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Apr 11 2017
parent David Nadlinger <code klickverbot.at> writes:
On Tuesday, 11 April 2017 at 18:13:11 UTC, Russel Winder wrote:

 I have only the data that compiling and linking a GtkD 
 application against a shared library is a lot shorter than 
 against a static library.


Sure, but that might be easily fixed, and if you really want to 
use shared libraries, you can always make Dub create/use a 
library in a local directory (and set the rpath appropriately, 
…). I don't quite see how any of this is related to the topic at 
hand (distro packaging).

I'd indeed be interested in some data on the linking times, 
though. Be sure to test with ld.gold as well, if you are not 
using it already.

  — David
Apr 11 2017
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 11 Apr 2017 14:21:57 +0000
schrieb Matthias Klumpp <mak debian.org>:


 can be used by Automake 
 (native),


Do you maintain D support for automake? I wrote some basic D support
for autoconf and libtool
(https://github.com/D-Programming-GDC/GDC/tree/master/libphobos/m4) but
no automake support except for some hacks in
https://github.com/D-Programming-GDC/GDC/blob/master/libphobos/d_rules.am

I guess I should upstream this some time.

-- Johannes
Apr 11 2017
parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 April 2017 at 23:02, Johannes Pfau via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Am Tue, 11 Apr 2017 14:21:57 +0000
 schrieb Matthias Klumpp <mak debian.org>:


 can be used by Automake
 (native),


 Do you maintain D support for automake? I wrote some basic D support
 for autoconf and libtool
 (https://github.com/D-Programming-GDC/GDC/tree/master/libphobos/m4) but
 no automake support except for some hacks in
 https://github.com/D-Programming-GDC/GDC/blob/master/libphobos/d_rules.am

 I guess I should upstream this some time.

 -- Johannes


Yeah, you should.  I upstreamed D support in DejaGNU back when I added
D tests to GDB.

https://www.mail-archive.com/dejagnu gnu.org/msg00723.html
May 01 2017
prev sibling next sibling parent reply qznc <qznc web.de> writes:
On Tuesday, 11 April 2017 at 12:56:59 UTC, Jonathan M Davis wrote:

 But if we just use dub - which _is_ the official packaging and 
 build tool - then we avoid these issues. Ideally, the compiler 
 and dub would be part of the distro, but libraries don't need 
 to be. And it sounds like that's basically how the Go and Rust 
 folks want to function as well. So, it would make sense for 
 these languages to simply not have their libraries be included 
 in distros. The build tools are plenty.


This is not compatible with Debian. Debian requires to include 
*everything*. You must be able to build every package without 
network access only from source packages.

Essentially, somebody must fix this dub issue: 
https://github.com/dlang/dub/issues/838
Apr 11 2017
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 11 Apr 2017 15:03:36 +0000
schrieb qznc <qznc web.de>:


 On Tuesday, 11 April 2017 at 12:56:59 UTC, Jonathan M Davis wrote:

 But if we just use dub - which _is_ the official packaging and 
 build tool - then we avoid these issues. Ideally, the compiler 
 and dub would be part of the distro, but libraries don't need 
 to be. And it sounds like that's basically how the Go and Rust 
 folks want to function as well. So, it would make sense for 
 these languages to simply not have their libraries be included 
 in distros. The build tools are plenty.  


 
 This is not compatible with Debian. Debian requires to include 
 *everything*. You must be able to build every package without 
 network access only from source packages.
 
 Essentially, somebody must fix this dub issue: 
 https://github.com/dlang/dub/issues/838


I created a similar issue 3 years ago:
https://github.com/dlang/dub/issues/342

A key point is how environment variables that everyone agreed
upon made "make" a flexible build tool, being able to use a
user specified compiler, installation directory, compiler
flags and more. This flexibility is relied on by Linux
packaging tools to produce binaries for a specific target.

Then there is the problem that at some point you rely on
non-Dlang libraries that only the system package manager can
provide and update. In the past this always ended in the
system package manager winning over the language tool as the
common program to update and clean the system.

Dicebot and bioinfornatics (Fedora) felt that dub was for
developers and wasn't tailored to system administration at
that time. p0nce, s-ludwig and markos (Debian) also gave input
and the net result was:

A Makefile generator for dub shall be written that exposes the
desired environment variables and can be used by existing
Linux package managers.

-- 
Marco
May 02 2017
prev sibling parent Enrico Weigelt, metux IT consult <enrico.weigelt gr13.net> writes:
On Tuesday, 11 April 2017 at 12:56:59 UTC, Jonathan M Davis wrote:


 But if we just use dub - which _is_ the official packaging and 
 build tool - then we avoid these issues. Ideally, the compiler 
 and dub would be part of the distro, but libraries don't need 
 to be.


But that would defeat a primary goal of distros: reproducable
builds, w/o depending on downloading anything from the internet.

Distros are doing that for good reasons: saves operators from
uncountable working hours (one of *the* major reason of Linux'
success).


 And it sounds like that's basically how the Go and Rust folks 
 want to function as well.


Right. Exactly why we completely banned Go from a project
just a few days ago.


--mtx
May 19 2017
prev sibling next sibling parent reply qznc <qznc web.de> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:

   1) Is there some perspective on D getting a defined ABI that 
 works with all major D compilers?
   2) What would the D community recommend on how to deal with 
 the ABI issues currently? A Linux distribution is a bunch of 
 tightly integrated software, and changing one piece in an 
 incompatible way (e.g. by building it with LDC instead of DMD) 
 will have consequences.
   3) Will DMD support more architectures in the near future? 
 How should the architecture issue be handled?


My prediction for Walters reply:

1) No. Not worth it, because templates, ctfe, etc.
3) No. Not worth it, because D frontend stuff is more important.


 Also: If you want to help out Debian's D team, feel free to 
 ping me or any other D team member (we are very short handed 
 and are only two active people right now). See 
 https://wiki.debian.org/D (caution, wiki page is very outdated, 
 last touched in 2012)


Tentative ping, but that Wiki page is not helpful. The linked svn 
repo is empty. Where and how do you work?

I guess the issue are still the same as you wrote here (except 1. 
is solved):
https://gist.github.com/ximion/fe6264481319dd94c8308b1ea4e8207a

So, mostly dub needs work, I guess.
Apr 10 2017
parent Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 14:33:34 UTC, qznc wrote:

 On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:

   1) Is there some perspective on D getting a defined ABI that 
 works with all major D compilers?
   2) What would the D community recommend on how to deal with 
 the ABI issues currently? A Linux distribution is a bunch of 
 tightly integrated software, and changing one piece in an 
 incompatible way (e.g. by building it with LDC instead of DMD) 
 will have consequences.
   3) Will DMD support more architectures in the near future? 
 How should the architecture issue be handled?


 My prediction for Walters reply:

 1) No. Not worth it, because templates, ctfe, etc.


That's short-sighted IMHO, because if the template doesn't 
change, ABI/API doesn't change. Also, some projects use D as 
better C and don't expose this functionality. It should be up to 
the project to set the level of API/ABI stability, and not to the 
compiler to make everything unstable by default.


 [...]
 Tentative ping, but that Wiki page is not helpful. The linked 
 svn repo is empty. Where and how do you work?


Yeah, the page is really poor, it was last touched in 2012. I 
made a few updates to at least link to the current Git repo.
We generally work on various Git repositories, but not all of 
them are run by the D team (e.g. libundead and libbiod as well as 
several games are things I am aware of that aren't D-team 
maintained but are part of other team's work).

One can find all stuff using D by testing the 
reverse-build-depends on the LDC and GDC compilers.


 I guess the issue are still the same as you wrote here (except 
 1. is solved):
 https://gist.github.com/ximion/fe6264481319dd94c8308b1ea4e8207a

 So, mostly dub needs work, I guess.


Yes, but since Meson is working well and Meson scripts are easy 
to write, it's not a super high priority item anymore.
As I said earlier, work as a distribution developer is pretty 
much always about reducing long-term maintenance cost, and not 
about less work short-term, which means we will gladly write 
Meson or Automake scripts to integrate software into Debian if 
there is a demand for it.

On Monday, 10 April 2017 at 15:11:01 UTC, Jack Stouffer wrote:

 On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:

   3) Will DMD support more architectures in the near future? 
 How should the architecture issue be handled?


 This can be definitively answered as "no", 
 https://issues.dlang.org/show_bug.cgi?id=15108



 *Walter Bright*:
 Doing an ARM back end wouldn't be that hard. It's much less 
 complex than x86. Most of the work would be deleting about half 
 of the x86 code generator :-)


:D - doesn't sound like a flat-out no, much more like there just 
wasn't someone doing the work yet.
Apr 10 2017
prev sibling next sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:

   3) Will DMD support more architectures in the near future? 
 How should the architecture issue be handled?


This can be definitively answered as "no", 
https://issues.dlang.org/show_bug.cgi?id=15108
Apr 10 2017
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 10 April 2017 at 15:11:01 UTC, Jack Stouffer wrote:

 On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:

   3) Will DMD support more architectures in the near future? 
 How should the architecture issue be handled?


 This can be definitively answered as "no", 
 https://issues.dlang.org/show_bug.cgi?id=15108


When templates are done I might give it a shot :)
Apr 10 2017
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2017 8:11 AM, Jack Stouffer wrote:

 On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:

   3) Will DMD support more architectures in the near future? How should the
 architecture issue be handled?


 This can be definitively answered as "no",
 https://issues.dlang.org/show_bug.cgi?id=15108


With it being open source now, this could change if someone wants to pick up
the 
flag.
Apr 10 2017
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Mon, Apr 10, 2017 at 11:40:12AM +0000, Matthias Klumpp via Digitalmars-d
wrote:
[...]

 Naturally, when the reference compiler is available in Debian, we
 would compile everything with that, as it is the development focus and
 the thing many people test with.

 We do, however, have quite a bit of bioinformatics and other D
 software in the archive where performance matters - so our users and
 the developers of that software (like BioD, potentially Mir, maybe
 even Vibe.d) will want the fastest performance and will ask us to
 compile the libraries with LDC or GDC.
 
 If we do that, we will run into the D ABI trap: Libraries compiled
 with compiler X can not be used from software compiled with D compiler
 Y. There is actually no ABI stability guarantee even between DMD
 releases.  This will make integrating D a huge pain. Recompiling the
 dependency-chain of a software from source when compiling a package
 using the "right" compiler and statically adding the code is forbidden
 by distro policy.  Having static libraries in the dependencies doesn't
 solve the issue.  Compiling each library with all D compilers is
 highly impractical and not really feasible.


This is not a hard problem to solve, IMO.  Just build the library into
two separate binaries, each with a SONAME that encodes the ABI it is
compatible with.  The resulting two .so's can either be distributed as
separate packages (for minimum bloat, if that's a concern), or as a
single package that contains both binaries (since they have different
SONAMEs this should not be a problem).  Then if you compile some
software X that depends on this library, it will pick up the correct
version of the library depending on which compiler you compiled with.



 The other issue is architecture support. In Debian we are strongly
 encouraged to support as many architectures as possible, to the point
 of having to justify why arch X is not supported.


I'd say this makes a good case for my proposed solution above.

DMD is unlikely to support other archs than amd64/ia32 in the
foreseeable future, so the justification for dmd being unavailable for
arch X would be that upstream DMD simply doesn't support it.  This,
however, should not prevent us from using gdc/ldc on those other archs,
so that we can still ship packages for those archs.  They will merely
require ldc rather than dmd. And obviously, libraries built for that
arch will only support the ldc SONAME, not the dmd one. (This may be an
argument for bundling both SONAMEs in a single package -- it gets messy
if we start shipping multiple libraries, some of which may not be
available on all archs. By shipping a single package that includes both
versions for ia32/amd64, we can simply omit the DMD-compiled version
from other archs.)

Unfortunately, I realize that this means some packages that require the
latest DMD would not be available on all archs, if they require features
unavailable in gdc/ldc.  But this problem may not be a huge one, now
that ldc is mostly up-to-date with dmd (at most 1 release behind, IIRC).
GDC may lag behind a bit more because it is unfortunately tied to GCC
releases, so we may have to forego using gdc for building newer D
packages. But we should be able to ship most D packages compiled with
both.

Furthermore, I wonder if we should standardize on ldc for most D
software instead of dmd, unless that software absolutely depends on
features only available on dmd.  My reasons are:

- While dmd compiles very fast, it consistently (IME) produces code that
  runs about 20-30% slower than code produced by gdc (and I presume ldc
  as well). Since we're talking about building Debian packages on
  Debian's buildd's, which are background batch processes, compilation
  speed is no big deal, but the performance of the executable *is* a big
  deal. The last thing we want is to give a poor impression of D by
  shipping official Debian packages that have subpar performance.

- DMD is unlikely to target other archs than ia32/amd64 in the
  foreseeable future, AFAIK, unless the recent relicensing triumph of
  dmd's backend makes this future more likely.  There will definitely be
  resistance among DDs because of lack of support for other archs. LDC,
  in contrast, supports all of Debian's supported archs.

- LDC is already available in Debian, meaning that we can already start
  adding D packages built with ldc without needing to work through the
  red tape involved in adding a new compiler to the archive.

- The only case where I can see we'd want to compile with dmd is if the
  target software uses features that are only available on dmd.  Since
  LDC's release cycle is now very close to DMD's, I don't foresee there
  will be too many D projects out there that can't be compiled with LDC.
  So there should really only be a small number of packages that we'd
  have to fallback to dmd, which will also be limited in supported archs
  -- meaning that it will probably be challenging to get it into the
  Debian archive in the first place.

- If we standardize on using ldc to compile Debian packages, we can
  sidestep the ABI compatibility issue (though the SONAME hack will
  probably still be necessary in order to provide users with libraries
  that are usable with dmd).

Because of this, my first inclination is to standardize on ldc rather
than dmd, where building Debian packages are concerned.

Of course, this in no way diminishes the priority in *distributing* dmd
in the Debian archive as -dev packages. Just don't build other Debian
packages with it. :-D


[...]

 Also: If you want to help out Debian's D team, feel free to ping me or
 any other D team member (we are very short handed and are only two
 active people right now). See https://wiki.debian.org/D (caution, wiki
 page is very outdated, last touched in 2012)


I'd like to be involved, since I *am* technically a Debian developer
(though haven't been very active for the past long while -- I do try to
at least keep the meager few packages I maintain up-to-date, though).


T

-- 
"Holy war is an oxymoron." -- Lazarus Long
Apr 10 2017
parent reply Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 17:29:04 UTC, H. S. Teoh wrote:

 On Mon, Apr 10, 2017 at 11:40:12AM +0000, Matthias Klumpp via 
 Digitalmars-d wrote: [...]

 [...]
 If we do that, we will run into the D ABI trap: Libraries 
 compiled with compiler X can not be used from software 
 compiled with D compiler Y. There is actually no ABI stability 
 guarantee even between DMD releases.  This will make 
 integrating D a huge pain. Recompiling the dependency-chain of 
 a software from source when compiling a package using the 
 "right" compiler and statically adding the code is forbidden 
 by distro policy.  Having static libraries in the dependencies 
 doesn't solve the issue.  Compiling each library with all D 
 compilers is highly impractical and not really feasible.


 This is not a hard problem to solve, IMO.  Just build the 
 library into two separate binaries, each with a SONAME that 
 encodes the ABI it is compatible with.


This would require to hack each and every build system to support 
this *and* if there is a pkgconfig file for the shared library, 
to change all depending software to check for multiple library 
names which is a bit crazy...


 The resulting two .so's can either be distributed as separate 
 packages (for minimum bloat, if that's a concern), or as a 
 single package that contains both binaries (since they have 
 different SONAMEs this should not be a problem).


Since one library == one package in Debian, it would have to be 
multiple packages, otherwise we would need to override Lintian 
errors/warnings, which is always a bad idea.


 Then if you compile some software X that depends on this 
 library, it will pick up the correct version of the library 
 depending on which compiler you compiled with.


Unfortunately not without special support in the software's build 
system :-/


 [...]
 DMD is unlikely to support other archs than amd64/ia32 in the 
 foreseeable future, so the justification for dmd being 
 unavailable for arch X would be that upstream DMD simply 
 doesn't support it.  This, however, should not prevent us from 
 using gdc/ldc on those other archs, so that we can still ship 
 packages for those archs.  They will merely require ldc rather 
 than dmd. And obviously, libraries built for that arch will 
 only support the ldc SONAME, not the dmd one. (This may be an 
 argument for bundling both SONAMEs in a single package -- it 
 gets messy if we start shipping multiple libraries, some of 
 which may not be available on all archs. By shipping a single 
 package that includes both versions for ia32/amd64, we can 
 simply omit the DMD-compiled version from other archs.)


Conditional build-dependencies are a bit annoying, but with a 
metapackage "d-compiler" or similar, using different D compilers 
on different architectures would definitely be possible.


 Unfortunately, I realize that this means some packages that 
 require the latest DMD would not be available on all archs, if 
 they require features unavailable in gdc/ldc.  But this problem 
 may not be a huge one, now that ldc is mostly up-to-date with 
 dmd (at most 1 release behind, IIRC). GDC may lag behind a bit 
 more because it is unfortunately tied to GCC releases, so we 
 may have to forego using gdc for building newer D packages. But 
 we should be able to ship most D packages compiled with both.


Compiling with multiple compilers is a really big effort with 
rather questionable gain, IMO.
But as far as LDCs compatibility with other D projects goes: That 
is really good, the only reason you sometimes can't compile some 
random D code with LDC might be bugs, but not old standard 
libraries.


 Furthermore, I wonder if we should standardize on ldc for most 
 D software instead of dmd, unless that software absolutely 
 depends on features only available on dmd.  My reasons are:

 - While dmd compiles very fast, it consistently (IME) produces 
 code that
   runs about 20-30% slower than code produced by gdc (and I 
 presume ldc
   as well). Since we're talking about building Debian packages 
 on
   Debian's buildd's, which are background batch processes, 
 compilation
   speed is no big deal, but the performance of the executable 
 *is* a big
   deal. The last thing we want is to give a poor impression of 
 D by
   shipping official Debian packages that have subpar 
 performance.

 - DMD is unlikely to target other archs than ia32/amd64 in the
   foreseeable future, AFAIK, unless the recent relicensing 
 triumph of
   dmd's backend makes this future more likely.  There will 
 definitely be
   resistance among DDs because of lack of support for other 
 archs. LDC,
   in contrast, supports all of Debian's supported archs.

 - LDC is already available in Debian, meaning that we can 
 already start
   adding D packages built with ldc without needing to work 
 through the
   red tape involved in adding a new compiler to the archive.


I agree with all of that, I think sticking with LDC might indeed 
be the least painful thing to do...


 - The only case where I can see we'd want to compile with dmd 
 is if the
   target software uses features that are only available on dmd.


Oddly, it usually is the other way round :P - some projects 
explicitly want LDC or GDC.


 [...]
 Of course, this in no way diminishes the priority in 
 *distributing* dmd in the Debian archive as -dev packages. Just 
 don't build other Debian packages with it. :-D


Indeed. It needs license review (urgh... ^^), but when it's 
through that there is no reason not to have it in Debian main and 
Ubuntu.


 [...]

 Also: If you want to help out Debian's D team, feel free to 
 ping me or any other D team member (we are very short handed 
 and are only two active people right now). See 
 https://wiki.debian.org/D (caution, wiki page is very 
 outdated, last touched in 2012)


 I'd like to be involved, since I *am* technically a Debian 
 developer (though haven't been very active for the past long 
 while -- I do try to at least keep the meager few packages I 
 maintain up-to-date, though).


It's unlikely that I will maintain the dmd package (I might 
co-maintain it or sponsor uploads though - reason is simply lack 
of time and -ETOOMANYPROJECTS), so any help there is highly 
appreciated and I'd help with it whenever I can (I just can't 
make long-term commitments).
If DMD is uploaded as part of the D team, spreading the 
maintenance cost will be rather simple too :-)
Apr 10 2017
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Mon, Apr 10, 2017 at 06:12:26PM +0000, Matthias Klumpp via Digitalmars-d
wrote:

 On Monday, 10 April 2017 at 17:29:04 UTC, H. S. Teoh wrote:

 On Mon, Apr 10, 2017 at 11:40:12AM +0000, Matthias Klumpp via
 Digitalmars-d wrote: [...]

 [...]
 If we do that, we will run into the D ABI trap: Libraries compiled
 with compiler X can not be used from software compiled with D
 compiler Y. There is actually no ABI stability guarantee even
 between DMD releases.  This will make integrating D a huge pain.
 Recompiling the dependency-chain of a software from source when
 compiling a package using the "right" compiler and statically
 adding the code is forbidden by distro policy.  Having static
 libraries in the dependencies doesn't solve the issue.  Compiling
 each library with all D compilers is highly impractical and not
 really feasible.


 
 This is not a hard problem to solve, IMO.  Just build the library
 into two separate binaries, each with a SONAME that encodes the ABI
 it is compatible with.


 
 This would require to hack each and every build system to support this
 *and* if there is a pkgconfig file for the shared library, to change
 all depending software to check for multiple library names which is a
 bit crazy...


Hmm.  I guess there's no easy way to make dmd/ldc emit dependencies with
modified SONAMEs?  So yeah, you're right, every software that depends on
said libraries would have to explicitly depend on a different SONAME
depending on what they were built with.  OK, crazy idea, nevermind. :-(


[...]

 - The only case where I can see we'd want to compile with dmd is if
   the target software uses features that are only available on dmd.


 
 Oddly, it usually is the other way round :P - some projects explicitly
 want LDC or GDC.


Well, if they explicitly want ldc/gdc, then we'll just have to do with
that.  But I think ldc should be able to compile anything that compiles
with gdc, so we might still be able to standardize on ldc by default.


[...]

 Of course, this in no way diminishes the priority in *distributing*
 dmd in the Debian archive as -dev packages. Just don't build other
 Debian packages with it. :-D


 
 Indeed. It needs license review (urgh... ^^), but when it's through
 that there is no reason not to have it in Debian main and Ubuntu.


One issue, though: if we standardize on compiling Debian packages with
ldc, then what do we do with libraries that are not ABI-compatible with
dmd?  Since users would expect that if they need libfoo, they'd just
`apt-get install libfoo-dev` and then they should be able to just run`
dmd -L-lfoo` and it should all magically "just work".


[...]

 I'd like to be involved, since I *am* technically a Debian developer
 (though haven't been very active for the past long while -- I do try
 to at least keep the meager few packages I maintain up-to-date,
 though).


 
 It's unlikely that I will maintain the dmd package (I might
 co-maintain it or sponsor uploads though - reason is simply lack of
 time and -ETOOMANYPROJECTS), so any help there is highly appreciated
 and I'd help with it whenever I can (I just can't make long-term
 commitments).  If DMD is uploaded as part of the D team, spreading the
 maintenance cost will be rather simple too :-)


Yeah, the reason I haven't been very active in Debian is also due to
ENOTIME, so I doubt I'd be able to keep up with maintaining dmd.  But if
it's spread among more D team members, it could be more workable.


T

-- 
Heuristics are bug-ridden by definition. If they didn't have bugs, they'd be
algorithms.
Apr 10 2017
next sibling parent reply Wyatt <wyatt.epp gmail.com> writes:
On Monday, 10 April 2017 at 18:46:31 UTC, H. S. Teoh wrote:

 Hmm.  I guess there's no easy way to make dmd/ldc emit 
 dependencies with modified SONAMEs?  So yeah, you're right, 
 every software that depends on said libraries would have to 
 explicitly depend on a different SONAME depending on what they 
 were built with.  OK, crazy idea, nevermind. :-(


Doesn't sounds that crazy; you already do it with GCC versions, 
right?  (Debian _does_ have something like that, right?  Where 
you can pick your C compiler.)

-Wyatt
Apr 10 2017
parent Matthias Klumpp <mak debian.org> writes:
On Monday, 10 April 2017 at 20:11:28 UTC, Wyatt wrote:

 On Monday, 10 April 2017 at 18:46:31 UTC, H. S. Teoh wrote:

 Hmm.  I guess there's no easy way to make dmd/ldc emit 
 dependencies with modified SONAMEs?  So yeah, you're right, 
 every software that depends on said libraries would have to 
 explicitly depend on a different SONAME depending on what they 
 were built with.  OK, crazy idea, nevermind. :-(


 Doesn't sounds that crazy; you already do it with GCC versions, 
 right?  (Debian _does_ have something like that, right?  Where 
 you can pick your C compiler.)


Yes, that's why all packages need to honor the CFLAGS/CC env var 
somehow or get the default C compiler from dpkg at build time, so 
we can easily apply new C flags globally, and projects to build 
the distro with Clang work.
We do never change the SONAME of anything, however. We do track 
symbols and might change the SOVERSION occasionally if breakage 
is found (making 3 -> 3a etc.), but the soname isn't changed.
The good thing is that Clang and GCC are (with few exceptions) 
very compatible.
Apr 11 2017
prev sibling parent reply Matthias Klumpp <mak debian.org> writes:
I apologize in advance for the large amount of mail that will 
likely follow, but I want to address all comments.

On Monday, 10 April 2017 at 18:46:31 UTC, H. S. Teoh wrote:

 [...]
 One issue, though: if we standardize on compiling Debian 
 packages with ldc, then what do we do with libraries that are 
 not ABI-compatible with dmd?  Since users would expect that if 
 they need libfoo, they'd just `apt-get install libfoo-dev` and 
 then they should be able to just run` dmd -L-lfoo` and it 
 should all magically "just work".


That's the problem I would like to see addressed (but given 
Walter's comment, it won't be feasible to resolve it in the near 
future).
We could simply do away with "don't use distro packages for your 
D programming", at least that's what Go recommends.
Apr 11 2017
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-11 at 11:55 +0000, Matthias Klumpp via Digitalmars-d
wrote:

=20


[=E2=80=A6]

 That's the problem I would like to see addressed (but given=C2=A0
 Walter's comment, it won't be feasible to resolve it in the near=C2=A0
 future).
 We could simply do away with "don't use distro packages for your=C2=A0
 D programming", at least that's what Go recommends.


The Go community advice against platform packages is principally
because they want up-to-date packages on old distributions.

Most programming languages are going this route, used programming
language tools and libraries are decreasingly packaged by the OS. Given
things like Anaconda and PyPI, all the Fedora and Debian Python
packaging is increasingly wasted =E2=80=93 except for those packages used b=
y
the distribution itself. So I can see Fedora and Debian packaging
Python and some bits and pieces, but otherwise it's not needed. Same is
true for Ruby, Go, Groovy, Java, etc.=20

Debian and Fedora really do need to reassess their packaging strategy
with respect to software development. Far too much effort is going into
packaging when rolling release is far better handled by language
specific systems. Go is going this route, Python likewise.

I suggest getting the compilers onto the distributions as a marketing
thing not as a real use thing. For me the way Rust works by having it's
own shell-based installer is the right way of installing languages.
SDKMAN! is an even more extensive system for all JVM-related systems.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Apr 11 2017
parent Enrico Weigelt, metux IT consult <enrico.weigelt gr13.net> writes:
On Tuesday, 11 April 2017 at 12:36:41 UTC, Russel Winder wrote:


 Most programming languages are going this route, used 
 programming language tools and libraries are decreasingly 
 packaged by the OS.


Exactly the reason why we banned Go quite recently.
Just far too expensive, especially w/ crosscompile.


 Given things like Anaconda and PyPI, all the Fedora and Debian 
 Python packaging is increasingly wasted – except for those 
 packages used by the distribution itself.


Not at all. These folks just completely ignore the whole purpose
of binary distros. And I regularily see that at operating costs.
One of the major reasons why Windows operating is so expensive is
the lack of proper package management and distro infrastructures.

Few years ago, I've introduced apt-based package management for
Zimbra extensions (for some solution provider who also develops
lots of own extensions and customizations). Cut down deployment
costs to less than 10%.


 So I can see Fedora and Debian packaging Python and some bits
 and pieces, but otherwise it's not needed. Same is true for 
 Ruby, Go, Groovy, Java, etc.


With ruby, just given up completely - the community has shown
itself really hostile against distros, as well as any kind of
decent release engineering, which makes TOC very very expensive.

Python and Java are manageable (thanks to the hard work of distro
folks). No idea about Groovy.

Go is horrible - banned it from all projects.


 Debian and Fedora really do need to reassess their packaging 
 strategy with respect to software development.


Software developers should reassess their methodologies with
respect to distros and operating - these are the folks who make
the software actually usable for the arbitrary user, and they
get all the basting if something goes wrong again.


 Far too much effort is going into packaging when rolling release
 is far better handled by language specific systems.


Far too much effort is going into proprietary / language specific
homebrewn tools, by people who believe their own little subspace
bubble was the whole universe and have no idea what matters in
professional operating.

Much of my professional work is being HoD - the interface between
development, operating and PMO - or training development teams.
The numbers on my table tell a clear story.


 I suggest getting the compilers onto the distributions as a 
 marketing thing not as a real use thing.


Why should Distros like Debian do any marketing ?
Selling stuff isn't any of their objectives.


 For me the way Rust works by having it's own shell-based 
 installer
 is the right way of installing languages.


For me (as HoD, release manager, auditor, embedded engineer, 
software
architect, etc) such things are an complete showstopper - 
reproducable
build and deployment is a hard requirement - anything else is just
dangerous and expensive.


--mtx
May 19 2017
prev sibling parent David Nadlinger <code klickverbot.at> writes:
On Monday, 10 April 2017 at 11:40:12 UTC, Matthias Klumpp wrote:

 Recompiling the dependency-chain of a software from source when 
 compiling a package using the "right" compiler and statically 
 adding the code is forbidden by distro policy.


Yet, from what I could find after a brief search, the Rust 
packaging guys are planning to do exactly that: 
https://wiki.debian.org/Teams/RustPackaging/Policy#Library_package_structure

You might want to get in touch with them, particularly regarding


 […] require policy exceptions, which is something we won't do 
 […]


before either side wastes more time on this than necessary.

  — David
Apr 10 2017