www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Practical Problems with distribution D projects

reply Assaf Gordon <agordon wi.mit.edu> writes:
Hello,

I working on a small program, which I'd like to make publicly available and
easy to distribute,install and use.

I'm trying to figure out the best way to distribute it.
Writing D code is a lot of fun, but it looks like preparing and packaging it is
not so much...

So far, I'm stuck at:

1.
Packaging for source-based distribution (ie a "tarball"):
I couldn't find a working autotools template (only few old discussions),
and so far it seems the best way is writing my own makefile (which I have).

My concern is that the parameters are so different between dmd/gdc/ldc that I
basically force the users to choose one.

I'm also not sure whether it will work on windows or not (I'm developing on
linux). For windows users, would a custom,hand-written make file just work?

Although "rdmd" is terrific, I would really prefer not to require it (since it
is 'dmd' specific and not free in the FOSS sense).


2.
How complicated it for potential users of the program to install a D compiler?

Based on my humble experience:
dmd: fine, as long as the downloadable binary version works. I tested only on
multiple debians and ubuntus. But it's not free, and it requires root
privileges.
gdc: only source code available, compiling is a bit tricky. After compilation,
it works.
ldc: the pre-compiled version doesn't work on old Ubuntu (10.4) or new Debian
(7). The source code version is messy to compile and I stopped after couple of
tries.

So, it seems, that if a random person (not unix expert) wants to compiler my
program, he/she will not have an easy time doing so.


3.
Distributing Binaries:
Since I already assume source distribution is not friendly, I consider also
making the pre-compiled binaries available for download.
But it looks like it's only possible to compile dynamically-linked binaries.

The only leads I've found are two old bugs:
http://d.puremagic.com/issues/show_bug.cgi?id=6952
http://d.puremagic.com/issues/show_bug.cgi?id=4376

And I couldn't find any better solution (on dlang.org or google).
---

All the above, makes me somewhat reconsider whether D is an optimal choice:
even if my program is useful in theory, it's of no use if very few people can
actually run it.

Are there any suggestions how to deal with these issues ?

Thanks,
  -gordon
Feb 24 2014
next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Tuesday, 25 February 2014 at 01:21:03 UTC, Assaf Gordon wrote:

 1.
 Packaging for source-based distribution (ie a "tarball"):
 I couldn't find a working autotools template (only few old 
 discussions),
 and so far it seems the best way is writing my own makefile 
 (which I have).

 My concern is that the parameters are so different between 
 dmd/gdc/ldc that I basically force the users to choose one.


GDC and LDC have gdmd/ldmd respectively, which wrap DMD-style 
command line arguments over the GCC/Clang-style arguments that 
`gdc` and `ldc` accept.


 I'm also not sure whether it will work on windows or not (I'm 
 developing on linux). For windows users, would a 
 custom,hand-written make file just work?


That depends almost entirely on the shell commands used in the 
target recipes. Also, it's easy to create reliance on a certain 
flavour of `make`, such as by using GNU-make-specific features; 
it's not a big problem though because GNU make is relatively easy 
to get on Windows through MinGW.


 Although "rdmd" is terrific, I would really prefer not to 
 require it (since it is 'dmd' specific and not free in the FOSS 
 sense).


`rdmd` itself should be free software, and supports `gdmd` and 
`ldmd` through the --compiler flag.

DMD itself is indeed not entirely free software.


 2.
 How complicated it for potential users of the program to 
 install a D compiler?

 Based on my humble experience:
 dmd: fine, as long as the downloadable binary version works. I 
 tested only on multiple debians and ubuntus. But it's not free, 
 and it requires root privileges.
 gdc: only source code available, compiling is a bit tricky. 
 After compilation, it works.
 ldc: the pre-compiled version doesn't work on old Ubuntu (10.4) 
 or new Debian (7). The source code version is messy to compile 
 and I stopped after couple of tries.

 So, it seems, that if a random person (not unix expert) wants 
 to compiler my program, he/she will not have an easy time doing 
 so.


Maybe we should have a generic Linux binary package that 
statically links all dependencies, which would then be usable on 
a wide variety of Linux systems without requiring root privileges.

But are there really no packages of `gdc` and `ldc` in various 
package repositories? I haven't used D on Linux in a while, but I 
find that hard to believe.


 3.
 Distributing Binaries:
 Since I already assume source distribution is not friendly, I 
 consider also making the pre-compiled binaries available for 
 download.
 But it looks like it's only possible to compile 
 dynamically-linked binaries.


I'm pretty sure the default still uses static linking, even with 
recent Linux releases? Either way, I'm sure there's a way to get 
the old behaviour. Dynamic linking of the runtime is a relatively 
recent ability.
Feb 24 2014
prev sibling next sibling parent "FreeSlave" <freeslave93 gmail.com> writes:
You can use hack in Makefile to find out whether the current 
platform is Windows or not. Check for SystemRoot environment 
variable in this way:

ifdef SystemRoot
	SYSTEM=Windows
else ifdef SYSTEMROOT
	SYSTEM=Windows
else
	SYSTEM=Other
endif

In non-Windows case use Unix uname command to determine current 
platform (Linux, Darwin, etc).

If you don't want to force user to install MSys or Cygwin, use 
Windows-shell commands instead Unix ones. Example:

ifeq ($(SYSTEM), Windows)
	RM=cmd //C del
	CP=cmd //C copy
else
	RM=rm -f
	CP=cp
endif

So you can achieve some flexibility here.
Sometimes developers write separate makefiles named windows.mak 
and posix.mak, but those ones still can share common parts via 
include directive.


Also you can write small build-system for your project in D 
itself. Derelict uses this approach. It also allows to ensure 
that D compiler was installed and works properly.
Feb 24 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 25 February 2014 at 01:21:03 UTC, Assaf Gordon wrote:

 dmd: fine, as long as the downloadable binary version works. I 
 tested only on multiple debians and ubuntus. But it's not free, 
 and it requires root privileges.


dmd does not require root. The install instructions about copying 
stuff into /etc and such are ridiculously overcomplicated - dmd 
actually just works* if you run it straight out of the zip as any 
regular user in any location.

* if the binary works on your distro, sometimes there's a version 
mismatch forcing you to build it yourself. That's not a hard 
thing to do either, the source is included and you can just make 
it as a regular user, but an extra step doesn't do 'just works'.


 But it looks like it's only possible to compile 
 dynamically-linked binaries.


This is more of a Linux problem than a D one. But most D programs 
will work normally, dynamically linking the C library *usually* 
works and that's all a D app really needs.

If you want more binary compatibility btw I'd say stick to 
building Windows exes, they almost always just work, even on 
Linux with wine.
Feb 24 2014
prev sibling next sibling parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2014-02-25 00:50:55 +0000, Assaf Gordon said:


 Hello,
 
 I working on a small program, which I'd like to make publicly available 
 and easy to distribute,install and use.
 
 I'm trying to figure out the best way to distribute it.
 Writing D code is a lot of fun, but it looks like preparing and 
 packaging it is not so much...
 
 So far, I'm stuck at:
 
 1.
 Packaging for source-based distribution (ie a "tarball"):
 I couldn't find a working autotools template (only few old discussions),
 and so far it seems the best way is writing my own makefile (which I have).
 
 My concern is that the parameters are so different between dmd/gdc/ldc 
 that I basically force the users to choose one.
 
 I'm also not sure whether it will work on windows or not (I'm 
 developing on linux). For windows users, would a custom,hand-written 
 make file just work?
 
 Although "rdmd" is terrific, I would really prefer not to require it 
 (since it is 'dmd' specific and not free in the FOSS sense).
 
 
 2.
 How complicated it for potential users of the program to install a D compiler?
 
 Based on my humble experience:
 dmd: fine, as long as the downloadable binary version works. I tested 
 only on multiple debians and ubuntus. But it's not free, and it 
 requires root privileges.
 gdc: only source code available, compiling is a bit tricky. After 
 compilation, it works.
 ldc: the pre-compiled version doesn't work on old Ubuntu (10.4) or new 
 Debian (7). The source code version is messy to compile and I stopped 
 after couple of tries.
 
 So, it seems, that if a random person (not unix expert) wants to 
 compiler my program, he/she will not have an easy time doing so.
 
 
 3.
 Distributing Binaries:
 Since I already assume source distribution is not friendly, I consider 
 also making the pre-compiled binaries available for download.
 But it looks like it's only possible to compile dynamically-linked binaries.
 
 The only leads I've found are two old bugs:
 http://d.puremagic.com/issues/show_bug.cgi?id=6952
 http://d.puremagic.com/issues/show_bug.cgi?id=4376
 
 And I couldn't find any better solution (on dlang.org or google).
 ---
 
 All the above, makes me somewhat reconsider whether D is an optimal 
 choice: even if my program is useful in theory, it's of no use if very 
 few people can actually run it.
 
 Are there any suggestions how to deal with these issues ?
 
 Thanks,
   -gordon



Have you looked at having dub be your autotool?
Feb 24 2014
next sibling parent Artem Tarasov <lomereiter gmail.com> writes:
On Tue, Feb 25, 2014 at 7:16 AM, Shammah Chancellor <anonymous coward.com>wrote:


 Have you looked at having dub be your autotool?


The complaint is about difficulties with installation of a compiler and
rdmd, and you suggest to bring in yet another dependency?

My experience is the same as described, I keep a VM on Amazon EC2 with
CentOS 5, which contains installed LDC2, just so that the compiled binaries
run on any Linux x86_64 machine no matter how old glibc version is.
Feb 24 2014
prev sibling parent reply Assaf Gordon <agordon wi.mit.edu> writes:
On 02/25/2014 12:09 AM, Artem Tarasov wrote:

 On Tue, Feb 25, 2014 at 7:16 AM, Shammah Chancellor
 <anonymous coward.com <mailto:anonymous coward.com>> wrote:

 Have you looked at having dub be your autotool?


 The complaint is about difficulties with installation of a compiler
 and rdmd, and you suggest to bring in yet another dependency?

 My experience is the same as described, I keep a VM on Amazon EC2
 with CentOS 5, which contains installed LDC2, just so that the
 compiled binaries run on any Linux x86_64 machine no matter how old
 glibc version is.


Thanks!
I was beginning to think I'm alone in this problem :)

I'm getting the impression that D is either windows-focused (which is a mostly
homogenous system everywhere and easy to work with), or
unix-experts/developers-focused (where people feel naturally comfortable with
downloading and compiling and fidgeting with programs).

-----

To address some of the other responses:

On 02/24/2014 09:09 PM, Adam D. Ruppe wrote:> On Tuesday, 25 February 2014 at
01:21:03 UTC, Assaf Gordon wrote:

 dmd does not require root. The install instructions about copying
 stuff into /etc and such are ridiculously overcomplicated - dmd
 actually just works* if you run it straight out of the zip as any
 regular user in any location.


 From a non-expert unix POV -
Either you download and install the package (deb/rpm) which is easy but
requires root,
or you download and compile the sources from the source zip - which is
complicated.
We can argue about what "complicated" means, obviously the members of this
forums are mostly experts, so they don't think that compiling from source is
tricky.
But if you download the zip, and try to look at it from a novice's POV:
The README.txt is not helpful.
There's no "configure" or "CMakefile" or "Makefile" - which are the most common
ways to "just make it work" for open source projects.
Going into "src", there's no Makefile, and the "readme.txt" is again not
helpful (talks about the license, not about the installation).
If you're have some experience, you can guess that "posix.mak" is the relevant
file, and indeed "make -f posix.mak" builds dmd correctly.
Then you have to repeat the same for "./src/dmd", "./src/druntime" and
"./src/phobos" - and then what?
update $PATH, $LD_LIBRARY_PATH ? would that be enough ?
All these might be easy for you - but not for a casual Linux user (which is my
intended target audience).

-----

On 02/25/2014 12:48 AM, Jesse Phillips wrote:

 If you want to distribute to desktop users, binary packages is the
 way to go. The issues with static linking are related to C
 libraries. If you develop in Linux you generally don't statically
 link.


You "generally don't statically link" because you can:
1. Readily and easily compile from source code on the local machine, or
2. Have binary distribution that perfectly matches your system (in the case of
debs/rpms from an official repository).

With D - I have none of these.
And when distribution just an executable to run on an unknown linux machine,
the best way is a statically linked binary.


 All your D code is statically linked, work has been done to support
 dynamic linking with Phobos, but for now that isn't default.


But I'm not asking for dynamic linking... I'm asking for static linking that
works...

----

On 02/25/2014 08:13 AM, Dicebot wrote:

 For building dub and/or rdmd are de-facto standard for any D
 developers. You should just start from source tarball and proceed
 from there to finding maintainers in specific distributives, possibly
 being the one for distro or two. Then conform packaging rules there.


My target audience is not D developers - it is unix casual users (not even
developers).
I want to make it the easiest for them to use my program.
If it's requires more than the standard "configure && make && make install" (or
the cmake equivalent), then many of them will not do it.


 Single Linux-wide binary distribution is mostly a myth and dmd shows
 it by its own.


Not sure what the "myth" is - but a statically-linked binaries work well (if
your program is self-contained and properly compiled), and there are many
examples of it.

----

On the upside: I now see that "dub" provides a good Debian repository, and a
HomeBrew formula (and by proxy, a LinuxBrew formula) - so perhaps that is the
way to go.

Still, if I may suggest:
There have been recent threads in the mailing list about "the future of D" and
creating "Killer Apps" and making D more popular - From my (biased) POV, one
big step would be to make D easily useable on more platforms - so the casual
(non-windows) users could also experiment with it, and developers would have
easier time to distribute software written in D.

Also, as long as DMD is not free, providing easy way to install LDC/GDC is
highly desired.

Thanks for reading so far,
   -gordon
Feb 25 2014
next sibling parent Mike Parker <aldacron gmail.com> writes:
On 2/25/2014 11:52 PM, Assaf Gordon wrote:

 On 02/25/2014 12:09 AM, Artem Tarasov wrote:



 On 02/24/2014 09:09 PM, Adam D. Ruppe wrote:> On Tuesday, 25 February
 2014 at 01:21:03 UTC, Assaf Gordon wrote:

 dmd does not require root. The install instructions about copying
 stuff into /etc and such are ridiculously overcomplicated - dmd
 actually just works* if you run it straight out of the zip as any
 regular user in any location.


  From a non-expert unix POV -
 Either you download and install the package (deb/rpm) which is easy but
 requires root,
 or you download and compile the sources from the source zip - which is
 complicated.


You don't need to compile anything. There are precompiled binaries for 
32- and 64-bit Linux in the zip. I'm no Linux expert (far, far from it). 
I'm not even what you would call a casual user. But when I do need to 
boot into Linux for one thing or another (which is rare), I unzip the 
latest binaries into my user directory and away I go. The only thing 
I've ever configured is putting the appropriate bin directory on the 
path, but that's it.
Feb 25 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 25 February 2014 at 14:51:48 UTC, Assaf Gordon wrote:

 My target audience is not D developers - it is unix casual 
 users (not even developers).
 I want to make it the easiest for them to use my program.
 If it's requires more than the standard "configure && make && 
 make install" (or the cmake equivalent), then many of them will 
 not do it.


Then push it into distro repositories. You are NOT helping by 
recommending them to do "make install", quite the contrary, 
harming whole ecosystem. Casual users should not be even aware 
that "make install" exists. Being one of package maintainers 
myself I really hate this kind of "helping" attitude, it causes 
only trouble.


 Single Linux-wide binary distribution is mostly a myth and dmd 
 shows
 it by its own.


 Not sure what the "myth" is - but a statically-linked binaries 
 work well (if your program is self-contained and properly 
 compiled), and there are many examples of it.


Except you never have truly statically linked binaries, there is 
always at least glibc. You can do it for small set of popular 
distributives, especially when conforming to 
http://en.wikipedia.org/wiki/Linux_Standard_Base but it simply 
does not work in general.

Please stop distributing software for Linux with Windows attitude.
Feb 25 2014
next sibling parent reply Assaf Gordon <agordon wi.mit.edu> writes:
On 02/25/2014 10:06 AM, Dicebot wrote:

 On Tuesday, 25 February 2014 at 14:51:48 UTC, Assaf Gordon wrote:

 My target audience is not D developers - it is unix casual users
 (not even developers). I want to make it the easiest for them to
 use my program. If it's requires more than the standard "configure
 && make && make install" (or the cmake equivalent), then many of
 them will not do it.


 Then push it into distro repositories. You are NOT helping by
 recommending them to do "make install", quite the contrary, harming
 whole ecosystem. Casual users should not be even aware that "make
 install" exists. Being one of package maintainers myself I really
 hate this kind of "helping" attitude, it causes only trouble.


I'm sorry it makes you upset - I can see how frustrating it is for a package
maintainer.

But I think "harming the ecosystem" is a bit of a hyperbole - in an ideal world
where Linux Distros contain every piece of software (and up to date version),
there is indeed very little need to install from source.

But there are several real-world constraints, at least in the environment I
work in:

1. Software needs to run on variety of servers and distributions, some are very
old (e.g. CentOS 5.4 or Ubuntu 10.04).
Upgrading is not an option for these environments.

2. Software needs to be installed without root - to a user directory (i.e.
using "configure --prefix" or similar).

3. Software is updated too frequently, or is too new, or (sadly) too
experimental (code-wise) for inclusion in a stable distribution package - but
it still needs to be used.

4. It takes time for a software to be properly packaged, and users want to
install it now. Also - it will not be available in the "stable" branch of the
distribution.

In addition, some distribution have extra requirements.
For example, Debian frowns upon compiling with static libraries when a shared
one is available.

If there is already a package in official Debian which is written in D, it
would be very helpful to see an example of the packaging process.


 Single Linux-wide binary distribution is mostly a myth and dmd
 shows it by its own.


 Not sure what the "myth" is - but a statically-linked binaries work
 well (if your program is self-contained and properly compiled), and
 there are many examples of it.


 Except you never have truly statically linked binaries, there is
 always at least glibc.


I'm not sure I understand this statement.
If I link with "gcc -static" and my program is properly built, the resulting
binary does not require any library, not even glibc.
It uses only Linux ABI syscalls.
Unless I'm mistaken?


 Please stop distributing software for Linux with Windows attitude.


Not sure what the "windows attitude" is, since I haven't developed software for
windows in 12 years...

My first choice would be to have a project in D which is easily compilable from
source code.
Either with "make" or "dub" or "configure" or anything else.
Once I have that, packaging (for a linux distribution) is straight forward, and
so is adding it to HomeBrew/LinuxBrew.

At the moment, I don't think it's possible (in the level of "easiness"
comparable with C/autotools-based projects).

The alternative (for linux) is distributing a binary program, and for that, a
static-binary is much more portable than a dynamic one.

Having GDC and LDC in Debian is an excellent start - I was not aware of them,
and it is certainly the right direction.
But those are still not available in the "stable" branches, and so, not easily
usable.
In the future this will definitely change.

-Gordon
Feb 25 2014
next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 25 February 2014 at 16:04:56 UTC, Assaf Gordon wrote:

 Not sure what the "windows attitude" is, since I haven't 
 developed software for windows in 12 years...


The Windows attitude is expecting things to actually work for the 
end user. It rarely actually works that way on linux.
Feb 25 2014
next sibling parent Assaf Gordon <agordon wi.mit.edu> writes:
On 02/25/2014 11:17 AM, Adam D. Ruppe wrote:

 On Tuesday, 25 February 2014 at 16:04:56 UTC, Assaf Gordon wrote:

 Not sure what the "windows attitude" is, since I haven't developed software
for windows in 12 years...


 The Windows attitude is expecting things to actually work for the end user. It
rarely actually works that way on linux.


Side-stepping the flame-war potential of this statement :) ,

I agree that Windows is homogenous enough, and have so many "built-in" runtime
DLLs that you can expect all systems to be almost identical in terms of what's
available. It makes distribution very easy for users (and developers).

It takes more effort to build portable software, and in windows you rarely have
to worry about it.

Once you do build a portable software properly, it "works" for the end user as
well.
Feb 25 2014
prev sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 25 February 2014 at 16:17:13 UTC, Adam D. Ruppe wrote:

 On Tuesday, 25 February 2014 at 16:04:56 UTC, Assaf Gordon 
 wrote:

 Not sure what the "windows attitude" is, since I haven't 
 developed software for windows in 12 years...


 The Windows attitude is expecting things to actually work for 
 the end user. It rarely actually works that way on linux.


If by "works for the end user" you mean that you go to a 
(different for every app) website and download a binary blob, 
give it administrator permissions, go through a graphical 
installer that by default installs 3 toolbars, an anti-virus 
scanner demo and changes your homepage, then downloads a whole 
other program that does the actual installation... then yes, I 
guess it does :)

Honestly, my experience of linux in terms of getting new 
applications has always been a highlight for me ever since I 
first started using it years ago.

apt-get install firefox
emerge firefox
pacman -S firefox
yast -i firefox
yum install firefox

How can any of those be said to not work for the end user? It's 
really very very simple and that's without even discussing the 
GUIs for the point-and-clickers.
Feb 25 2014
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 25 February 2014 at 17:04:11 UTC, John Colvin wrote:

 If by "works for the end user" you mean that you go to a 
 (different for every app) website


I always thought that was the sane way to do it, different 
website for every app, though I loathe and despise chasing down 
library dependencies.

static linking ftw


 other program that does the actual installation... then yes, I 
 guess it does :)


Eh, my standard is dmd: get a zip and run the program with a 
reasonable expectation that it will just work. I don't like 
installers either.


 How can any of those be said to not work for the end user?


yum install php

gets php 5.1, but my other app needs 5.2

something something epel key whatever

yum update php

got php 5.2! .... but php-gd isn't in that repo, so it still is 
5.1 and doesn't work.

wget http://php.net/downloads/whatever/php-5.2.tar.gz
tar zxf blah
cd blah
./configure

missing library X

yum install libx-devel

./configure

missing library Y

yum install liby-devel
no such package

*spend half an hour searching teh web*



./configure

missing library Z

yum install libz-devel
no such package

wget http://libz.org/downloads/libz.tar.gz
tar zxf
cd
./configure

missing library


You get the idea. finally get that working

cd php
./configure

library Z is version 3.1 but php requries 2.8


here we go again

*six hours later*


OH NO I FORGOT TO ADD --enable-gd TO CONFIGURE :-( :-( :-( here 
we go again

./configure --enable-gd --enable-whatever-else ....
<snip>
sudo make install

$ php
Segmentation fault



AAaaaAaaAaaaAaaRrrrReGGGHHhhhhGH. OK, I admit, I'm combining two 
different horror stories into one here, a php one and a recurring 
hassle I've had with gtk, but still, the point stands.


 It's really very very simple and that's without even
 discussing the GUIs for the point-and-clickers.


When it works, it is great, when it doesn't, you're in a world of 
pain. Getting into the repo can be a pain too, so can compiling 
everything.

On Windows, you just compile on your box and it tends to work on 
others. On Linux, you need a dozen different distros installed to 
build the compatible binaries to package...
Feb 26 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 25 February 2014 at 16:04:56 UTC, Assaf Gordon wrote:

 But I think "harming the ecosystem" is a bit of a hyperbole - 
 in an ideal world where Linux Distros contain every piece of 
 software (and up to date version), there is indeed very little 
 need to install from source.


Apologise for overly harsh reaction, it is rather frequent source 
of frustration for me which makes reaction unacceptably nervous.

That "ideal" state is pretty much in your hands. There are as 
many packages in repos as people are willing to put - not like 
there is some important boss who decides what needs to be 
packages.

I know it is not very convenient with Debian because of how 
conservative it is but in Arch just asking in #archlinux   
freenode is likely to result in kind soul willing to help with 
packaging questions. Or even easier, just contacting me as it is 
my direct responsibility to maintain all D-related projects in 
Arch [community] ;)

As general rule of a thumb - if you application is useful, there 
always be someone willing to maintain package for it, assuming 
build process is not utterly insane.


 But there are several real-world constraints, at least in the 
 environment I work in:

 1. Software needs to run on variety of servers and 
 distributions, some are very old (e.g. CentOS 5.4 or Ubuntu 
 10.04).
 Upgrading is not an option for these environments.

 2. Software needs to be installed without root - to a user 
 directory (i.e. using "configure --prefix" or similar).

 3. Software is updated too frequently, or is too new, or 
 (sadly) too experimental (code-wise) for inclusion in a stable 
 distribution package - but it still needs to be used.

 4. It takes time for a software to be properly packaged, and 
 users want to install it now. Also - it will not be available 
 in the "stable" branch of the distribution.


I see, your requirements do not match use case of a typical 
desktop application. However, if you are fine with full static 
linking why not simply distribute binary tarballs for set of 
systems you want to support? It will put burden of having proper 
D compilation environment away from the end user.

It is worth noting though that for any normal application 
requiring package manager access (not full root) is expected and 
desired. Fast update issue can be solved by providing own 
upstream repository (i.e. launchpad)


 If there is already a package in official Debian which is 
 written in D, it would be very helpful to see an example of the 
 packaging process.


Unfortunately all Debian/Ubuntu D packages I am aware about are 
found in non-official repo maintained by Jordi : 
http://d-apt.sourceforge.net/

Debian is probably hardest distro out there to deal with this 
kind of stuff.


 I'm not sure I understand this statement.
 If I link with "gcc -static" and my program is properly built, 
 the resulting binary does not require any library, not even 
 glibc.
 It uses only Linux ABI syscalls.
 Unless I'm mistaken?


I think it will work but I'd recommend to provide an option to 
build system to chose both dynamically and statically linked 
binary to simplify life for those who _may_ want to package it 
later.


 Not sure what the "windows attitude" is, since I haven't 
 developed software for windows in 12 years...


I was referring to extremely common attempts to provide 
all-in-one "installers" ignoring normal practices of target 
systems that often happen when those used to Windows world start 
porting their stuff to Linux. And/or thinking about all distros 
as single target "Linux" system.


 My first choice would be to have a project in D which is easily 
 compilable from source code.
 Either with "make" or "dub" or "configure" or anything else.
 Once I have that, packaging (for a linux distribution) is 
 straight forward, and so is adding it to HomeBrew/LinuxBrew.


dub is the way to go in general, but it is not expected to be 
ever installed on user systems (well, unless it is Gentoo :)) - 
it is to be used by you or anyone willing to (re)package your 
application.


 The alternative (for linux) is distributing a binary program, 
 and for that, a static-binary is much more portable than a 
 dynamic one.


Within your requirements it may be a best choice. In general 
though I'd say that perfect portability is a flaw rather than 
merit. When speaking about desktop applications, for exampe, as a 
user of any operating systems I usually want software that is 
adapted to conform practices of this specific system and hardly 
appreciate how portable it is.
Feb 25 2014
parent reply Assaf Gordon <agordon wi.mit.edu> writes:
On 02/25/2014 05:41 PM, Dicebot wrote:

 I see, your requirements do not match use case of a typical desktop
 application.


This is the root cause of our different POVs - I should have explained it more
clearly.

My users are people who have access to Linux machines which they do not control.
I can expect them to be able to handle the standard/easy steps of compiling
software, but not much more.

This is the case in many universities, were the computer "cluster" is made of
whatever Linux the IT department used.
You won't get root access on those servers. And many time the distribution is
very old and not updated on purpose (for the sake of stability).
To make matters more complicated, many additional programs are installed in
places like "/usr/local/" and "/opt/".
It might make you cringe you as a maintainer (it makes me cringe, as a
root-less user) ... but that's life :)

Now, I have to be realistic:
My program will be useful to some degree, but not a life-changing Killer-App.
If it's easy to use and install, people will use it (and by proxy, perhaps use
D as well, if they want to hack the code).
If it's too complicated, they'll give up and look for a different solution.

So my goal to begin with, was to find the easiest way to distribute my D
program to unix platforms.


However, if you are fine with full static linking why
 not simply distribute binary tarballs for set of systems you want to
 support? It will put burden of having proper D compilation
 environment away from the end user.


We've come full circle :)


something, there is no way to build a statically linked binary with D on Linux.
If there is a way, I would love to know about it.
I would gladly spend the time to create VMs for each common system setup and
package a static binary for it.

  

 I know it is not very convenient with Debian because of how
 conservative it is but in Arch just asking in #archlinux   freenode
 is likely to result in kind soul willing to help with packaging
 questions. Or even easier, just contacting me as it is my direct
 responsibility to maintain all D-related projects in Arch [community]
 ;)


I got one package into Debian, and it took some effort. But it was worth it,
because I learned alot, and it did make my program better (distribution wise).
Arch and Gentoo might be friendlier (and HomeBrew/LinuxBrew even easier), but
the people I'm aiming for aren't using Arch or Gentoo...

I do have high hopes for HomeBrew/LinuxBrew, as "brew install dmd" and "brew
install dub" seems to work nicely on some linux systems.

Best,
  -gordon
Feb 26 2014
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 26 February 2014 at 17:08:38 UTC, Assaf Gordon 
wrote:


 unless I'm missing something, there is no way to build a 
 statically linked binary with D on Linux.


They really aren't far from it out of the box:

$ dmd hellod.d
$ ldd hellod
         linux-gate.so.1 =>  (0xffffe000)
         libpthread.so.0 => /lib/libpthread.so.0 (0xf76db000)
         libm.so.6 => /lib/libm.so.6 (0xf76b5000)
         librt.so.1 => /lib/librt.so.1 (0xf76ac000)
         libc.so.6 => /lib/libc.so.6 (0xf7549000)
         /lib/ld-linux.so.2 (0xf7730000)


It isn't fully static, but those libraries are all pretty basic 
things and generally available on any linux install. The biggest 
problem is probably the libc version not matching on older 
installations.

One option might be to package those .so files right with your 
executable. A poor man's static link.


Anyway, let's try the -static switch we can use to do statically 
linked libc on some programs.

$ dmd hellod.d -c
$ gcc hellod.o -o hellod -m32 
-L/home/me/d/dmd2/linux/bin32/../lib32 -Xlinker --export-dynamic 
-l:libphobos2.a -lpthread -lm -lrt -static

undefined reference to `__tls_get_addr'


BTW if you run "dmd -v yourfile.d" the last line of output is the 
linker command line it runs (via gcc). Then you can copy/paste 
that and tweak the arguments. Here, I added -static.


However, that is *not* a D problem per se, that's a libc 
function. Maybe my system just doesn't have the right stuff 
installed to build a static threaded app.


 I would gladly spend the time to create VMs for each common 
 system setup and package a static binary for it.



If you're doing that, you don't need to do anything special, 
since the vm will link the appropriate libc version anyway, so 
the default build should work.


I betcha if I sent you my binary from my slackware box over to 
your computer it would work even if you are a different distro.
Feb 26 2014
next sibling parent reply Assaf Gordon <agordon wi.mit.edu> writes:
On 02/26/2014 12:34 PM, Adam D. Ruppe wrote:

 On Wednesday, 26 February 2014 at 17:08:38 UTC, Assaf Gordon wrote:


something, there is no way to build a statically linked binary with D on Linux.


 They really aren't far from it out of the box:

 $ dmd hellod.d
 $ ldd hellod
          linux-gate.so.1 =>  (0xffffe000)
          libpthread.so.0 => /lib/libpthread.so.0 (0xf76db000)
          libm.so.6 => /lib/libm.so.6 (0xf76b5000)
          librt.so.1 => /lib/librt.so.1 (0xf76ac000)
          libc.so.6 => /lib/libc.so.6 (0xf7549000)
          /lib/ld-linux.so.2 (0xf7730000)


 It isn't fully static, but those libraries are all pretty basic things and
generally available on any linux install. The biggest problem is probably the
libc version not matching on older installations.


This might seem "generally available" on linux, but it's not.
It might be "generally available" when you assume you're dealing with "recent"
"desktop" versions of linux.
But it's not at all when dealing with servers (as Artem already mentioned
before: http://forum.dlang.org/post/mailman.290.1393304990.6445.digital
ars-d puremagic.com ) .

As a quick example, try to download and run the pre-compiled binaries of "ldc2"
from the dlang.org website on a Stable Debian - they will fail with
incompatible libc versions.


 One option might be to package those .so files right with your executable. A
poor man's static link.


Now that's what I'd call "windows attitude" - package all dependencies
together, in what I think was termed "DLL hell" :)

I think that low-level libraries (e.g. ld-linux, libc, linux-gate, librt,
libpthread) are tightly coupled to the system (and the kernel), you really
don't want to run incompatible ones on an arbitrary linux system.



 Anyway, let's try the -static switch we can use to do statically linked libc
on some programs.

 $ dmd hellod.d -c
 $ gcc hellod.o -o hellod -m32 -L/home/me/d/dmd2/linux/bin32/../lib32 -Xlinker
--export-dynamic -l:libphobos2.a -lpthread -lm -lrt -static

 undefined reference to `__tls_get_addr'


 BTW if you run "dmd -v yourfile.d" the last line of output is the linker
command line it runs (via gcc). Then you can copy/paste that and tweak the
arguments. Here, I added -static.


 However, that is *not* a D problem per se, that's a libc function. Maybe my
system just doesn't have the right stuff installed to build a static threaded
app.


I'd argue the opposite: it is a D problem (or libphobos2/druntime problem,
which is a major part of D).

The missing symbol "__tls_get_addr" is referenced from the D function
getTLSRange() in "druntime/src/rt/sections_linux.d".
And the way druntime uses it is apparently not compatible with static linking
(I know very little about this low-level stuff).


 I betcha if I sent you my binary from my slackware box over to your computer
it would work even if you are a different distro.


You'd be surprised what kind of old versions of linuxes are out there.
I have tried it with C-based projects, and based on my humble experience,
anything except a real static binary will eventually cause problems to users.
Feb 26 2014
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 26 February 2014 at 18:52:55 UTC, Assaf Gordon 
wrote:

 It might be "generally available" when you assume you're 
 dealing with "recent" "desktop" versions of linux.


I've had good luck with them on my servers too but I haven't used 
many of the distros.

But I have had the same libc version problems you see too.


 I think that low-level libraries (e.g. ld-linux, libc, 
 linux-gate, librt, libpthread) are tightly coupled to the 
 system (and the kernel), you really don't want to run 
 incompatible ones on an arbitrary linux system.


If this were true, static linking would be useless! Static 
linking is basically just copying the library functions into your 
binary instead of leaving a reference to the library.


 I'd argue the opposite: it is a D problem (or 
 libphobos2/druntime problem, which is a major part of D).


I'm not sure because searching the web for people having this 
problem with C++ suggests the fix is to update the libc on their 
development computer. So that might work with us too.... but then 
again if it breaks on your computer too who knows.


 I have tried it with C-based projects, and based on my humble 
 experience, anything except a real static binary will 
 eventually cause problems to users.


I agree, this is why I prefer distributing Windows exes :) but if 
you know your target computers it might not be a big problem, 
just compile with a libc close enough to them.
Feb 26 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 26 February 2014 at 17:34:45 UTC, Adam D. Ruppe 
wrote:

 Anyway, let's try the -static switch we can use to do 
 statically linked libc on some programs.

 $ dmd hellod.d -c
 $ gcc hellod.o -o hellod -m32 
 -L/home/me/d/dmd2/linux/bin32/../lib32 -Xlinker 
 --export-dynamic -l:libphobos2.a -lpthread -lm -lrt -static

 undefined reference to `__tls_get_addr'


Huh, I was sure this works. Unpleasant surprise.

Have filed a bug report : 
https://d.puremagic.com/issues/show_bug.cgi?id=12268

( I don't have time to properly answer holy war topics, probably 
later :) )
Feb 26 2014
prev sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 25 February 2014 at 15:06:14 UTC, Dicebot wrote:

 On Tuesday, 25 February 2014 at 14:51:48 UTC, Assaf Gordon 
 wrote:

 My target audience is not D developers - it is unix casual 
 users (not even developers).
 I want to make it the easiest for them to use my program.
 If it's requires more than the standard "configure && make && 
 make install" (or the cmake equivalent), then many of them 
 will not do it.


 Then push it into distro repositories. You are NOT helping by 
 recommending them to do "make install", quite the contrary, 
 harming whole ecosystem. Casual users should not be even aware 
 that "make install" exists. Being one of package maintainers 
 myself I really hate this kind of "helping" attitude, it causes 
 only trouble.


Personally I have to compile all my software by hand at work, 
because I don't have root/sudo access.
Although I agree that in general one should try to work within 
the package managers as much as possible, many (most?) 
distributions don't have even half decent capabilities for 
source-based distribution within the package manager, especially 
when working without root.
Feb 25 2014
prev sibling next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 25 February 2014 at 14:51:48 UTC, Assaf Gordon wrote:

 Either you download and install the package (deb/rpm) which is 
 easy but requires root,
 or you download and compile the sources from the source zip -


OR you download and run the binary zip. The instructions aren't 
very good, but this is really easy.

download dmd.version-you-want.zip
unzip dmd.zip

run dmd2/linux/bin32/dmd yourfile.d

that's all you have to do. You don't have to build anything (most 
the time), you don't have to mess with PATH or LD_LIBRARY_PATH, 
you don't have to copy anything into a system folder. You just 
run the program straight out of the zip.
Feb 25 2014
parent reply Artem Tarasov <lomereiter gmail.com> writes:
On Tue, Feb 25, 2014 at 7:06 PM, Adam D. Ruppe <destructionator gmail.com>wrote:


 OR you download and run the binary zip. The instructions aren't very good,
 but this is really easy.


Yes, that works great on my Mint laptop and, I guess, the majority of
modern distros.
But let's try your advice on CentOS 5 (and please don't ask me why some
people still use it -- I have no idea, too):
$ uname -a

EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
$ wget http://downloads.dlang.org/releases/2014/dmd.2.065.0.zip -q
$ unzip -q dmd.2.065.0.zip
$ cat test.d
import std.stdio; void main() { writeln("Hello, old Linux system!"); }
$ ./dmd2/linux/bin64/dmd -L-L/usr/lib64 test.d
/usr/bin/ld: cannot find -l:libphobos2.a
collect2: ld returned 1 exit status
--- errorlevel 1

O-ops!..
Feb 25 2014
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 25 February 2014 at 16:03:46 UTC, Artem Tarasov wrote:

 /usr/bin/ld: cannot find -l:libphobos2.a


Weird. That might just be a config problem.
Feb 25 2014
parent Shammah Chancellor <anonymous coward.com> writes:
On 2014-02-25 16:21:24 +0000, Adam D. Ruppe said:


 On Tuesday, 25 February 2014 at 16:03:46 UTC, Artem Tarasov wrote:

 /usr/bin/ld: cannot find -l:libphobos2.a


 
 Weird. That might just be a config problem.


The dmd.conf file has the wrong options set for the path.  I ran into 
this problem on MacOS X as well.

-S.
Feb 25 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 25 February 2014 at 14:51:48 UTC, Assaf Gordon wrote:

 Also, as long as DMD is not free, providing easy way to install 
 LDC/GDC is highly desired.


btw, fresh LDC has recently made it to Debian sid : 
https://packages.debian.org/sid/ldc , thanks to Konstantinos 
Margaritis
Feb 25 2014
prev sibling parent reply "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Tuesday, 25 February 2014 at 14:51:48 UTC, Assaf Gordon wrote:

 Also, as long as DMD is not free, providing easy way to install 
 LDC/GDC is highly desired.


On my Ubuntu laptop (standard repos as far as I remember) I have 
to "apt-get install gdc" to install it. I don't understand what 
you meant with "easy way to install". Are you talking about 
getting it from sources or what?
Feb 25 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 25 February 2014 at 15:12:08 UTC, Francesco Cattoglio 
wrote:

 On my Ubuntu laptop (standard repos as far as I remember) I 
 have to "apt-get install gdc" to install it. I don't understand 
 what you meant with "easy way to install". Are you talking 
 about getting it from sources or what?


It is a rather old gdc, I can't recommend to use it ;)
Feb 25 2014
parent reply "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Tuesday, 25 February 2014 at 15:18:06 UTC, Dicebot wrote:

 On Tuesday, 25 February 2014 at 15:12:08 UTC, Francesco 
 Cattoglio wrote:

 On my Ubuntu laptop (standard repos as far as I remember) I 
 have to "apt-get install gdc" to install it. I don't 
 understand what you meant with "easy way to install". Are you 
 talking about getting it from sources or what?


 It is a rather old gdc, I can't recommend to use it ;)


since gdc --version output is 4.8.1, what version would it be? 
like 2.060 or more like 2.052?
Feb 25 2014
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 25 February 2014 at 15:24:19 UTC, Francesco Cattoglio 
wrote:

 since gdc --version output is 4.8.1, what version would it be? 
 like 2.060 or more like 2.052?


Hm, ones I see in package list are for 4.6 - are you using some 
sort of testing repo? 4.8.* can possibly be pretty new, you can 
check fronend version by compiling test program with "pragma(msg, 
__VERSION__)"
Feb 25 2014
parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Tuesday, 25 February 2014 at 15:31:19 UTC, Dicebot wrote:

 On Tuesday, 25 February 2014 at 15:24:19 UTC, Francesco 
 Cattoglio wrote:

 since gdc --version output is 4.8.1, what version would it be? 
 like 2.060 or more like 2.052?


 Hm, ones I see in package list are for 4.6 - are you using some 
 sort of testing repo? 4.8.* can possibly be pretty new, you can 
 check fronend version by compiling test program with 
 "pragma(msg, __VERSION__)"


2062L! 2.062 is decently recent I'd say ;)
Perhaps is from some launchpad repo, but nothing too much 
trascendental.
Anyway, my point was that GDC and LDC are easily available under 
most used distros, given a bit of patience. I mean, we can 
probably package them faster than the distribution updates them 
(I'm referring to Ubuntu 6-months cycle in particular)
Feb 25 2014
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Tuesday, 25 February 2014 at 01:21:03 UTC, Assaf Gordon wrote:

 Hello,

 I working on a small program, which I'd like to make publicly 
 available and easy to distribute,install and use.


If you want to do distribution to developers, dub[1] is the way 
to go.

If you want to distribute to desktop users, binary packages is 
the way to go. The issues with static linking are related to C 
libraries. If you develop in Linux you generally don't statically 
link. All your D code is statically linked, work has been done to 
support dynamic linking with Phobos, but for now that isn't 
default.

1. http://code.dlang.org/
Feb 24 2014
prev sibling next sibling parent =?UTF-8?B?IlRow6lv?= Bueno" <munrek gmx.com> writes:
On Tuesday, 25 February 2014 at 01:21:03 UTC, Assaf Gordon wrote:

 I couldn't find a working autotools template (only few old 
 discussions),
 and so far it seems the best way is writing my own makefile 
 (which I have).


https://github.com/bioinfornatics/MakefileForD
Feb 24 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
For building dub and/or rdmd are de-facto standard for any D 
developers. You should just start from source tarball and proceed 
from there to finding maintainers in specific distributives, 
possibly being the one for distro or two. Then conform packaging 
rules there.

Single Linux-wide binary distribution is mostly a myth and dmd 
shows it by its own.
Feb 25 2014