www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - D for microservices: ldc, rdmd, dub now available on Alpine x86_64

reply Mathias Lang <pro.mathias.lang gmail.com> writes:
Hi all,
Recently there have been inquiries about support for D on Alpine 
Linux, a distribution mostly used in combination with Docker to 
create lightweight container images for microservices.

At BPF Korea, we're working on a blockchain written in D, and 
wanted to be able to easily test and distribute our node using 
Alpine images, but there was no package for it yet.

However, thanks to the work of many contributors before (Joakim 
Noah, yshui, Petar Kirov/zombinedev, and many others), most of 
the porting was already done and it was just a matter of fixing a 
few small issues and and creating the package definitions.

A package for `dub` (v1.18.0), `dtools` (ddemangle & rdmd), and 
`ldc` (v1.18.0) are now available in the `testing` repository of 
Alpine Linux edge. As `testing` is not enabled by default, you 
will need to specify the repository (or add it to your 
`/etc/apk/repositories`) when installing the packages.
For example:
```
apk --no-cache add -X 
http://dl-cdn.alpinelinux.org/alpine/edge/testing ldc ldc-static 
dtools-rdmd dub
```
This command needs to originate from an `alpine:edge` image as it 
links with a recent libc and LLVM.
If you just want the compiler, you still need to provide `ldc` & 
`ldc-static` for things to work out of the box.

More complete example of how we use it to build a D program, 
using multi-stage builds:
```

FROM alpine:edge AS Builder
ARG DUB_OPTIONS
RUN apk --no-cache add build-base git <other dependencies>
RUN apk --no-cache add -X 
http://dl-cdn.alpinelinux.org/alpine/edge/testing ldc ldc-static 
dtools-rdmd dub
ADD . /root/myproject/
WORKDIR /root/myproject/

doesn't play well with docker

`--skip-registry=all`
RUN dub build --compiler=ldc2 ${DUB_OPTIONS}


FROM alpine:edge
COPY --from=Builder /root/project/executable /usr/bin/executable
RUN apk --no-cache add libexecinfo <runtime dependencies>
WORKDIR /root/
ENTRYPOINT [ "/usr/bin/executable" ]
```
`DUB_OPTIONS` can be used to select a build, for example enabling 
coverage in a CI pipeline.

What's next ?
1) There is a pending PR 
(https://github.com/alpinelinux/aports/pull/12006) to have GDC 
working on all architectures alpine supports, not just x86_64.
2) Adding a package for gdmd
3) Rebuild packages based on GDC, so that all architectures are 
supported.
4) Move the packages to community so they are available out of 
the box. It would be great for it to happen by the end of the 
month, as the next alpine release would be around end of December 
according to their schedule, but that depends on how long PR take 
to be reviewed.
5) A DMD package for x86 and x86_64 shouldn't be hard to make 
either
Nov 04 2019
next sibling parent aliak <something something.com> writes:
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote:
 Hi all,
 Recently there have been inquiries about support for D on 
 Alpine Linux, a distribution mostly used in combination with 
 Docker to create lightweight container images for microservices.

 [...]
This is great! Much thanks to all for all the work towards this!
Nov 05 2019
prev sibling next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote:
 [..]
That's great news! Thanks a lot for your hard work!

 which doesn't play well with docker
I have been meaning to add a docker and CI friendly command to dub that fetches all dependencies without building them. In the Node.js world they have the `npm ci` [0] command. [0]: https://docs.npmjs.com/cli/ci.html
Nov 05 2019
parent =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
Am 05.11.2019 um 10:48 schrieb Petar Kirov [ZombineDev]:
 On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote:
 [..]
That's great news! Thanks a lot for your hard work!

 doesn't play well with docker
I have been meaning to add a docker and CI friendly command to dub that fetches all dependencies without building them. In the Node.js world they have the `npm ci` [0] command. [0]: https://docs.npmjs.com/cli/ci.html
Sounds like "dub upgrade --missing-only" will do that, as it implicitly downloads all dependencies in addition to filling the (non-existent in this case) gaps.
Nov 05 2019
prev sibling next sibling parent reply Dejan Lekic <dejan.lekic gmail.com> writes:
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote:
 Hi all,
 Recently there have been inquiries about support for D on 
 Alpine Linux, a distribution mostly used in combination with 
 Docker to create lightweight container images for microservices.

 At BPF Korea, we're working on a blockchain written in D, and 
 wanted to be able to easily test and distribute our node using 
 Alpine images, but there was no package for it yet.
I am confused... Why do you need LDC, DMD, dub, etc on Alpine? Can't you build it anywhere and just put the final artifact (or set of artifacts) inside an Alpine-based container?
Nov 05 2019
parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Tue, Nov 5, 2019 at 12:05 PM Dejan Lekic via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:
 On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote:
 Hi all,
 Recently there have been inquiries about support for D on
 Alpine Linux, a distribution mostly used in combination with
 Docker to create lightweight container images for microservices.

 At BPF Korea, we're working on a blockchain written in D, and
 wanted to be able to easily test and distribute our node using
 Alpine images, but there was no package for it yet.
I am confused... Why do you need LDC, DMD, dub, etc on Alpine? Can't you build it anywhere and just put the final artifact (or set of artifacts) inside an Alpine-based container?
Generally no, because Apline use musl libc instead of glibc, so there are some issues with that
Nov 05 2019
parent reply Jacob Carlborg <doob me.com> writes:
On Tuesday, 5 November 2019 at 11:49:20 UTC, Daniel Kozak wrote:

 Generally no, because Apline use musl libc instead of glibc, so 
 there are some issues with that
The correct way is to use static linking and putting only the binary in a Docker image, i.e. "from scratch" [1] ;). But using Alpine and musl will help with building the binary. [1] https://hub.docker.com/_/scratch
Nov 05 2019
parent reply sarn <sarn theartofmachinery.com> writes:
On Tuesday, 5 November 2019 at 12:20:04 UTC, Jacob Carlborg wrote:
 On Tuesday, 5 November 2019 at 11:49:20 UTC, Daniel Kozak wrote:

 Generally no, because Apline use musl libc instead of glibc, 
 so there are some issues with that
The correct way is to use static linking and putting only the binary in a Docker image, i.e. "from scratch" [1] ;). But using Alpine and musl will help with building the binary. [1] https://hub.docker.com/_/scratch
And the neat way to do that is with a multi-stage build: one Dockerfile, with an Alpine container building the binary, then copying to a FROM scratch container: https://docs.docker.com/develop/develop-images/multistage-build/ The musl build is practically necessary because glibc has effectively given up standalone static binary support. So, thanks BPF Korea :)
Nov 05 2019
parent Jacob Carlborg <doob me.com> writes:
On 2019-11-06 02:02, sarn wrote:

 And the neat way to do that is with a multi-stage build: one Dockerfile, 
 with an Alpine container building the binary, then copying to a FROM 
 scratch container
I've used the "smith" tool as well [1]. It has some additonal help with dynamically linked code. It will use "ldd", recursively, to track dependencies and automatically add those. It will also do a couple of other things, like making the filesystem read only.
 The musl build is practically necessary because glibc has effectively 
 given up standalone static binary support.  So, thanks BPF Korea :)
Yeah, it helps. Although it still possible to statically link with glibc, although you might get some warnings. [1] https://github.com/oracle/smith -- /Jacob Carlborg
Nov 06 2019
prev sibling next sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote:
 At BPF Korea, we're working on a blockchain written in D
Hello, Sorry if this has been said already: would you consider being listed in https://dlang.org/orgs-using-d.html? Thanks
Nov 05 2019
prev sibling next sibling parent reply user <user dlang.io> writes:
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote:
 ```
 apk --no-cache add -X 
 http://dl-cdn.alpinelinux.org/alpine/edge/testing ldc 
 ldc-static dtools-rdmd dub
 ```
A hello world vibe project doesn't build for me using a Dockerfile using your template. I tried to add missing deps, but I couldn't really figure out what's missing. I get an error during the `dub build` step: ``` eventcore 0.8.48: building configuration "epoll"... /root/.dub/packages/eventcore-0.8.48/eventcore/source/eventcore/drivers/posix/ rocesses.d(316,10): Error: module `core.sys.posix.sys.wait` import `idtype_t` not found` ``` My full Dockefile: ``` FROM alpine:edge as builder RUN apk --no-cache add build-base git RUN apk --no-cache add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing ldc ldc-static dtools-rdmd dub RUN apk --no-cache add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing libevent ibevent-dev WORKDIR /tmp/app ADD source ./source ADD dub.json ./ RUN dub build --compiler=ldc2 ```
Nov 13 2019
parent Mathias Lang <pro.mathias.lang gmail.com> writes:
On Wednesday, 13 November 2019 at 12:27:52 UTC, user wrote:
 On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote:
 ```
 apk --no-cache add -X 
 http://dl-cdn.alpinelinux.org/alpine/edge/testing ldc 
 ldc-static dtools-rdmd dub
 ```
A hello world vibe project doesn't build for me using a Dockerfile using your template. I tried to add missing deps, but I couldn't really figure out what's missing. I get an error during the `dub build` step: ``` eventcore 0.8.48: building configuration "epoll"... /root/.dub/packages/eventcore-0.8.48/eventcore/source/eventcore/drivers/posix/ rocesses.d(316,10): Error: module `core.sys.posix.sys.wait` import `idtype_t` not found` ``` My full Dockefile: ``` FROM alpine:edge as builder RUN apk --no-cache add build-base git RUN apk --no-cache add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing ldc ldc-static dtools-rdmd dub RUN apk --no-cache add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing libevent ibevent-dev WORKDIR /tmp/app ADD source ./source ADD dub.json ./ RUN dub build --compiler=ldc2 ```
Unfortunately recent versions of Vibe.d seems to be broken with Alpine. We use eventcore v0.8.43, Vibe.d v0.8.6, vibe-core v1.7.0. I'll see if I can get a CI in place for Vibe (and fix the issues). Thanks for reporting!
Nov 25 2019
prev sibling next sibling parent reply Mathias Lang <pro.mathias.lang gmail.com> writes:
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote:
 What's next ?
 1) There is a pending PR 
 (https://github.com/alpinelinux/aports/pull/12006) to have GDC 
 working on all architectures alpine supports, not just x86_64.
 2) Adding a package for gdmd
 3) Rebuild packages based on GDC, so that all architectures are 
 supported.
 4) Move the packages to community so they are available out of 
 the box. It would be great for it to happen by the end of the 
 month, as the next alpine release would be around end of 
 December according to their schedule, but that depends on how 
 long PR take to be reviewed.
 5) A DMD package for x86 and x86_64 shouldn't be hard to make 
 either
Time for an update! The GDC PR have been merged, and followed by another large fix to make it work on most architectures. So if you use Alpine edge, you can just run `apk add gcc-gdc` and you'll get GDC-9.2.0 (with a lot of patches), which is a 2.076.0 frontend, but allow you to bootstrap efficiently. The package is present on ARMv7, AArch64, x86, x86_64, and s390x. Only ppc64le is not supported as GDC hasn't been ported to it (at least not as of the time of 9.2.0 release). Additionally, `gdmd` has been added to `testing` to make it easier to build packages. LDC has been updated to v1.19.0 and the package definition fixed (only `ldc` and `ldc-runtime` present now, no more `ldc-static`). Dub v1.19.0 is also available, also built with GDC, on all the architectures GDC supports. The LDC package is not going to be cross-architecture in the near future, but it should be able to correctly cross-compile once LDC a version matching 2.090.1 is released (most likely LDC 1.20.0). I am now working on finalizing the LDC package so all tests pass on x86 & x86_64, and to provide a DMD package. While there are still a few details to flesh out, most things should work perfectly on x86_64 (we've been using it for months), so feel free to test it and report back. I also hope we can use it in some of the official dlang repositories (e.g. dlang-tour and the dub testsuite). The best place to report issues with the packages is on the Alpine Gitlab ( https://gitlab.alpinelinux.org/alpine/aports/issues) and to tag Geod24.
Jan 14 2020
next sibling parent reply aberba <karabutaworld gmail.com> writes:
On Wednesday, 15 January 2020 at 04:00:26 UTC, Mathias Lang wrote:
 On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote:
 [...]
Time for an update! The GDC PR have been merged, and followed by another large fix to make it work on most architectures. So if you use Alpine edge, you can just run `apk add gcc-gdc` and you'll get GDC-9.2.0 (with a lot of patches), which is a 2.076.0 frontend, but allow you to bootstrap efficiently. The package is present on ARMv7, AArch64, x86, x86_64, and s390x. Only ppc64le is not supported as GDC hasn't been ported to it (at least not as of the time of 9.2.0 release). [...]
It would be nice to have PBF Korea listed on https://dlang.org/orgs-using-d.html (organizations using D)
Jan 15 2020
parent aberba <karabutaworld gmail.com> writes:
On Wednesday, 15 January 2020 at 11:46:21 UTC, aberba wrote:
 On Wednesday, 15 January 2020 at 04:00:26 UTC, Mathias Lang 
 wrote:
 On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang 
 wrote:
 [...]
Time for an update! The GDC PR have been merged, and followed by another large fix to make it work on most architectures. So if you use Alpine edge, you can just run `apk add gcc-gdc` and you'll get GDC-9.2.0 (with a lot of patches), which is a 2.076.0 frontend, but allow you to bootstrap efficiently. The package is present on ARMv7, AArch64, x86, x86_64, and s390x. Only ppc64le is not supported as GDC hasn't been ported to it (at least not as of the time of 9.2.0 release). [...]
It would be nice to have PBF Korea listed on https://dlang.org/orgs-using-d.html (organizations using D)
Oops, BPF, my bad
Jan 15 2020
prev sibling parent reply kinke <kinke gmx.net> writes:
On Wednesday, 15 January 2020 at 04:00:26 UTC, Mathias Lang wrote:
 The LDC package is not going to be cross-architecture in the 
 near future, but it should be able to correctly cross-compile 
 once LDC a version matching 2.090.1 is released (most likely 
 LDC 1.20.0).
What's the reason for that requirement? I'll soon release LDC v1.20.0-beta1, so is there something from DMD stable you'd need as prerequisite? While I've never fully understood the popularity of Alpine-based docker images (okay, the base image is tiny, but if all images are based on the same, say, Ubuntu image, the few hundred MB only need to be stored once on each host), it'd probably be nice to have the official prebuilt LDC Linux packages linked against musl, to get rid of the glibc dependency. I assume that would be enough to make it runnable on almost all Linux x64 hosts.
Jan 15 2020
parent Mathias Lang <pro.mathias.lang gmail.com> writes:
On Wednesday, 15 January 2020 at 11:48:29 UTC, kinke wrote:
 On Wednesday, 15 January 2020 at 04:00:26 UTC, Mathias Lang 
 wrote:
 The LDC package is not going to be cross-architecture in the 
 near future, but it should be able to correctly cross-compile 
 once LDC a version matching 2.090.1 is released (most likely 
 LDC 1.20.0).
What's the reason for that requirement? I'll soon release LDC v1.20.0-beta1, so is there something from DMD stable you'd need as prerequisite? While I've never fully understood the popularity of Alpine-based docker images (okay, the base image is tiny, but if all images are based on the same, say, Ubuntu image, the few hundred MB only need to be stored once on each host), it'd probably be nice to have the official prebuilt LDC Linux packages linked against musl, to get rid of the glibc dependency. I assume that would be enough to make it runnable on almost all Linux x64 hosts.
Nothing from DMD, but some fixes in Druntime. The `stat` struct definition was broken (among other things), so anything involving files was failing on other architecture. This was fixed in https://github.com/dlang/druntime/pull/2899 . Regarding Alpine's popularity: I think it's a matter of convenience. Just like distributing a single binary (or, like in your case, a standalone package) is easier to deal with, being able to distribute a tiny image packaging your application is great (because the economy of scale argument only holds if you have many images on the same host and don't wipe it frequently). Also, the Ubuntu base image is much slower to build, and just I like my compilation time to be low, I like my image build time to be low. I think the README of alpine covers it pretty well: https://github.com/alpinelinux/docker-alpine#why .
Jan 15 2020
prev sibling parent Newbie2019 <newbie2019 gmail.com> writes:
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote:
 Hi all,
 Recently there have been inquiries about support for D on 
 Alpine Linux, a distribution mostly used in combination with 
 Docker to create lightweight container images for microservices.

 At BPF Korea, we're working on a blockchain written in D, and 
 wanted to be able to easily test and distribute our node using 
 Alpine images, but there was no package for it yet.

 However, thanks to the work of many contributors before (Joakim 
 Noah, yshui, Petar Kirov/zombinedev, and many others), most of 
 the porting was already done and it was just a matter of fixing 
 a few small issues and and creating the package definitions.

 A package for `dub` (v1.18.0), `dtools` (ddemangle & rdmd), and 
 `ldc` (v1.18.0) are now available in the `testing` repository 
 of Alpine Linux edge. As `testing` is not enabled by default, 
 you will need to specify the repository (or add it to your 
 `/etc/apk/repositories`) when installing the packages.
 For example:
 ```
 apk --no-cache add -X 
 http://dl-cdn.alpinelinux.org/alpine/edge/testing ldc 
 ldc-static dtools-rdmd dub
 ```
this not install the lto static lib, how can i build lto static lib for alpine ?
Jan 16 2020