www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D for microservices

reply Joakim <dlang joakim.fea.st> writes:
I just read the following two week-old comment on the ldc issue 
tracker, when someone tried to run D on Alpine linux:

"For now everything works(?) but I think the process could be 
improved.. Would be really cool to have LDC easily building 
alpine containers + static D binaries for microservice and 
tooling development. I'm pretty tired of reading Go code :)"
https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550

It strikes me that microservices are a great way for new 
programming languages like D to get tried and gain some uptake, 
but that D might not be that easy to deploy to that scenario yet.

So this is a question for those deploying microservices, as I'm 
not in that field, what can the D devs do to make it as easy as 
possible to get D microservices up and running, make some Docker 
and Alpine containers with ldc/dub/vibe.d preinstalled publicly 
available?  What else, what kinds of libraries do you normally 
use?

This is a niche that D and all newer languages should target.  
How do we do it?
Oct 21 2017
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:

 I just read the following two week-old comment on the ldc issue 
 tracker, when someone tried to run D on Alpine linux:

 [...]


Rather a container with dub/ldc configured to compile with musl 
libc I think.


 What else, what kinds of libraries do you normally use?


Given current selection I bet most are hand-rolled except maybe 
vibe.d and common serialization libs.
Oct 22 2017
prev sibling next sibling parent qznc <qznc web.de> writes:
On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:

 This is a niche that D and all newer languages should target.  
 How do we do it?


Optimize the TechEmpower benchmark? Vibe.d looks quite weak there.

https://www.techempower.com/benchmarks/
Oct 22 2017
prev sibling next sibling parent Mengu <mengukagan gmail.com> writes:
On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:

 I just read the following two week-old comment on the ldc issue 
 tracker, when someone tried to run D on Alpine linux:

 [...]


rock solid, easy, common-dev-proof, huge std lib. like that of 
golang.
Oct 22 2017
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-10-22 04:48, Joakim wrote:

 I just read the following two week-old comment on the ldc issue tracker, 
 when someone tried to run D on Alpine linux:
 
 "For now everything works(?) but I think the process could be improved.. 
 Would be really cool to have LDC easily building alpine containers + 
 static D binaries for microservice and tooling development. I'm pretty 
 tired of reading Go code :)"
 https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550
 
 It strikes me that microservices are a great way for new programming 
 languages like D to get tried and gain some uptake, but that D might not 
 be that easy to deploy to that scenario yet.
 
 So this is a question for those deploying microservices, as I'm not in 
 that field, what can the D devs do to make it as easy as possible to get 
 D microservices up and running, make some Docker and Alpine containers 
 with ldc/dub/vibe.d preinstalled publicly available?  What else, what 
 kinds of libraries do you normally use?
 
 This is a niche that D and all newer languages should target. How do we 
 do it?


* Support full static linking using DMD, which requires the TLS 
implementation to be modified
* Support musl as the standard C library, I've discussed that before [1]
* Database drivers for the common databases (PostgreSQL, MySQL, SQLite) 
compatible with vibe.d
* Database driver abstraction on top of the above drivers, perhaps some 
lightweight ORM library
* RabbitMQ library compatible with vibe.d
* Serialization to/from JSON and YAML
* Official Docker images with DMD and LDC wouldn't hurt
* Pre-compiled DMD that works on Alpine. Fully statically linked DMD 
would help here

That's what I can think of for now.

[1] http://forum.dlang.org/post/nhem1l$1ee3$1 digitalmars.com

-- 
/Jacob Carlborg
Oct 23 2017
next sibling parent reply Laeeth Isharc <laeeth laeeth.com> writes:
On Monday, 23 October 2017 at 12:08:52 UTC, Jacob Carlborg wrote:

 On 2017-10-22 04:48, Joakim wrote:

 I just read the following two week-old comment on the ldc 
 issue tracker, when someone tried to run D on Alpine linux:
 
 "For now everything works(?) but I think the process could be 
 improved.. Would be really cool to have LDC easily building 
 alpine containers + static D binaries for microservice and 
 tooling development. I'm pretty tired of reading Go code :)"
 https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550
 
 It strikes me that microservices are a great way for new 
 programming languages like D to get tried and gain some 
 uptake, but that D might not be that easy to deploy to that 
 scenario yet.
 
 So this is a question for those deploying microservices, as 
 I'm not in that field, what can the D devs do to make it as 
 easy as possible to get D microservices up and running, make 
 some Docker and Alpine containers with ldc/dub/vibe.d 
 preinstalled publicly available?  What else, what kinds of 
 libraries do you normally use?
 
 This is a niche that D and all newer languages should target. 
 How do we do it?


 * Support full static linking using DMD, which requires the TLS 
 implementation to be modified
 * Support musl as the standard C library, I've discussed that 
 before [1]
 * Database drivers for the common databases (PostgreSQL, MySQL, 
 SQLite) compatible with vibe.d
 * Database driver abstraction on top of the above drivers, 
 perhaps some lightweight ORM library
 * RabbitMQ library compatible with vibe.d
 * Serialization to/from JSON and YAML
 * Official Docker images with DMD and LDC wouldn't hurt
 * Pre-compiled DMD that works on Alpine. Fully statically 
 linked DMD would help here

 That's what I can think of for now.

 [1] http://forum.dlang.org/post/nhem1l$1ee3$1 digitalmars.com


Can you elaborate on how the TLS implementation needs to be 
changed?

If someone wanted to work on rabbit bindings/wrapper, I might be 
open to sponsoring that.  I'm not in love with Rabbit (one node 
uses more than 40% of memory so the node goes down, taking the 
cluster with it.  really?), but we use it currently.

I made a start on bindings for librabbitmq here:
https://github.com/kaleidicassociates/rabbitmq-d


Laeeth.
Oct 23 2017
parent Jacob Carlborg <doob me.com> writes:
On 2017-10-23 14:17, Laeeth Isharc wrote:


 Can you elaborate on how the TLS implementation needs to be changed?



-m64 -static -L/root/.dvm/compilers/dmd-2.076.1/linux/bin/../lib64 
-Xlinker -Bstatic -lphobos2  -lpthread -lm -lrt -ldl

/root/.dvm/compilers/dmd-2.076.1/linux/bin/../lib64/libphobos2.a(sections_el
_shared_782_420.o): 
In function `_D2rt19sections_elf_shared11getTLSRangeFNbNimmZAv':
src/rt/sections_elf_shared.d:(.text._D2rt19sections_elf_shared11getTLSRangeFNbNimmZAv[_D2rt19sections_elf_shared11getTLSRan
eFNbNimmZAv]+0x38): 
undefined reference to `__tls_get_addr'
collect2: error: ld returned 1 exit status

I need to manually invoke GCC to link because DMD will pass "-Xlinker 
--export-dynamic -Xlinker -Bdynamic" and I need to add "-static".


 If someone wanted to work on rabbit bindings/wrapper, I might be open to 
 sponsoring that.  I'm not in love with Rabbit (one node uses more than 
 40% of memory so the node goes down, taking the cluster with it.  
 really?), but we use it currently.
 
 I made a start on bindings for librabbitmq here:
 https://github.com/kaleidicassociates/rabbitmq-d


Is that compatible with vibe.d?

-- 
/Jacob Carlborg
Oct 23 2017
prev sibling next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Monday, 23 October 2017 at 12:08:52 UTC, Jacob Carlborg wrote:

 On 2017-10-22 04:48, Joakim wrote:

 I just read the following two week-old comment on the ldc 
 issue tracker, when someone tried to run D on Alpine linux:
 
 "For now everything works(?) but I think the process could be 
 improved.. Would be really cool to have LDC easily building 
 alpine containers + static D binaries for microservice and 
 tooling development. I'm pretty tired of reading Go code :)"
 https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550
 
 It strikes me that microservices are a great way for new 
 programming languages like D to get tried and gain some 
 uptake, but that D might not be that easy to deploy to that 
 scenario yet.
 
 So this is a question for those deploying microservices, as 
 I'm not in that field, what can the D devs do to make it as 
 easy as possible to get D microservices up and running, make 
 some Docker and Alpine containers with ldc/dub/vibe.d 
 preinstalled publicly available?  What else, what kinds of 
 libraries do you normally use?
 
 This is a niche that D and all newer languages should target. 
 How do we do it?


 * Support full static linking using DMD, which requires the TLS 
 implementation to be modified
 * Support musl as the standard C library, I've discussed that 
 before [1]


I can probably whip this together in a couple days, as I added 
the Bionic support before, but it's not something I need, so it'd 
just be a donation for you and others.  That's why I'm trying to 
figure out who exactly wants this and if others want to chip in.


 * Database drivers for the common databases (PostgreSQL, MySQL, 
 SQLite) compatible with vibe.d
 * Database driver abstraction on top of the above drivers, 
 perhaps some lightweight ORM library
 * RabbitMQ library compatible with vibe.d
 * Serialization to/from JSON and YAML


Some of this stuff is on dub, but nothing that interests me or 
that I'd chip in with.  Good to see that RabbitMQ supports binary 
protocols like AMQP and MQTT though, would be nuts if it were all 
just text like STOMP or especially JSON/YAML.


 * Official Docker images with DMD and LDC wouldn't hurt
 * Pre-compiled DMD that works on Alpine. Fully statically 
 linked DMD would help here


I'm sure someone could put these together if the above stuff 
worked.  The question is who's interested in volunteering to help 
put this all together?
Oct 23 2017
parent reply Jacob Carlborg <doob me.com> writes:
On 2017-10-23 17:35, Joakim wrote:


 I'm sure someone could put these together if the above stuff worked.  
 The question is who's interested in volunteering to help put this all 
 together?


Yeah, exactly. My plate if already full and all this is lower down on 
the priority list.

-- 
/Jacob Carlborg
Oct 23 2017
parent reply Joakim <dlang joakim.fea.st> writes:
On Monday, 23 October 2017 at 18:18:28 UTC, Jacob Carlborg wrote:

 On 2017-10-23 17:35, Joakim wrote:


 I'm sure someone could put these together if the above stuff 
 worked.  The question is who's interested in volunteering to 
 help put this all together?


 Yeah, exactly. My plate if already full and all this is lower 
 down on the priority list.


Yuxuan Shui has ported druntime to Musl over the last couple 
months:

https://github.com/dlang/druntime/pulls?q=is%3Apr+author%3Ayshui+is%3Aclosed

With his changes, all druntime unit tests pass, now only a few 
asserts tripping in the additional tests for shared libraries.  I 
added a couple more tweaks to get all but one of the Phobos unit 
tests passing too.

With a last tiny patch to add a command-line flag to dmd to 
easily use this port- already added to the latest ldc 1.8 beta, 
though it doesn't have all the druntime support: 
https://github.com/ldc-developers/ldc/pull/2373 - anyone can use 
D with Alpine/Musl containers for microservices.
Feb 22 2018
next sibling parent reply Suliman <evermind live.ru> writes:
It would be nice if anyone will rewrite Musl to betterC :)
Feb 22 2018
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 22/02/2018 10:17 PM, Suliman wrote:

 It would be nice if anyone will rewrite Musl to betterC :)


Combine it with dmc's libc and we're starting to get a reasonable state.
Feb 22 2018
prev sibling next sibling parent reply aberba <karabutaworld gmail.com> writes:
On Thursday, 22 February 2018 at 08:54:04 UTC, Joakim wrote:

 On Monday, 23 October 2017 at 18:18:28 UTC, Jacob Carlborg 
 wrote:

 [...]


 Yuxuan Shui has ported druntime to Musl over the last couple 
 months:

 https://github.com/dlang/druntime/pulls?q=is%3Apr+author%3Ayshui+is%3Aclosed

 With his changes, all druntime unit tests pass, now only a few 
 asserts tripping in the additional tests for shared libraries.  
 I added a couple more tweaks to get all but one of the Phobos 
 unit tests passing too.

 With a last tiny patch to add a command-line flag to dmd to 
 easily use this port- already added to the latest ldc 1.8 beta, 
 though it doesn't have all the druntime support: 
 https://github.com/ldc-developers/ldc/pull/2373 - anyone can 
 use D with Alpine/Musl containers for microservices.


That makes me very happy!! Very much appreciated.
Feb 22 2018
parent reply Joakim <dlang joakim.fea.st> writes:
On Thursday, 22 February 2018 at 21:59:27 UTC, aberba wrote:

 On Thursday, 22 February 2018 at 08:54:04 UTC, Joakim wrote:

 On Monday, 23 October 2017 at 18:18:28 UTC, Jacob Carlborg 
 wrote:

 [...]


 Yuxuan Shui has ported druntime to Musl over the last couple 
 months:

 https://github.com/dlang/druntime/pulls?q=is%3Apr+author%3Ayshui+is%3Aclosed

 With his changes, all druntime unit tests pass, now only a few 
 asserts tripping in the additional tests for shared libraries.
  I added a couple more tweaks to get all but one of the Phobos 
 unit tests passing too.

 With a last tiny patch to add a command-line flag to dmd to 
 easily use this port- already added to the latest ldc 1.8 
 beta, though it doesn't have all the druntime support: 
 https://github.com/ldc-developers/ldc/pull/2373 - anyone can 
 use D with Alpine/Musl containers for microservices.


 That makes me very happy!! Very much appreciated.


I've updated ldc master to use Yuxuan's druntime port, which 
means the upcoming ldc 1.8 release will likely be a viable 
Alpine/Musl cross-compiler out of the box, provided you give it a 
C/Musl cross-compiler/linker to work with, by using the bundled 
ldc-build-runtime tool:

https://wiki.dlang.org/Building_LDC_runtime_libraries

Cross-compiling has not been tested however: do people normally 
cross-compile to Alpine containers or build their software in 
Alpine itself?  If the latter, building ldc from master on 
Alpine/x64 3.7.0 passes all of the druntime/phobos stdlib unit 
tests but one, so you're good to go there, as long as you don't 
need a pre-built binary. Maybe we'll put one out for ldc with the 
upcoming 1.8 release.
Feb 24 2018
next sibling parent reply yawniek <yawniek srtnwz.com> writes:
On Saturday, 24 February 2018 at 10:03:07 UTC, Joakim wrote:

 I've updated ldc master to use Yuxuan's druntime port, which 
 means the upcoming ldc 1.8 release will likely be a viable 
 Alpine/Musl cross-compiler out of the box, provided you give it 
 a C/Musl cross-compiler/linker to work with, by using the 
 bundled ldc-build-runtime tool:

 https://wiki.dlang.org/Building_LDC_runtime_libraries

 Cross-compiling has not been tested however: do people normally 
 cross-compile to Alpine containers or build their software in 
 Alpine itself?  If the latter, building ldc from master on 
 Alpine/x64 3.7.0 passes all of the druntime/phobos stdlib unit 
 tests but one, so you're good to go there, as long as you don't 
 need a pre-built binary. Maybe we'll put one out for ldc with 
 the upcoming 1.8 release.


great stuff, thank you! this will be very useful!

Q: what would be needed to build a single binary (a la golang) 
that works in a FROM SCRATCH docker container?
Feb 25 2018
next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Sunday, 25 February 2018 at 16:51:09 UTC, yawniek wrote:

 great stuff, thank you! this will be very useful!

 Q: what would be needed to build a single binary (a la golang) 
 that works in a FROM SCRATCH docker container?


I don't know, presumably you're referring to the static linking 
support Jacob mentioned earlier in this thread.  I have not tried 
that.

On Sunday, 25 February 2018 at 17:48:34 UTC, aberba wrote:

 I usually ship and compile code in Alpine itself. Once I have 
 an ldc compiler with Alpine as base image,  I'm good to go. 
 Some platforms like OpenShift will rebuild when a release is 
 triggered in git master... Copying binary require some hacks.


OK, I will look at releasing a native ldc binary for Alpine with 
the upcoming 1.8 release.
Feb 25 2018
parent reply aberba <karabutaworld gmail.com> writes:
On Sunday, 25 February 2018 at 22:12:38 UTC, Joakim wrote:

 On Sunday, 25 February 2018 at 16:51:09 UTC, yawniek wrote:

 great stuff, thank you! this will be very useful!

 Q: what would be needed to build a single binary (a la golang) 
 that works in a FROM SCRATCH docker container?


 I don't know, presumably you're referring to the static linking 
 support Jacob mentioned earlier in this thread.  I have not 
 tried that.

 On Sunday, 25 February 2018 at 17:48:34 UTC, aberba wrote:

 I usually ship and compile code in Alpine itself. Once I have 
 an ldc compiler with Alpine as base image,  I'm good to go. 
 Some platforms like OpenShift will rebuild when a release is 
 triggered in git master... Copying binary require some hacks.


 OK, I will look at releasing a native ldc binary for Alpine 
 with the upcoming 1.8 release.


LDC 1.8 is out!
Mar 05 2018
parent reply Joakim <dlang joakim.fea.st> writes:
On Monday, 5 March 2018 at 14:34:44 UTC, aberba wrote:

 On Sunday, 25 February 2018 at 22:12:38 UTC, Joakim wrote:

 On Sunday, 25 February 2018 at 16:51:09 UTC, yawniek wrote:

 great stuff, thank you! this will be very useful!

 Q: what would be needed to build a single binary (a la 
 golang) that works in a FROM SCRATCH docker container?


 I don't know, presumably you're referring to the static 
 linking support Jacob mentioned earlier in this thread.  I 
 have not tried that.

 On Sunday, 25 February 2018 at 17:48:34 UTC, aberba wrote:

 I usually ship and compile code in Alpine itself. Once I have 
 an ldc compiler with Alpine as base image,  I'm good to go. 
 Some platforms like OpenShift will rebuild when a release is 
 triggered in git master... Copying binary require some hacks.


 OK, I will look at releasing a native ldc binary for Alpine 
 with the upcoming 1.8 release.


 LDC 1.8 is out!


The Alpine build is up, let me know if you have any problems.  
Note the changelog entry that says you'll need to install llvm 
and maybe other packages from the Alpine package manager first.
Mar 05 2018
next sibling parent aberba <karabutaworld gmail.com> writes:
On Monday, 5 March 2018 at 17:58:25 UTC, Joakim wrote:

 On Monday, 5 March 2018 at 14:34:44 UTC, aberba wrote:

 On Sunday, 25 February 2018 at 22:12:38 UTC, Joakim wrote:

 On Sunday, 25 February 2018 at 16:51:09 UTC, yawniek wrote:

 [...]


 I don't know, presumably you're referring to the static 
 linking support Jacob mentioned earlier in this thread.  I 
 have not tried that.

 On Sunday, 25 February 2018 at 17:48:34 UTC, aberba wrote:

 [...]


 OK, I will look at releasing a native ldc binary for Alpine 
 with the upcoming 1.8 release.


 LDC 1.8 is out!


 The Alpine build is up, let me know if you have any problems.  
 Note the changelog entry that says you'll need to install llvm 
 and maybe other packages from the Alpine package manager first.


Nice. I usually use the docker images though.
Mar 06 2018
prev sibling parent reply Andrew Benton <Andrew.Benton675 gmail.com> writes:
On Monday, 5 March 2018 at 17:58:25 UTC, Joakim wrote:

 On Monday, 5 March 2018 at 14:34:44 UTC, aberba wrote:

 [snip]


 The Alpine build is up, let me know if you have any problems.  
 Note the changelog entry that says you'll need to install llvm 
 and maybe other packages from the Alpine package manager first.


I took a shot at dockerizing dub-registry with the alpine build 
using multistage builds.  I got ldc installed and dub built.  
Unfortunately when I tried to build dub-registry, I found that 
vibe-d's vibe-core, libevent, and libasync configurations can't 
run under musl due to some of the networking dependencies in 
druntime core, e.g. `core.sys.linux.netinet.in_`.

I realize that's something that you can't really prevent as the 
packager, but it's still relevant to the status of D for 
microservices.

Bad news: not being able to build vibe.d is a pretty big hit for 
this objective.
Good news: there are other libraries we can try and we know how 
to fix this problem.
Mar 07 2018
next sibling parent reply Radu <void null.pt> writes:
On Thursday, 8 March 2018 at 06:49:07 UTC, Andrew Benton wrote:

 On Monday, 5 March 2018 at 17:58:25 UTC, Joakim wrote:

 [...]


 I took a shot at dockerizing dub-registry with the alpine build 
 using multistage builds.  I got ldc installed and dub built.  
 Unfortunately when I tried to build dub-registry, I found that 
 vibe-d's vibe-core, libevent, and libasync configurations can't 
 run under musl due to some of the networking dependencies in 
 druntime core, e.g. `core.sys.linux.netinet.in_`.

 I realize that's something that you can't really prevent as the 
 packager, but it's still relevant to the status of D for 
 microservices.

 Bad news: not being able to build vibe.d is a pretty big hit 
 for this objective.
 Good news: there are other libraries we can try and we know how 
 to fix this problem.


This guys says that vide.d works 
https://forum.dlang.org/thread/gikoeutmdyvolfshpqqq forum.dlang.org
Mar 07 2018
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Thursday, 8 March 2018 at 07:20:53 UTC, Radu wrote:


 This guys says that vide.d works 
 https://forum.dlang.org/thread/gikoeutmdyvolfshpqqq forum.dlang.org


Yes, it's pretty straightforward:

1. Build on Ubuntu, or some other dist
2. Statically link the whole binary with LDC, then you don't need 
to use Musl
3. Run it on Alpine

--
/Jacob Carlborg
Mar 07 2018
parent reply Andrew Benton <Andrew.Benton675 gmail.com> writes:
On Thursday, 8 March 2018 at 07:54:29 UTC, Jacob Carlborg wrote:

 On Thursday, 8 March 2018 at 07:20:53 UTC, Radu wrote:


 This guys says that vide.d works 
 https://forum.dlang.org/thread/gikoeutmdyvolfshpqqq forum.dlang.org


 Yes, it's pretty straightforward:

 1. Build on Ubuntu, or some other dist
 2. Statically link the whole binary with LDC, then you don't 
 need to use Musl
 3. Run it on Alpine

 --
 /Jacob Carlborg


Looks like that works out much nicer than using alpine as the 
first stage right now.  I'm not sure anyone will be too upset 
about the extra 7MB.
Mar 08 2018
parent reply aberba <karabutaworld gmail.com> writes:
On Thursday, 8 March 2018 at 09:35:08 UTC, Andrew Benton wrote:

 On Thursday, 8 March 2018 at 07:54:29 UTC, Jacob Carlborg wrote:

 On Thursday, 8 March 2018 at 07:20:53 UTC, Radu wrote:


 This guys says that vide.d works 
 https://forum.dlang.org/thread/gikoeutmdyvolfshpqqq forum.dlang.org


 Yes, it's pretty straightforward:

 1. Build on Ubuntu, or some other dist
 2. Statically link the whole binary with LDC, then you don't 
 need to use Musl
 3. Run it on Alpine

 --
 /Jacob Carlborg


 Looks like that works out much nicer than using alpine as the 
 first stage right now.  I'm not sure anyone will be too upset 
 about the extra 7MB.


Building directly in alpine will simplify automated builds too.
Mar 08 2018
parent reply Tamas <user dlang.io> writes:
On Thursday, 8 March 2018 at 22:53:11 UTC, aberba wrote:

 On Thursday, 8 March 2018 at 09:35:08 UTC, Andrew Benton wrote:

 On Thursday, 8 March 2018 at 07:54:29 UTC, Jacob Carlborg 
 wrote:

 On Thursday, 8 March 2018 at 07:20:53 UTC, Radu wrote:


 This guys says that vide.d works 
 https://forum.dlang.org/thread/gikoeutmdyvolfshpqqq forum.dlang.org


 Yes, it's pretty straightforward:

 1. Build on Ubuntu, or some other dist
 2. Statically link the whole binary with LDC, then you don't 
 need to use Musl
 3. Run it on Alpine

 --
 /Jacob Carlborg


 Looks like that works out much nicer than using alpine as the 
 first stage right now.  I'm not sure anyone will be too upset 
 about the extra 7MB.


 Building directly in alpine will simplify automated builds too.


Simple Vibe.d app talking to Redis, packed into docker containers:
https://github.com/tam4s/hello-redis

The takeaway is that I could not use Alpine as a host image, 
because I could not build the app statically on ubuntu.

warning: Using 'dlopen' in statically linked applications 
requires at runtime the shared libraries from the glibc version 
used for linking
Mar 11 2018
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2018-03-11 12:10, Tamas wrote:


 Simple Vibe.d app talking to Redis, packed into docker containers:
 https://github.com/tam4s/hello-redis
 
 The takeaway is that I could not use Alpine as a host image, because I 
 could not build the app statically on ubuntu.
 
 warning: Using 'dlopen' in statically linked applications requires at 
 runtime the shared libraries from the glibc version used for linking


It's just a warning. If you're not using "dlopen" or any of your 
dependencies you can just ignore the warning.

-- 
/Jacob Carlborg
Mar 11 2018
prev sibling parent Joakim <dlang joakim.fea.st> writes:
On Sunday, 11 March 2018 at 11:10:09 UTC, Tamas wrote:

 On Thursday, 8 March 2018 at 22:53:11 UTC, aberba wrote:

 On Thursday, 8 March 2018 at 09:35:08 UTC, Andrew Benton wrote:

 On Thursday, 8 March 2018 at 07:54:29 UTC, Jacob Carlborg 
 wrote:

 On Thursday, 8 March 2018 at 07:20:53 UTC, Radu wrote:


 This guys says that vide.d works 
 https://forum.dlang.org/thread/gikoeutmdyvolfshpqqq forum.dlang.org


 Yes, it's pretty straightforward:

 1. Build on Ubuntu, or some other dist
 2. Statically link the whole binary with LDC, then you don't 
 need to use Musl
 3. Run it on Alpine

 --
 /Jacob Carlborg


 Looks like that works out much nicer than using alpine as the 
 first stage right now.  I'm not sure anyone will be too upset 
 about the extra 7MB.


 Building directly in alpine will simplify automated builds too.


 Simple Vibe.d app talking to Redis, packed into docker 
 containers:
 https://github.com/tam4s/hello-redis

 The takeaway is that I could not use Alpine as a host image, 
 because I could not build the app statically on ubuntu.

 warning: Using 'dlopen' in statically linked applications 
 requires at runtime the shared libraries from the glibc version 
 used for linking


I've put up a new binary release of ldc 1.9 for Alpine, which 
fixes the vibe.d issues by pulling in two upstream commits from 
druntime, adds druntime and Phobos as shared libraries, and 
includes dub and rdmd:

https://github.com/ldc-developers/ldc/releases

I tested by unpacking that release in an Alpine VPS, adding its 
bin/ to my path, and simply running the following:

dub fetch vibe-d
dub build vibe-d

The list of Alpine packages needed to run ldc are listed in the 
release notes. You can also cross-compile using the regular linux 
build of ldc by using ldc-build-runtime, the included tool to 
rebuild the stdlib for other platforms, and these instructions 
from the wiki:

https://wiki.dlang.org/Building_LDC_runtime_libraries#Usage_for_cross-compilation

I'm looking at adding a flag to dmd to enable building for the 
Musl C runtime:

https://github.com/dlang/dmd/pull/8020

While the Musl port is mostly there, it appears that yshui didn't 
bother porting all of druntime, so there may still be other dub 
packages that need other missing C declarations. However, I'm 
done working on this port, beyond finishing off the above pull 
and the druntime pull linked from the release notes, as I don't 
use Alpine, containers, or microservices. I simply chipped in 
because I have porting experience and thought I could push D for 
microservices a bit farther along. I'll keep putting out official 
builds of ldc for Alpine though, as long as there's demand for 
them.

If you'd like to use D in Alpine containers, now's the time to 
contribute to whatever else is missing.  A good idea of the 
current status can be found in the first post of the dmd pull 
linked above.

This may be a good project for the hackathon, as it's mostly 
polishing up a bunch of small things.
May 05 2018
prev sibling parent Andrew Benton <Andrew.Benton675 gmail.com> writes:
On Thursday, 8 March 2018 at 07:20:53 UTC, Radu wrote:

 On Thursday, 8 March 2018 at 06:49:07 UTC, Andrew Benton wrote:

 On Monday, 5 March 2018 at 17:58:25 UTC, Joakim wrote:
 [snip]


 This guys says that vide.d works 
 https://forum.dlang.org/thread/gikoeutmdyvolfshpqqq forum.dlang.org


Unfortunately the link that's associated with is based on an 
ubuntu build which should be using glibc, not alpine w/ musl.
Mar 08 2018
prev sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Thursday, 8 March 2018 at 06:49:07 UTC, Andrew Benton wrote:

 On Monday, 5 March 2018 at 17:58:25 UTC, Joakim wrote:

 [...]


 I took a shot at dockerizing dub-registry with the alpine build 
 using multistage builds.  I got ldc installed and dub built.  
 Unfortunately when I tried to build dub-registry, I found that 
 vibe-d's vibe-core, libevent, and libasync configurations can't 
 run under musl due to some of the networking dependencies in 
 druntime core, e.g. `core.sys.linux.netinet.in_`.

 I realize that's something that you can't really prevent as the 
 packager, but it's still relevant to the status of D for 
 microservices.

 Bad news: not being able to build vibe.d is a pretty big hit 
 for this objective.
 Good news: there are other libraries we can try and we know how 
 to fix this problem.


What is the exact error? Maybe report it here:

https://github.com/lindt/docker-dmd/issues/1
Mar 08 2018
parent reply Andrew Benton <Andrew.Benton675 gmail.com> writes:
On Thursday, 8 March 2018 at 08:07:23 UTC, Joakim wrote:

 On Thursday, 8 March 2018 at 06:49:07 UTC, Andrew Benton wrote:

 On Monday, 5 March 2018 at 17:58:25 UTC, Joakim wrote:
 [snip]


 What is the exact error? Maybe report it here:

 https://github.com/lindt/docker-dmd/issues/1


I built out LDC on Alpine using the v1.8.0 LDC release.  It's 
available now through Docker Hub and Github.

Docker: https://hub.docker.com/r/andrewbenton/alpine-ldc/
Github: https://github.com/andrewbenton/alpine-ldc/

I get errors like below when trying to compile dub-registry:


/root/.dub/packages/eventcore-0.8.28/eventcore/source/eventcore/driver
/posix/dns.d(11,9): Error: module core.sys.posix.netdb import 'AI_ADDRCONFIG'
not found
/root/.dub/packages/eventcore-0.8.28/eventcore/source/eventcore/driver
/posix/dns.d(11,9): Error: module core.sys.posix.netdb import 'AI_V4MAPPED' not
found
/root/.dub/packages/eventcore-0.8.28/eventcore/source/eventcore/drivers/po
ix/sockets.d(13,9): Error: module core.sys.posix.netdb import 'AI_ADDRCONFIG'
not found
/root/.dub/packages/eventcore-0.8.28/eventcore/source/eventcore/drivers/po
ix/sockets.d(13,9): Error: module core.sys.posix.netdb import 'AI_V4MAPPED' not
found
/root/.dub/packages/eventcore-0.8.28/eventcore/source/eventcore/drivers/pos
x/sockets.d(47,10): Error: module core.sys.linux.netinet.in_ import
'IP_ADD_MEMBERSHIP' not found
/root/.dub/packages/eventcore-0.8.28/eventcore/source/eventcore/drivers/pos
x/sockets.d(47,10): Error: module core.sys.linux.netinet.in_ import
'IP_MULTICAST_LOOP' not found, did you mean enum member 'IPV6_MULTICAST_LOOP'?
ldc2 failed with exit code 1.
Mar 08 2018
parent Radu <void null.pt> writes:
On Thursday, 8 March 2018 at 09:28:02 UTC, Andrew Benton wrote:

 On Thursday, 8 March 2018 at 08:07:23 UTC, Joakim wrote:

 [...]


 I built out LDC on Alpine using the v1.8.0 LDC release.  It's 
 available now through Docker Hub and Github.

 Docker: https://hub.docker.com/r/andrewbenton/alpine-ldc/
 Github: https://github.com/andrewbenton/alpine-ldc/

 I get errors like below when trying to compile dub-registry:


[...]




OK you found an issue :). Indeed the file only covered Glibc so 
that's why vibe.d didn't compile for Musl. I fixed the issue for 
uClibc (1) and it works now (builds and runs).

Probably the fix for Musl is similar.

1. https://github.com/dlang/druntime/pull/2134
Mar 08 2018
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2018-02-25 17:51, yawniek wrote:


 Q: what would be needed to build a single binary (a la golang) that
 works in a FROM SCRATCH docker container?


Build a completely statically linked binary by compiling using LDC and 
add the "-static" flag.

-- 
/Jacob Carlborg
Mar 01 2018
prev sibling parent aberba <karabutaworld gmail.com> writes:
On Saturday, 24 February 2018 at 10:03:07 UTC, Joakim wrote:

 On Thursday, 22 February 2018 at 21:59:27 UTC, aberba wrote:

 On Thursday, 22 February 2018 at 08:54:04 UTC, Joakim wrote:

 [...]


 That makes me very happy!! Very much appreciated.


 I've updated ldc master to use Yuxuan's druntime port, which 
 means the upcoming ldc 1.8 release will likely be a viable 
 Alpine/Musl cross-compiler out of the box, provided you give it 
 a C/Musl cross-compiler/linker to work with, by using the 
 bundled ldc-build-runtime tool:

 https://wiki.dlang.org/Building_LDC_runtime_libraries

 Cross-compiling has not been tested however: do people normally 
 cross-compile to Alpine containers or build their software in 
 Alpine itself?  If the latter, building ldc from master on 
 Alpine/x64 3.7.0 passes all of the druntime/phobos stdlib unit 
 tests but one, so you're good to go there, as long as you don't 
 need a pre-built binary. Maybe we'll put one out for ldc with 
 the upcoming 1.8 release.


I usually ship and compile code in Alpine itself. Once I have an 
ldc compiler with Alpine as base image,  I'm good to go. Some 
platforms like OpenShift will rebuild when a release is triggered 
in git master... Copying binary require some hacks.
Feb 25 2018
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2018-02-22 09:54, Joakim wrote:


 Yuxuan Shui has ported druntime to Musl over the last couple months:
 
 https://github.com/dlang/druntime/pulls?q=is%3Apr+author%3Ayshui+is%3Aclosed 
 
 
 With his changes, all druntime unit tests pass, now only a few asserts 
 tripping in the additional tests for shared libraries.  I added a couple 
 more tweaks to get all but one of the Phobos unit tests passing too.
 
 With a last tiny patch to add a command-line flag to dmd to easily use 
 this port- already added to the latest ldc 1.8 beta, though it doesn't 
 have all the druntime support: 
 https://github.com/ldc-developers/ldc/pull/2373 - anyone can use D with 
 Alpine/Musl containers for microservices.


That's great.

-- 
/Jacob Carlborg
Feb 23 2018
prev sibling parent reply Adam Wilson <flyboynw gmail.com> writes:
On 10/23/17 05:08, Jacob Carlborg wrote:

 * Database drivers for the common databases (PostgreSQL, MySQL, SQLite)
 compatible with vibe.d
 * Database driver abstraction on top of the above drivers, perhaps some
 lightweight ORM library


I've been looking pretty extensively at these two items recently.

If the database drivers are compatible with Vibe.d AND we wish to 
provide a common abstraction layer for them (presumably via Phobos) then 
order for the abstraction layer to aware of the whether the driver is 
making a blocking or non-blocking call we must include Vibe.D in the 
abstraction layer. Ergo, we must include at least the vibe-core package 
in Phobos, or more preferably, DRT.

I had heard noises about that a few months ago. Anything happening on 
that front?

An event loop is a key piece of a project (Async/Await) that I want to 
work on, and having it in DRuntime would make that project fantastically 
simpler. IMHO, the bulk of the time required is in getting an event loop 
into DRT, the rest is a LOT of relatively straightforward compiler lowering.

IMHO, DRT is in significant need of an event loop system. This would 
allow us to simplify a large number of problems (Async/Await, GUI's, IO, 
etc). As near as I can tell, the problem isn't so much doing the work, 
but getting the required sign-off's for inclusion into DRT.

Another problem that I've been made aware of is that vibe-core may not 
be ideal in certain situations. As this would be landed in DRT itself 
this would obviously need to be addressed.

What would the appetite be for working together to come up with a 
reasonably generic event loop for DRT that vibe and other systems could 
then leverage?

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
Oct 23 2017
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 23/10/2017 11:02 PM, Adam Wilson wrote:

 On 10/23/17 05:08, Jacob Carlborg wrote:

 * Database drivers for the common databases (PostgreSQL, MySQL, SQLite)
 compatible with vibe.d
 * Database driver abstraction on top of the above drivers, perhaps some
 lightweight ORM library


 
 I've been looking pretty extensively at these two items recently.
 
 If the database drivers are compatible with Vibe.d AND we wish to 
 provide a common abstraction layer for them (presumably via Phobos) then 
 order for the abstraction layer to aware of the whether the driver is 
 making a blocking or non-blocking call we must include Vibe.D in the 
 abstraction layer. Ergo, we must include at least the vibe-core package 
 in Phobos, or more preferably, DRT.
 
 I had heard noises about that a few months ago. Anything happening on 
 that front?
 
 An event loop is a key piece of a project (Async/Await) that I want to 
 work on, and having it in DRuntime would make that project fantastically 
 simpler. IMHO, the bulk of the time required is in getting an event loop 
 into DRT, the rest is a LOT of relatively straightforward compiler 
 lowering.
 
 IMHO, DRT is in significant need of an event loop system. This would 
 allow us to simplify a large number of problems (Async/Await, GUI's, IO, 
 etc). As near as I can tell, the problem isn't so much doing the work, 
 but getting the required sign-off's for inclusion into DRT.
 
 Another problem that I've been made aware of is that vibe-core may not 
 be ideal in certain situations. As this would be landed in DRT itself 
 this would obviously need to be addressed.
 
 What would the appetite be for working together to come up with a 
 reasonably generic event loop for DRT that vibe and other systems could 
 then leverage?
 


*whispers* heyyy, heard about SPEW[0]?

[0] 
https://github.com/Devisualization/spew/blob/master/src/base/cf/spew/event_loop/defs.d
Oct 23 2017
parent Adam Wilson <flyboynw gmail.com> writes:
On 10/23/17 18:51, rikki cattermole wrote:

 On 23/10/2017 11:02 PM, Adam Wilson wrote:

 On 10/23/17 05:08, Jacob Carlborg wrote:

 * Database drivers for the common databases (PostgreSQL, MySQL, SQLite)
 compatible with vibe.d
 * Database driver abstraction on top of the above drivers, perhaps some
 lightweight ORM library


 I've been looking pretty extensively at these two items recently.

 If the database drivers are compatible with Vibe.d AND we wish to
 provide a common abstraction layer for them (presumably via Phobos)
 then order for the abstraction layer to aware of the whether the
 driver is making a blocking or non-blocking call we must include
 Vibe.D in the abstraction layer. Ergo, we must include at least the
 vibe-core package in Phobos, or more preferably, DRT.

 I had heard noises about that a few months ago. Anything happening on
 that front?

 An event loop is a key piece of a project (Async/Await) that I want to
 work on, and having it in DRuntime would make that project
 fantastically simpler. IMHO, the bulk of the time required is in
 getting an event loop into DRT, the rest is a LOT of relatively
 straightforward compiler lowering.

 IMHO, DRT is in significant need of an event loop system. This would
 allow us to simplify a large number of problems (Async/Await, GUI's,
 IO, etc). As near as I can tell, the problem isn't so much doing the
 work, but getting the required sign-off's for inclusion into DRT.

 Another problem that I've been made aware of is that vibe-core may not
 be ideal in certain situations. As this would be landed in DRT itself
 this would obviously need to be addressed.

 What would the appetite be for working together to come up with a
 reasonably generic event loop for DRT that vibe and other systems
 could then leverage?


 *whispers* heyyy, heard about SPEW[0]?

 [0]
 https://github.com/Devisualization/spew/blob/master/src/base/cf/spew/event_loop/defs.d


You've mentioned it a few times. ;-)

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
Oct 25 2017
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-10-24 00:02, Adam Wilson wrote:


 I've been looking pretty extensively at these two items recently.
 
 If the database drivers are compatible with Vibe.d AND we wish to 
 provide a common abstraction layer for them (presumably via Phobos) then 
 order for the abstraction layer to aware of the whether the driver is 
 making a blocking or non-blocking call we must include Vibe.D in the 
 abstraction layer. Ergo, we must include at least the vibe-core package 
 in Phobos, or more preferably, DRT.


It can be an optional dependency. Looking at ddb [1] it's not that much 
code that will be different if vibe.d is used or not. It's basically 
only reading and writing to the socket that is different [2]. Dub 
provides predefined version for different dependencies, i.e. 
Have_vibe_d_core.


 I had heard noises about that a few months ago. Anything happening on 
 that front?


No, not as far as I know. Sönke seems really busy recently.


 What would the appetite be for working together to come up with a 
 reasonably generic event loop for DRT that vibe and other systems could 
 then leverage?


I would be really nice to database support directly in the standard 
library but it's not critical for me. It takes a lot of work with and 
massive overhead to get something into Phobos. It's also going to be 
really slow get in updates. I'm not sure if it's worth it.

[1] https://github.com/pszturmaj/ddb
[2] 
https://github.com/pszturmaj/ddb/blob/master/source/ddb/postgres.d#L189-L246

-- 
/Jacob Carlborg
Oct 23 2017
parent reply Adam Wilson <flyboynw gmail.com> writes:
On 10/23/17 23:29, Jacob Carlborg wrote:

 On 2017-10-24 00:02, Adam Wilson wrote:


 I've been looking pretty extensively at these two items recently.

 If the database drivers are compatible with Vibe.d AND we wish to
 provide a common abstraction layer for them (presumably via Phobos)
 then order for the abstraction layer to aware of the whether the
 driver is making a blocking or non-blocking call we must include
 Vibe.D in the abstraction layer. Ergo, we must include at least the
 vibe-core package in Phobos, or more preferably, DRT.


 It can be an optional dependency. Looking at ddb [1] it's not that much
 code that will be different if vibe.d is used or not. It's basically
 only reading and writing to the socket that is different [2]. Dub
 provides predefined version for different dependencies, i.e.
 Have_vibe_d_core.


This of course makes the assumption that we clean-room our own protocol 
implementations which I am entirely against. Better to use what already 
exists.


 I had heard noises about that a few months ago. Anything happening on
 that front?


 No, not as far as I know. Sönke seems really busy recently.


That is what I was afraid of.


 What would the appetite be for working together to come up with a
 reasonably generic event loop for DRT that vibe and other systems
 could then leverage?


 I would be really nice to database support directly in the standard
 library but it's not critical for me. It takes a lot of work with and
 massive overhead to get something into Phobos. It's also going to be
 really slow get in updates. I'm not sure if it's worth it.

 [1] https://github.com/pszturmaj/ddb
 [2]
 https://github.com/pszturmaj/ddb/blob/master/source/ddb/postgres.d#L189-L246


I actually don't think the slow updates are huge problem as these DB 
interface libraries are pretty slow to change themselves. For example, 
ADO.NET didn't change significantly from it's 1.0 release until the 
arrival of Async/Await in .NET 4.5, which was 10 years later. The 
biggest addition prior to Async was TVP support and that was tiny 
comparatively and came in 2005.

The libpq5 interface has barely changed in years. I can't speak to MySQL 
but IIRC it hasn't changed much either, at least not in a way that would 
effect the abstraction layer.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
Oct 25 2017
parent reply Jacob Carlborg <doob me.com> writes:
On 2017-10-26 00:53, Adam Wilson wrote:


 This of course makes the assumption that we clean-room our own protocol 
 implementations which I am entirely against. Better to use what already 
 exists.


I'm entirely against anything that is not compatible with vibe.d ;)


 I actually don't think the slow updates are huge problem as these DB 
 interface libraries are pretty slow to change themselves. For example, 
 ADO.NET didn't change significantly from it's 1.0 release until the 
 arrival of Async/Await in .NET 4.5, which was 10 years later. The 
 biggest addition prior to Async was TVP support and that was tiny 
 comparatively and came in 2005.
 
 The libpq5 interface has barely changed in years. I can't speak to MySQL 
 but IIRC it hasn't changed much either, at least not in a way that would 
 effect the abstraction layer.


I'm more concerned that I don't think we'll manage to implement a 
complete API and 100% bug free at the first try.

-- 
/Jacob Carlborg
Oct 25 2017
parent reply Adam Wilson <flyboynw gmail.com> writes:
On 10/25/17 23:57, Jacob Carlborg wrote:

 On 2017-10-26 00:53, Adam Wilson wrote:


 This of course makes the assumption that we clean-room our own
 protocol implementations which I am entirely against. Better to use
 what already exists.


 I'm entirely against anything that is not compatible with vibe.d ;)


My apologies, something rather the other direction. Instead of forcing 
compat with vibe.d, going to vibe.d and say: "here is our standard 
event-loop, it has everything you need, you'll need to use it for all 
the other goodness to work". I know others can make good arguments about 
why the vibe event-loop is insufficient, and I'll let them make them. 
(Something about not supporting GUI loops, paging Mr. Cattermole). If 
that is really the case I don't see how being entirely vibe.d compatible 
and meeting the universal standard requirements of Phobos is possible.

There would need to be a requirements gathering phase so that the 
community as a whole can bring their use-cases before we dove into code.


 I actually don't think the slow updates are huge problem as these DB
 interface libraries are pretty slow to change themselves. For example,
 ADO.NET didn't change significantly from it's 1.0 release until the
 arrival of Async/Await in .NET 4.5, which was 10 years later. The
 biggest addition prior to Async was TVP support and that was tiny
 comparatively and came in 2005.

 The libpq5 interface has barely changed in years. I can't speak to
 MySQL but IIRC it hasn't changed much either, at least not in a way
 that would effect the abstraction layer.


 I'm more concerned that I don't think we'll manage to implement a
 complete API and 100% bug free at the first try.


Depends on how one defines first try. Phobos as a pretty solid process 
for making sure these things are reasonably bug free. And near as I can 
tell, Phobos is pretty good about accepting bug fixes.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
Oct 26 2017
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 26/10/2017 11:25 AM, Adam Wilson wrote:

 On 10/25/17 23:57, Jacob Carlborg wrote:

 On 2017-10-26 00:53, Adam Wilson wrote:


 This of course makes the assumption that we clean-room our own
 protocol implementations which I am entirely against. Better to use
 what already exists.


 I'm entirely against anything that is not compatible with vibe.d ;)


 
 My apologies, something rather the other direction. Instead of forcing 
 compat with vibe.d, going to vibe.d and say: "here is our standard 
 event-loop, it has everything you need, you'll need to use it for all 
 the other goodness to work". I know others can make good arguments about 
 why the vibe event-loop is insufficient, and I'll let them make them. 
 (Something about not supporting GUI loops, paging Mr. Cattermole). If 
 that is really the case I don't see how being entirely vibe.d compatible 
 and meeting the universal standard requirements of Phobos is possible.
 
 There would need to be a requirements gathering phase so that the 
 community as a whole can bring their use-cases before we dove into code.


The problem isn't the event loop design.
Its a fairly solved problem.

The way vibe.d's works is very specific to their use case (which isn't 
wrong if you only consider them). You can't 'hook' into it. Which makes 
it very undesirable for Phobos. Since it won't cover most use cases. 
Even if the source they are using is compatible with an external GUI 
event loop (which it should be for Windows from what I've read).

So I wouldn't be starting with vibe.d's event loop model. Quite to the 
contrary, kill it. Build something that will last throughout the ages 
for everyone and put this problem to rest.
Oct 26 2017
prev sibling next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, October 26, 2017 03:25:24 Adam Wilson via Digitalmars-d wrote:

 On 10/25/17 23:57, Jacob Carlborg wrote:

 I'm more concerned that I don't think we'll manage to implement a
 complete API and 100% bug free at the first try.


 Depends on how one defines first try. Phobos as a pretty solid process
 for making sure these things are reasonably bug free. And near as I can
 tell, Phobos is pretty good about accepting bug fixes.


The bigger problem is API bugs. The review process is rigorous enough to
weed out a lot of stuff, but the end result is typically an API that we
_think_ looks good but usually hasn't seen much real world use (much as its
design may be based on real world experience). And if it turns out that the
API has problems, it can be very difficult to fix that in a way that doesn't
break code. Sometimes, we're able to reasonably do something about it and
sometimes not. In theory, std.experimental would mitigate that, and that's
where anything that was accepted would go first, but the process for getting
new modules into Phobos is very much geared towards getting an API right up
front rather than getting something that's close and iterating towards where
it ultimately should be.

What would probably be better in general would be to be writing stuff that
ends up on code.dlang.org first and gets some real world use there and then
look at getting it into Phobos later rather than aiming directly for Phobos.
Not only would that likely help lead towards better code being in Phobos,
but we'd still get useful stuff even if it didn't make it into Phobos.

- Jonathan M Davis
Oct 26 2017
parent reply Adam Wilson <flyboynw gmail.com> writes:
On 10/26/17 17:51, Jonathan M Davis wrote:

 On Thursday, October 26, 2017 03:25:24 Adam Wilson via Digitalmars-d wrote:

 On 10/25/17 23:57, Jacob Carlborg wrote:

 I'm more concerned that I don't think we'll manage to implement a
 complete API and 100% bug free at the first try.


 Depends on how one defines first try. Phobos as a pretty solid process
 for making sure these things are reasonably bug free. And near as I can
 tell, Phobos is pretty good about accepting bug fixes.


 The bigger problem is API bugs. The review process is rigorous enough to
 weed out a lot of stuff, but the end result is typically an API that we
 _think_ looks good but usually hasn't seen much real world use (much as its
 design may be based on real world experience). And if it turns out that the
 API has problems, it can be very difficult to fix that in a way that doesn't
 break code. Sometimes, we're able to reasonably do something about it and
 sometimes not. In theory, std.experimental would mitigate that, and that's
 where anything that was accepted would go first, but the process for getting
 new modules into Phobos is very much geared towards getting an API right up
 front rather than getting something that's close and iterating towards where
 it ultimately should be.

 What would probably be better in general would be to be writing stuff that
 ends up on code.dlang.org first and gets some real world use there and then
 look at getting it into Phobos later rather than aiming directly for Phobos.
 Not only would that likely help lead towards better code being in Phobos,
 but we'd still get useful stuff even if it didn't make it into Phobos.

 - Jonathan M Davis


I understand the concern, however, I can see potential mitigations. For 
one, steal an API concept from somewhere else. I've had reasonable 
success so far stealing ADO.NET and the refactoring it into something 
more idiomatic. Using that as a starting point gave me a pretty good 
understanding of what was needed and it gave me a prototype API that has 
been battle-tested.

Has anything from code.dlang.org been pulled into Phobos yet? I'm not 
aware of anything. code.dlang.org is where Phobos projects go to quietly 
die in obscurity.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
Oct 26 2017
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, October 26, 2017 19:19:57 Adam Wilson via Digitalmars-d wrote:

 On 10/26/17 17:51, Jonathan M Davis wrote:

 On Thursday, October 26, 2017 03:25:24 Adam Wilson via Digitalmars-d 




wrote:

 On 10/25/17 23:57, Jacob Carlborg wrote:

 I'm more concerned that I don't think we'll manage to implement a
 complete API and 100% bug free at the first try.


 Depends on how one defines first try. Phobos as a pretty solid process
 for making sure these things are reasonably bug free. And near as I can
 tell, Phobos is pretty good about accepting bug fixes.


 The bigger problem is API bugs. The review process is rigorous enough to
 weed out a lot of stuff, but the end result is typically an API that we
 _think_ looks good but usually hasn't seen much real world use (much as
 its design may be based on real world experience). And if it turns out
 that the API has problems, it can be very difficult to fix that in a
 way that doesn't break code. Sometimes, we're able to reasonably do
 something about it and sometimes not. In theory, std.experimental would
 mitigate that, and that's where anything that was accepted would go
 first, but the process for getting new modules into Phobos is very much
 geared towards getting an API right up front rather than getting
 something that's close and iterating towards where it ultimately should
 be.

 What would probably be better in general would be to be writing stuff
 that ends up on code.dlang.org first and gets some real world use there
 and then look at getting it into Phobos later rather than aiming
 directly for Phobos. Not only would that likely help lead towards
 better code being in Phobos, but we'd still get useful stuff even if it
 didn't make it into Phobos.

 - Jonathan M Davis


 I understand the concern, however, I can see potential mitigations. For
 one, steal an API concept from somewhere else. I've had reasonable
 success so far stealing ADO.NET and the refactoring it into something
 more idiomatic. Using that as a starting point gave me a pretty good
 understanding of what was needed and it gave me a prototype API that has
 been battle-tested.

 Has anything from code.dlang.org been pulled into Phobos yet? I'm not
 aware of anything. code.dlang.org is where Phobos projects go to quietly
 die in obscurity.


Nothing has been pulled into Phobos from anywhere in a while. The last thing
was probably checkedint, and before that it was std.allocator - both of
which came from Andrei. ndslice was in std.experimental for a little while
but Ilya removed it, because he decided that it wasn't working to have it in
Phobos, because he couldn't continue to improve on it there. Beyond that, I
think that the last thing was the logger, which has just languished in
std.experimental. It's my understanding that it needs some more work, but
basically, once we added std.experimental, nothing has made it into Phobos
proper. And over the last few years, there haven't been many attempts to get
anything into Phobos, so not much has even made it into std.experimental.

There was the GSoC project for a new XML parser, but that project seems to
have died after the student got too busy with school after GSoC, and Sonke's
std.json replacement didn't finish making its way through the review
process, and I think that Sonke has given up on getting it in (if I
understand correctly, there was just too much disagreement over what the
std.json replacement should look like). Overall, people have just stopped
trying to get major stuff into Phobos. New functions get added to existing
modules, but pretty much no one seems to want to go through the Phobos
review process to get anything into Phobos.

code.dlang.org is where folks put anything that they're doing that they want
to make available, and IIRC, the only item from there that's even attempted
to make it into Phobos was Sonke's JSON parser. Much as some folks continue
to talk about getting some piece of functionality into Phobos, no one is
trying if it's anything major.

So, it's not like Phobos projects are going to code.dlang.org to die. In
general, they simply aren't even being attempted, and any serious projects
that do exist are on code.dlang.org. Some do sit there unfinished (e.g.
std_experimental_xml), but largely, the authors just don't want to go to the
effort of getting the code into Phobos.

And honestly, in general, at this point, I don't think that I'd want to
bother either. It's quite a lot of work to get something through the Phobos
review process, and once it's in Phobos, you lose control over it. If I work
on something major, I can just put it up on code.dlang.org, and anyone who
wants to use it can, and I have full control over what I do with the code
base without having to get approval from Andrei or anyone else as to what I
do with it. So, unless we're talking about something that practically needs
to be in the standard library, I doubt that I'd bother trying even if I
wrote the code and was maintaining it. And most stuff really doesn't need to
be in the standard library, much as it might be nice.

But even if the goal is to get some of this stuff into Phobos, I still think
that we're better off putting it up on code.dlang.org first and getting it
battle-tested instead of pushing to get it put directly into Phobos.

- Jonathan M Davis
Oct 26 2017
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-10-26 12:25, Adam Wilson wrote:


 My apologies, something rather the other direction. Instead of forcing 
 compat with vibe.d, going to vibe.d and say: "here is our standard 
 event-loop, it has everything you need, you'll need to use it for all 
 the other goodness to work". I know others can make good arguments about 
 why the vibe event-loop is insufficient, and I'll let them make them. 


My concern is not about the event loop, it's about asynchronous IO. 
vibe.d needs to use asynchronous IO and I assume that's regardless what 
kind of event loop implementation is used. Does all the existing 
database drivers that you want to use support asynchronous IO?

-- 
/Jacob Carlborg
Oct 27 2017
parent Adam Wilson <flyboynw gmail.com> writes:
On 10/27/17 00:18, Jacob Carlborg wrote:

 On 2017-10-26 12:25, Adam Wilson wrote:


 My apologies, something rather the other direction. Instead of forcing
 compat with vibe.d, going to vibe.d and say: "here is our standard
 event-loop, it has everything you need, you'll need to use it for all
 the other goodness to work". I know others can make good arguments
 about why the vibe event-loop is insufficient, and I'll let them make
 them.


 My concern is not about the event loop, it's about asynchronous IO.
 vibe.d needs to use asynchronous IO and I assume that's regardless what
 kind of event loop implementation is used. Does all the existing
 database drivers that you want to use support asynchronous IO?


PgSQL/MySQL/MSSQL all do, I think that covers about 90% of usage. IIRC 
Oracle does as well.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
Oct 27 2017
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 10/23/2017 06:02 PM, Adam Wilson wrote:

 On 10/23/17 05:08, Jacob Carlborg wrote:

 * Database drivers for the common databases (PostgreSQL, MySQL, SQLite)
 compatible with vibe.d
 * Database driver abstraction on top of the above drivers, perhaps some
 lightweight ORM library


 
 I've been looking pretty extensively at these two items recently.
 
 If the database drivers are compatible with Vibe.d AND we wish to 
 provide a common abstraction layer for them (presumably via Phobos) then 
 order for the abstraction layer to aware of the whether the driver is 
 making a blocking or non-blocking call we must include Vibe.D in the 
 abstraction layer.


Mysql-native is vibe.d-compatible, but also works when you have zero 
vibe.d dependencies (in which case it switches to Phobos's sockets 
instead of using Vibe's sockets.)
May 05 2018
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 05/05/2018 02:30 PM, Nick Sabalausky (Abscissa) wrote:

 On 10/23/2017 06:02 PM, Adam Wilson wrote:

 On 10/23/17 05:08, Jacob Carlborg wrote:

 * Database drivers for the common databases (PostgreSQL, MySQL, SQLite)
 compatible with vibe.d
 * Database driver abstraction on top of the above drivers, perhaps some
 lightweight ORM library


 I've been looking pretty extensively at these two items recently.

 If the database drivers are compatible with Vibe.d AND we wish to 
 provide a common abstraction layer for them (presumably via Phobos) 
 then order for the abstraction layer to aware of the whether the 
 driver is making a blocking or non-blocking call we must include 
 Vibe.D in the abstraction layer.


 
 Mysql-native is vibe.d-compatible, but also works when you have zero 
 vibe.d dependencies (in which case it switches to Phobos's sockets 
 instead of using Vibe's sockets.)


Sorry, didn't notice this was a half-year-old thread.
May 05 2018
prev sibling next sibling parent reply Laeeth Isharc <laeeth laeeth.com> writes:
On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:

 I just read the following two week-old comment on the ldc issue 
 tracker, when someone tried to run D on Alpine linux:

 "For now everything works(?) but I think the process could be 
 improved.. Would be really cool to have LDC easily building 
 alpine containers + static D binaries for microservice and 
 tooling development. I'm pretty tired of reading Go code :)"
 https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550

 It strikes me that microservices are a great way for new 
 programming languages like D to get tried and gain some uptake, 
 but that D might not be that easy to deploy to that scenario 
 yet.

 So this is a question for those deploying microservices, as I'm 
 not in that field, what can the D devs do to make it as easy as 
 possible to get D microservices up and running, make some 
 Docker and Alpine containers with ldc/dub/vibe.d preinstalled 
 publicly available?  What else, what kinds of libraries do you 
 normally use?

 This is a niche that D and all newer languages should target.  
 How do we do it?


We're going a bit in that direction, although it's a different 
thing in our kind of industry from a web company - we have fewer 
services and many fewer requests, but latency matters more for 
example.  I was thinking we might use Go for some services that 
integrate and monitor things, but we could also use D.

https://jobs.github.com/positions/8e98eac8-b504-11e7-9da8-0737a3dcef18

 From the comment:
"Would be really cool to have LDC easily building alpine 
containers + static D binaries"

How can we generate a static binary ?  I asked about this before, 
and the response was that it's a bad idea because of security 
vulns and so on.  True if you are running on a conventional Linux 
host.  But on the other hand, it's not that great if when the 
system phobos is upgraded all the D binaries break.  You can 
write a script using rdmd or dub scripting feature, but that 
doesn't work for more complex programs.

And on the other hand, for deployment, it's much easier to copy 
over one binary.  (In a sense it's funny that the question was 
asked in the context of containers, because containers are kind 
of an alternative to having a single binary).  When you're 
properly set-up of course it doesn't matter, but when you're 
moving towards that, a single binary is much easier.

I guess I can figure out the answer to this question easily 
enough, but I think giving people the option might help with 
adoption of D for micro services.  For example it's really just 
not that fun to make an AWS Lambda using D - being able to easily 
build a static binary would make the process much more pleasant.

I wrote this up a while back:
http://awslambda-d.readthedocs.io/en/latest/

"Since dmd links phobos dynamically on linux, and phobos/druntime 
aren't installed on the AWS lambda server, we will need to upload 
these to the servers and tell the application where to find them. 
(I should really have appended to LD_LIBRARY_PATH as I did with 
PATH).

Now one can follow the regular instructions for AWS Lambda: 
create a .zip file with the D binary, the JS file, and the 
following libraries (update version numbers as appropriate):

libcrypto.so.1.0.0
libphobos2.so.0.67
libevent-2.0.so.5
libssl.so.1.0.0
"

Alpine is nice - would be good to have this as a standard target 
in time.


Laeeth.
Oct 23 2017
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2017-10-23 14:13, Laeeth Isharc wrote:


 How can we generate a static binary ?


It's already supported by LDC, using the -static flag. This Linux binary 
[1] is statically linked.

[1] https://github.com/jacob-carlborg/remarkify/releases

-- 
/Jacob Carlborg
Oct 23 2017
prev sibling next sibling parent Cym13 <cpicard openmailbox.org> writes:
On Monday, 23 October 2017 at 12:13:12 UTC, Laeeth Isharc wrote:

 ...
 And on the other hand, for deployment, it's much easier to copy 
 over one binary.  (In a sense it's funny that the question was 
 asked in the context of containers, because containers are kind 
 of an alternative to having a single binary).  When you're 
 properly set-up of course it doesn't matter, but when you're 
 moving towards that, a single binary is much easier.
 ...


A bit OT but this part made me think of [1] of which you may 
appreciate the insight. Right now I kind of feel like with D we 
get the downside in size of fat binaries without the upside in 
portability (although solutions exist for the dedicated and 
patient ones). This might proove an important drawback in the 
future... we'll see.

[1]: 
http://www.smashcompany.com/technology/why-would-anyone-choose-docker-over-fat-binaries
Oct 25 2017
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, October 23, 2017 12:13:12 Laeeth Isharc via Digitalmars-d wrote:

 How can we generate a static binary ?  I asked about this before,
 and the response was that it's a bad idea because of security
 vulns and so on.  True if you are running on a conventional Linux
 host.  But on the other hand, it's not that great if when the
 system phobos is upgraded all the D binaries break.  You can
 write a script using rdmd or dub scripting feature, but that
 doesn't work for more complex programs.


I believe that it works just fine if you do the linking step yourself, but
dmd doesn't properly handle it with -L-static:

https://issues.dlang.org/show_bug.cgi?id=6952

I have no idea how easy it would be to the linking manually with dub.

Alternatively, I believe that you can statically link against any 3rd party
libraries that you're using by linking to the .a file rather than by the
library name. That won't work for glibc, but much as I'd love to statically
link that, it's my understanding that doing so is generally a bad idea. I
don't recall the reasons though - and with the increased stability in
linux's ABI, I would have thought that it would be far less of an issue than
it would have been in the past. Certainly, the concept of being able to
statically link a program and then just run it on most any recent linux
distro without recompiling it sounds very appealing. Yes, you'll have to
recompile it for any security updates to any of your dependencies, whereas
that can be avoided with dynamic linking, but depending on what you're
doing, that often isn't really an issue, and it can be managed if it is
(heck, you're doing basically the same thing when maintaining something like
a docker image).

- Jonathan M Davis
Oct 26 2017
prev sibling parent reply aberba <karabutaworld gmail.com> writes:
On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:

 I just read the following two week-old comment on the ldc issue 
 tracker, when someone tried to run D on Alpine linux:

 "For now everything works(?) but I think the process could be 
 improved.. Would be really cool to have LDC easily building 
 alpine containers + static D binaries for microservice and 
 tooling development. I'm pretty tired of reading Go code :)"
 https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550

 It strikes me that microservices are a great way for new 
 programming languages like D to get tried and gain some uptake, 
 but that D might not be that easy to deploy to that scenario 
 yet.


Its the future. Netflix, Google, Facebook, etc. all have open 
source tools around microservices. Its currently ruled by 
JavaScript > Go > Java. JavaScript being the leader.

They have these in common:
1. Easy to deploy their code in docker containers including 
alpine Linux.
2. They have APIs for major cloud services. Both official and 
third-party.
3. Good support for networking. HTTP, Websockets, IPC*, etc. 
Mostly HTTP.

That's it. The major advantages besides the hype.

Lets see about D based on these requirements.
1. Kind of. There are dmd-ldc-dub docker images on docker hub. 
One by sociomantic. None using Alpine Linux as base though. Most 
people prefer alpine cus its lightweight (not a requirement).
2. Nope. Official APIs comes when there's mass adoption for that 
language and has good ROI. No complete and production ready cloud 
services API that I know of. Seems D users in that area roll out 
something on their own for private use with features they need. 
But most cloud PaaS provide support for custom runtimes which D 
has one for Heroku. Major candidate is Google AppEngine.
3. Phobos doesn't have a D HTTP API. Its sad but we have 
"requests" at code.dlang.org which works. We have vibe.d for http 
servers (aka nodejs of D) and seems popular based on threads 
about it and downloads.


 So this is a question for those deploying microservices, as I'm 
 not in that field, what can the D devs do to make it as easy as 
 possible to get D microservices up and running, make some 
 Docker and Alpine containers with ldc/dub/vibe.d preinstalled 
 publicly available?  What else, what kinds of libraries do you 
 normally use?


There several database APIs at cods.dlang.org. I don't know why 
some complain about no db APIs.


 This is a niche that D and all newer languages should target.  
 How do we do it?


Good question.
Oct 28 2017
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Saturday, 28 October 2017 at 14:55:25 UTC, aberba wrote:

 On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:

 I just read the following two week-old comment on the ldc 
 issue tracker, when someone tried to run D on Alpine linux:

 "For now everything works(?) but I think the process could be 
 improved.. Would be really cool to have LDC easily building 
 alpine containers + static D binaries for microservice and 
 tooling development. I'm pretty tired of reading Go code :)"
 https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550

 It strikes me that microservices are a great way for new 
 programming languages like D to get tried and gain some 
 uptake, but that D might not be that easy to deploy to that 
 scenario yet.


 Its the future.


Highly doubt that. It really depend on the scale of your 
operations.


 Netflix, Google, Facebook, etc. all have open source tools 
 around microservices. Its currently ruled by JavaScript > Go > 
 Java. JavaScript being the leader.

 They have these in common:
 1. Easy to deploy their code in docker containers including 
 alpine Linux.


Interestingly while Docker may seem all the rage in startups I 
find its use limited to test environments in the real world.

Also Java fat jars were super easy to deploy ages before docker. 
They are also a great deal smaller.


 2. They have APIs for major cloud services. Both official and 
 third-party.
 3. Good support for networking. HTTP, Websockets, IPC*, etc. 
 Mostly HTTP.


Honestly APIs these days can be written in anything that is able 
to put together a few HTTP responses. Unless you are doing 
serious work like:
- DBs
- Search engines
- ML pipelines
- Real-time event processing systems
....

Any semimodern language/technology with a several hosts can 
manage to saturate 1Gbit link. Some take a certain amount of 
tuning others work out of the box. If you go for 40gbit/s it may 
be important to choose the right technology otherwise it’s all a 
matter of taste.
Oct 28 2017