www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Static linking D apps for distribution to the cloud, etc.

reply David J Kordsmeier <dkords gmail.com> writes:
Hi forums, I have what should be a simple question.  I have an 
application which I would like to distribute to the cloud onto a 
linux system which I will not have root access.  I thought it 
should be easier for me to perform static linking of this D app 
instead of packing the application with library dependencies.

If I try to do this, using my most simple example below, with 
ldc2 (1.15.0) using the --static flag, I get various warnings: 
warning: Using 'dlopen' in statically linked applications 
requires at runtime the shared libraries from the glibc version 
used for linking
warning: Using 'gethostbyaddr' in statically linked applications 
requires at runtime the shared libraries from the glibc version 
used for linking

If I use gdc (4.9.0 2.065) for this, I get linker errors, such 
as: undefined reference to `curl_easy_setopt'

I'll go back and review the best practices for using curl with 
phobos.  I still am struggling to make this work, as something 
that would be relatively simple in C to perform (statically link 
to any libraries I would like).  Without being an expert on 
phobos library construction, what is the recommended practice for 
app distribution?  Does anyone have a great sample project to 
point to?  I've looked at the dlang-tour 
https://github.com/dlang-tour/core/blob/master/dub.sdl , and this 
indeed seems like the right track, however, I still get the curl 
warnings and the app won't run.

I have also tried building and dynamically linking, which works 
fine, but I've had trouble pulling the dependent libraries for 
distribution, and namely having trouble with finding curl at 
runtime.

```
import std.net.curl, std.stdio;
int main() {
   auto http = HTTP("dlang.org");
   http.onReceiveHeader = (in char[] key, in char[] value) { 
writeln(key, ": ", value);  };
   http.onReceive = (ubyte[] data) { /+ drop +/ return 
data.length;  };
   http.perform();
   return 0;
}
```
Oct 08
next sibling parent reply kinke <noone nowhere.com> writes:
On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
wrote:
 I'll go back and review the best practices for using curl with 
 phobos.  I still am struggling to make this work, as something 
 that would be relatively simple in C to perform (statically 
 link to any libraries I would like).
Phobos 2.069 switched to loading libcurl at runtime, see https://dlang.org/changelog/2.069.0.html#curl-dynamic-loading. It mentions that it's still possible to link it statically into the executable.
Oct 08
parent reply Jon Degenhardt <jond noreply.com> writes:
On Tuesday, 8 October 2019 at 19:35:03 UTC, kinke wrote:
 On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
 wrote:
 I'll go back and review the best practices for using curl with 
 phobos.  I still am struggling to make this work, as something 
 that would be relatively simple in C to perform (statically 
 link to any libraries I would like).
Phobos 2.069 switched to loading libcurl at runtime, see https://dlang.org/changelog/2.069.0.html#curl-dynamic-loading. It mentions that it's still possible to link it statically into the executable.
The TSV Utilities have used static linking for Linux prebuilt release binaries for a couple years now without any problems. The tools do not use curl or gethostbyaddr, but there are still plenty of warning messages. (Example: https://travis-ci.org/eBay/tsv-utils/jobs/593432965) Static linking was suggested by Jacob Carlborg, who pointed out that the pre-built binaries wouldn't run on all Linux distributions. Static linking fixed it, with no subsequent problems reported (https://github.com/eBay/tsv-utils/issues/67).
Oct 08
parent reply David J Kordsmeier <dkords gmail.com> writes:
On Tuesday, 8 October 2019 at 22:23:39 UTC, Jon Degenhardt wrote:
 On Tuesday, 8 October 2019 at 19:35:03 UTC, kinke wrote:
 On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
 wrote:
 I'll go back and review the best practices for using curl 
 with phobos.  I still am struggling to make this work, as 
 something that would be relatively simple in C to perform 
 (statically link to any libraries I would like).
Phobos 2.069 switched to loading libcurl at runtime, see https://dlang.org/changelog/2.069.0.html#curl-dynamic-loading. It mentions that it's still possible to link it statically into the executable.
The TSV Utilities have used static linking for Linux prebuilt release binaries for a couple years now without any problems. The tools do not use curl or gethostbyaddr, but there are still plenty of warning messages. (Example: https://travis-ci.org/eBay/tsv-utils/jobs/593432965) Static linking was suggested by Jacob Carlborg, who pointed out that the pre-built binaries wouldn't run on all Linux distributions. Static linking fixed it, with no subsequent problems reported (https://github.com/eBay/tsv-utils/issues/67).
Thank you all forum folks for the replies. My use case for curl is simply HTTP/HTTPS, not libcurl specifically. That said, I was making a best effort to not use external dependencies per my requirements. My feeling is that a platform should provide needs for modern computing, and web is a part of that. I hope this question "goes away" in the future and there becomes a modern library alternative to the std.net for http as standard part of phobos. So a number of great responses here, and I am going to explore the export option as well as the http2 (from arsd) option. John, I'll check out the tsv-utils build as well, your CI server set up looks really great. I am very interested to learn the best practices of distribution of prebuilt D apps and libraries.
Oct 08
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 9 October 2019 at 01:10:23 UTC, David J Kordsmeier 
wrote:
 HTTP/HTTPS

 as well as the http2 (from arsd) option.
So let me warn you ahead of time that my http2.d still has an outside dependency: OpenSSL. And that can sometimes be tricky as openssl broke binary compatibility between 1.1 and 1.0. I was planning on making it dynamically load that to work around it :S so your -static build MIGHT work just I'm not sure. TLS/SSL is one of the few things I don't DIY and feel a third-party dependency is justified on, I just try to make it seamless and... it usually works on Linux but not always. Helper switches: -version=older_openssl if you get linker errors and have 1.0 on the system or -version=newer_openssl if you get linker errors and have 1.1 on. I *have* statically linked OpenSSL with this library before so I know it works, just I also had to jump through a few hoops to make it work. However I was on Windows then so Linux might be easier... or might be harder. I don't know. but if it does work that also gives a nice way around the 1.0/1.1 problem! if you do try it lemme know how it goes plz.
Oct 08
parent David J Kordsmeier <dkords gmail.com> writes:
On Wednesday, 9 October 2019 at 01:29:59 UTC, Adam D. Ruppe wrote:
 On Wednesday, 9 October 2019 at 01:10:23 UTC, David J 
 Kordsmeier wrote:
 HTTP/HTTPS

 as well as the http2 (from arsd) option.
So let me warn you ahead of time that my http2.d still has an outside dependency: OpenSSL. And that can sometimes be tricky as openssl broke binary compatibility between 1.1 and 1.0. I was planning on making it dynamically load that to work around it :S so your -static build MIGHT work just I'm not sure. TLS/SSL is one of the few things I don't DIY and feel a third-party dependency is justified on, I just try to make it seamless and... it usually works on Linux but not always. Helper switches: -version=older_openssl if you get linker errors and have 1.0 on the system or -version=newer_openssl if you get linker errors and have 1.1 on. I *have* statically linked OpenSSL with this library before so I know it works, just I also had to jump through a few hoops to make it work. However I was on Windows then so Linux might be easier... or might be harder. I don't know. but if it does work that also gives a nice way around the 1.0/1.1 problem! if you do try it lemme know how it goes plz.
Thank you Adam for your work in the D community. Thanks for the heads up. Yes, definitely OpenSSL (or any security library) isn't had for free in a secure Internet and I will need to make some effort to ensure I get that working. In my limited use case (does not need to run ANYWHERE, but in a specific environment) it should be possible. Will update as soon as I know.
Oct 08
prev sibling next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
wrote:
 Hi forums, I have what should be a simple question.  I have an 
 application which I would like to distribute to the cloud onto 
 a linux system which I will not have root access.  I thought it 
 should be easier for me to perform static linking of this D app 
 instead of packing the application with library dependencies.

 [...]
What do you need from Curl? There is also the option to not use Curl but e.g. the http client from here https://github.com/adamdruppe/arsd/blob/master/http2.d There is also vibe-d and several other http implementations available. Kind regards Andre
Oct 08
prev sibling next sibling parent mipri <mipri minimaltype.com> writes:
On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
wrote:
 warning: Using 'gethostbyaddr' in statically linked 
 applications requires at runtime the shared libraries from the 
 glibc version used for linking
This is a well-known problem with GNU libc. Your best bets are 1. accepting dynamic linking on the target platform, and setting up a build system that's similar enough to it that deployment is easy. Maybe setting LD_LIBRARY_PATH to point to deployed libraries that you can't get installed on the taget. 2. using something like flatpak or appimage. 3. using DNS resolution from a library outside of libc, and hacking phobos to use that instead of gethostbyaddr.
Oct 08
prev sibling next sibling parent Arun Chandrasekaran <aruncxy gmail.com> writes:
On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
wrote:
 Hi forums, I have what should be a simple question.  I have an 
 application which I would like to distribute to the cloud onto 
 a linux system which I will not have root access.  I thought it 
 should be easier for me to perform static linking of this D app 
 instead of packing the application with library dependencies.

 [...]
Take a look at https://github.com/ikod/dlang-requests It offers more features than the libcurl wrapper in std.net.curl.
Oct 08
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
wrote:
 Hi forums, I have what should be a simple question.  I have an 
 application which I would like to distribute to the cloud onto 
 a linux system which I will not have root access.  I thought it 
 should be easier for me to perform static linking of this D app 
 instead of packing the application with library dependencies.
Alternatively you can build inside a Docker container with the same distribution and version as the machine running in the cloud. Then you should be able to do dynamic linking and run the binary in the cloud without any problem. -- /Jacob Carlborg
Oct 09
parent Laeeth Isharc <Laeeth laeeth.com> writes:
On Wednesday, 9 October 2019 at 09:33:18 UTC, Jacob Carlborg 
wrote:
 On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
 wrote:
 Hi forums, I have what should be a simple question.  I have an 
 application which I would like to distribute to the cloud onto 
 a linux system which I will not have root access.  I thought 
 it should be easier for me to perform static linking of this D 
 app instead of packing the application with library 
 dependencies.
Alternatively you can build inside a Docker container with the same distribution and version as the machine running in the cloud. Then you should be able to do dynamic linking and run the binary in the cloud without any problem. -- /Jacob Carlborg
And of course what seems to be merely a statically linked binary can be a container with its own filesystem and carry dependencies with it. https://blog.jessfraz.com/post/nerd-sniped-by-binfmt_misc/ https://github.com/genuinetools/binctr
Oct 09
prev sibling parent Jacob Carlborg <doob me.com> writes:
On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
wrote:
 Hi forums, I have what should be a simple question.  I have an 
 application which I would like to distribute to the cloud onto 
 a linux system which I will not have root access.  I thought it 
 should be easier for me to perform static linking of this D app 
 instead of packing the application with library dependencies.
You can also try to build the binary on Alpine and do static linking. Alpine uses musl as the C standard library instead of GNU. musl is designed for static linking. -- /Jacob Carlborg
Oct 09