www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Downloading files over TLS

reply Jacob Carlborg <doob me.com> writes:
Downloading files over TLS. This seems that it's something that 
should be quite simple to do. My high level goals are 
cross-platform and easy distribution. I don't need anything fancy 
just a simple API like this:

download("https://url.com", "/local/file");

Because of these goals, I have a few requirements, which I don't 
think are unreasonable:

* Cross-platform (macOS, Linux, Windows, possible FreeBSD)

* Minimal runtime requirements (I don't mind a few compile time 
requirements). I also don't mind runtime requirements that are 
guaranteed to exist

* Ideally using the platform provided TLS implementation, or at 
least platform provided certificates (I've been bitten several 
times due to old certificates making request fail)

* On Linux I would really like to have a fully statically linked 
binary, this will make distribution easier

* Due to the previous requirement, I think using musl will be 
required. For most parts glibc works fine for static linking, but 
I think in the case of networking it does not work. The DNS 
lookup (or something like that) doesn't work with static linking

I think the main problem of all this is picking the right TLS 
implementation.

The obvious candidate is std.net.curl, but I'm going to explain 
why that is not ideal, at least not how it's currently 
implemented. std.net.curl relies on OpenSSL, which is deprecated 
on several platforms. On several platforms it's not the main, 
platform provided TLS implementation. As far as I know, it also 
provides its own certificates and doesn't rely on the ones 
provided by the platform.

I going to break down some alternatives per platform.

macOS:
* std.net.curl. Depends on libcurl and OpenSSL. OpenSSL has been 
deprecated on macOS for many years. In one wants to use OpenSSL 
anyway, one need to ship it with the binary (statically link it). 
For other problems, see above

* requests [1]. This is a potential better alternative than 
std.net.curl since it doesn't rely on libcurl. But, it still 
relies on OpenSSL, so it has the same problems as std.net.curl

* vibe.d [2]. Does not depend on libcurl, but does depend on 
OpenSSL. But, it also supports Botan, which is promising. I'll 
discuss Botan separately

* Hunt [3]. Same problem as the other ones, relies on OpenSSL

* Arsd [4]. Relies on OpenSSL

* SecureTransport [5]. This is an Apple specific library provided 
by the platform. It seems to more or less follow the same idea as 
OpenSSL, it's independent on the transport layer and works with 
BSD sockets. Initially this looks perfect, it's provided by the 
platform and uses certificates provided by the platform. The main 
problem is that it has been deprecated. It also only supports TLS 
up to version 1.2 and since it's deprecated, it's likely that it 
won't support any newer versions

* Network [6]. This is an Apple specific library provided by the 
platform. This is the recommend alternative to SecureTransport. 
The problem is that this is not just an alternative TLS 
implementation, it's a completely different alternative to BSD 
sockets. The API is completely different and will require some 
extra layers to to provide a cross-platform API. This means that 
I cannot use any of the existing library to just add TLS, it will 
be a completely different implementation, which might be ok. 
Another big problem is that it only available on macOS 10.14 and 
later. I have not decided yet if this is acceptable or not

* NSURLSession [7]. This is an  Apple specific Objective-C class, 
provided by the platform. It's higher level than the Network 
framework and is available on all versions of macOS. This will 
again require a some extra layers to hide the platform specific 
API. There's also a problem with LDC which has very limited 
support for Objective-C interoperability.

Linux:
As far as I know, OpenSSL (or some version of it, like LibreSSL) 
is what's provided by the platform. So I guess that has to be 
acceptable on Linux.

* std.net.curl. The main problem with std.net.curl on Linux is 
that it uses `dlopen` to load the TLS library. `dlopen` doesn't 
work for a statically linked executable. Side note, `dlopen` 
doesn't work on iOS (for other reasons) so it would be nice if 
this could be fixed

* requests. Seems to do that same as std.net.curl, uses `dlopen`

* For the other alternatives (mentioned in the macOS section), I 
haven't investigated if they use `dlopen` or not

Windows:
I don't know that much of this platform.

* std.net.curl and basically all other options already mentioned 
relies on OpenSSL, which is not provided by the platform

* SChannel. As far as I know, this the the platform provided 
implementation of TLS on Windows.

* Are there any high level APIs, like NSURLSession, on Windows 
that can be used to download files?

Botan:

This seems like a good candidate. It even exists a full D port of 
it [8].

* This is not used by any specific platform but it can use the 
platform provided certificates, which is a plus.

* The downside is that the code to access the platform provided 
certificates has not been ported to D (or it was added to a newer 
version of the C++ implementation).

* I think the the D port of Botan does not verify certificates, 
which is not great. It might be due to the previous point

* I also don't know if the D port is updated to match the latest 
version of the C++ implementation. By experience I know this can 
be a huge task, so I don't expect it

* Botan only supports up to TLS 1.2 (even the C++ 
implementation). Due to the previous point it's unclear if the D 
port of Botan will get support for TLS 1.3

I know there are other alternative TLS implementation, but I have 
not looked into detail on these.

A question regarding libcurl. I know that curl (the CLI tool) can 
take advantage of SecureTransport and SChannel as alternatives to 
OpenSSL. Due to std.net.curl containing some OpenSSL related 
code, is TLS handled completely outside of libcurl? So 
std.net.curl  can't easily take advantage of SecureTransport and 
SChannel the same way as curl does?

Is there anything obvious I'm missing or why does this seem so 
difficult? Do I have too high expectations and requirements?

[1] https://code.dlang.org/packages/requests
[2] https://code.dlang.org/packages/vibe-d
[3] https://code.dlang.org/packages/hunt
[4] https://code.dlang.org/packages/arsd-official%3Ahttp
[5] 
https://developer.apple.com/documentation/security/secure_transport
[6] https://developer.apple.com/documentation/network
[7] 
https://developer.apple.com/documentation/foundation/nsurlsession
[8] https://code.dlang.org/packages/botan

--
/Jacob Carlborg
Jun 26 2020
next sibling parent reply ikod <geller.garry gmail.com> writes:
On Friday, 26 June 2020 at 10:12:09 UTC, Jacob Carlborg wrote:

 Downloading files over TLS. This seems that it's something that 
 should be quite simple to do. My high level goals are 
 cross-platform and easy distribution. I don't need anything 
 fancy just a simple API like this:

 download("https://url.com", "/local/file");


....


 Is there anything obvious I'm missing or why does this seem so 
 difficult? Do I have too high expectations and requirements?

 [1] https://code.dlang.org/packages/requests
 [2] https://code.dlang.org/packages/vibe-d
 [3] https://code.dlang.org/packages/hunt
 [4] https://code.dlang.org/packages/arsd-official%3Ahttp
 [5] 
 https://developer.apple.com/documentation/security/secure_transport
 [6] https://developer.apple.com/documentation/network
 [7] 
 https://developer.apple.com/documentation/foundation/nsurlsession
 [8] https://code.dlang.org/packages/botan

 --
 /Jacob Carlborg


Hello,

re `requests` - it uses dlopen (and variants for OSX and Windows, 
see 
https://github.com/ikod/dlang-requests/blob/master/source/requests
ssl_adapter.d#L50). The reason for dlopen is simple - compatibility with both
openssl ver 1.0 and 1.1 (which have incompatible interfaces). To solve this
problem I expose common interface for Requests internal needs, and detect and
use different underlying openssl interfaces depending on library version.

I'm sure it is possible to detect library version at build time, 
and then use static linking. It just require some time investment 
to find most acceptable solution on how to do this.

Re Windows - I'd be happy to use native SecureChannel but I have 
zero windows programming experience. I'm completely open for any 
help in this field.

Bottom line - problem with SSL/TLS libraries lies in 
incompatibility between platforms and even inside the single 
library.
Jun 26 2020
parent reply Jacob Carlborg <doob me.com> writes:
On Friday, 26 June 2020 at 11:10:27 UTC, ikod wrote:


 Hello,

 re `requests` - it uses dlopen (and variants for OSX and 
 Windows, see 
 https://github.com/ikod/dlang-requests/blob/master/source/requests
ssl_adapter.d#L50). The reason for dlopen is simple - compatibility with both
openssl ver 1.0 and 1.1 (which have incompatible interfaces). To solve this
problem I expose common interface for Requests internal needs, and detect and
use different underlying openssl interfaces depending on library version.


Oh, it's that bad. That's disappointing.


 I'm sure it is possible to detect library version at build 
 time, and then use static linking. It just require some time 
 investment to find most acceptable solution on how to do this.


I'm using a script (written in D) in my tool, DStep [1][2], to 
identify which versions of libclang is available and to select 
between dynamic link (not using `dlopen`) and static linking of 
libclang. It works pretty well.


 Re Windows - I'd be happy to use native SecureChannel but I 
 have zero windows programming experience. I'm completely open 
 for any help in this field.


I don't have much experience with programming on Windows either. 
macOS is my main platform.


 Bottom line - problem with SSL/TLS libraries lies in 
 incompatibility between platforms and even inside the single 
 library.


Yes. I'm trying to identify the best solution, which ideally 
requires the least amount of work.

[1] 
https://github.com/jacob-carlborg/dstep/blob/master/configure.d
[2] 
https://github.com/jacob-carlborg/dstep/blob/master/dub.json#L34-L41

--
/Jacob Carlborg
Jun 26 2020
parent ikod <geller.garry gmail.com> writes:
On Friday, 26 June 2020 at 11:41:27 UTC, Jacob Carlborg wrote:

 On Friday, 26 June 2020 at 11:10:27 UTC, ikod wrote:


 [...]


 Oh, it's that bad. That's disappointing.


 [...]


 I'm using a script (written in D) in my tool, DStep [1][2], to 
 identify which versions of libclang is available and to select 
 between dynamic link (not using `dlopen`) and static linking of 
 libclang. It works pretty well.



 [1] 
 https://github.com/jacob-carlborg/dstep/blob/master/configure.d
 [2] 
 https://github.com/jacob-carlborg/dstep/blob/master/dub.json#L34-L41


Thanks, will try to improve this (at least to add static openssl 
linking, as this is not first time somebody asked about it)


 --
 /Jacob Carlborg
Jun 26 2020
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
On Friday, 26 June 2020 at 10:12:09 UTC, Jacob Carlborg wrote:

 Downloading files over TLS. This seems that it's something that 
 should be quite simple to do. My high level goals are 
 cross-platform and easy distribution. I don't need anything 
 fancy just a simple API like this:

 download("https://url.com", "/local/file");


Maybe just start wget or something like that?


 * Network [6]. This is an Apple specific library provided by 
 the platform. This is the recommend alternative to 
 SecureTransport. The problem is that this is not just an 
 alternative TLS implementation, it's a completely different 
 alternative to BSD sockets. The API is completely different and 
 will require some extra layers to to provide a cross-platform 
 API. This means that I cannot use any of the existing library 
 to just add TLS, it will be a completely different 
 implementation, which might be ok. Another big problem is that 
 it only available on macOS 10.14 and later. I have not decided 
 yet if this is acceptable or not


Since you want the latest certificate storage, you intend to 
support only the latest system. Many root certificates will 
timeout now.


 * Are there any high level APIs, like NSURLSession, on Windows 
 that can be used to download files?


https://docs.microsoft.com/en-us/windows/win32/winhttp/about-winhttp
Jun 26 2020
parent Jacob Carlborg <doob me.com> writes:
On 2020-06-26 14:41, Kagamin wrote:


 Maybe just start wget or something like that?


The point was to avoid runtime dependencies.


 Since you want the latest certificate storage, you intend to support 
 only the latest system. Many root certificates will timeout now.


I didn't say the latest certificate storage. I said the platform 
provided. If the system provided certificate expires I guess a lot of 
things won't work. An alternative would be to both ship with a 
certificate and be able to use the system provided one and try both


 https://docs.microsoft.com/en-us/windows/win32/winhttp/about-winhttp


I'll take a look, thanks.

-- 
/Jacob Carlborg
Jun 26 2020
prev sibling next sibling parent reply User <user user.com> writes:
On Friday, 26 June 2020 at 10:12:09 UTC, Jacob Carlborg wrote:

 Windows:
 I don't know that much of this platform.

 * std.net.curl and basically all other options already 
 mentioned relies on OpenSSL, which is not provided by the 
 platform

 * SChannel. As far as I know, this the the platform provided 
 implementation of TLS on Windows.

 * Are there any high level APIs, like NSURLSession, on Windows 
 that can be used to download files?


It is possible to statically link libcurl into your application. 
No need to use OpenSSL as libcurl can be built with SChannel.
Jun 26 2020
parent Jacob Carlborg <doob me.com> writes:
On 2020-06-26 14:43, User wrote:


 It is possible to statically link libcurl into your application. No need 
 to use OpenSSL as libcurl can be built with SChannel.


That sounds good.

It looks like I remembered wrong. std.net.curl uses `dlopen` on libcurl, 
not OpenSSL. This might be less of an issue assuming libcurl is built 
with the platform provided TLS implementation. Just make sure it's 
possible to statically link libcurl, without using `dlopen`.

-- 
/Jacob Carlborg
Jun 26 2020
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 26 June 2020 at 10:12:09 UTC, Jacob Carlborg wrote:

 * Arsd [4]. Relies on OpenSSL


Yeah, I've been wanting to change that and use the native apis 
for years but like I just haven't been able to figure out the 
documentation of them.

Though for plain download, on Windows there's a high level 
function you can just call that's relatively easy... just the 
rest of the module needs low level access so it isn't super 
useful.

And of course Mac does it differently, I don't know much about 
there.


 The main problem is that it has been deprecated.


That's the problem of everything Apple makes. Kinda drives me 
nuts trying to keep up with their endless churn and constantly 
Think Different campaigns.


 * Are there any high level APIs, like NSURLSession, on Windows 
 that can be used to download files?


https://docs.microsoft.com/en-us/windows/win32/winhttp/winhttp-sessions-overview

I guess that is more middle level but it isn't too hard to use. I 
think there's a flat out url -> bytes function too but I don't 
remember what it is.

Regardless, the Windows functions may look familiar if you have 
done AJAX - that was based on an IE object which was based on the 
Windows API.
Jun 26 2020
parent Jacob Carlborg <doob me.com> writes:
On 2020-06-26 15:16, Adam D. Ruppe wrote:


 Yeah, I've been wanting to change that and use the native apis for years 
 but like I just haven't been able to figure out the documentation of them.


That would be nice.


 That's the problem of everything Apple makes. Kinda drives me nuts 
 trying to keep up with their endless churn and constantly Think 
 Different campaigns.


Hehe, yeah.


 https://docs.microsoft.com/en-us/windows/win32/winhttp/winht
p-sessions-overview 
 
 
 I guess that is more middle level but it isn't too hard to use. I think 
 there's a flat out url -> bytes function too but I don't remember what 
 it is.
 
 Regardless, the Windows functions may look familiar if you have done 
 AJAX - that was based on an IE object which was based on the Windows API.


Thanks, I'll take a look.

-- 
/Jacob Carlborg
Jun 26 2020