www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - std.net.curl and libcurl.so

reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
Hello all,

As some have you may have followed, I've been working on 
snap-packaging LDC.  However, I've run into an issue when it 
comes to programs that use std.net.curl.

Here's a simple example:

------------------------
void main ()
{
     import std.net.curl : get;
     auto website = "http://dlang.org/".get;
}
------------------------

When I compile with my snap-packaged LDC, the program builds, but 
segfaults when I run it.  Here's the gdb backtrace:

Thread 1 "curlget" received signal SIGSEGV, Segmentation fault.
0x000000000046b048 in gc.gc.Gcx.smallAlloc(ubyte, ref ulong, 
uint) ()
(gdb) bt

uint) ()

const(TypeInfo)) ()


const(TypeInfo)) ()

std.array.Appender!(char[]).Appender.ensureAddable(ulong) ()

std.uni.toCase!(std.uni.toLowerIndex(dchar), 1043, 
std.uni.toLowerTab(ulong), 
char[]).toCase(char[]).__foreachbody2(ref ulong, ref dchar) ()


std.net.curl.HTTP.Impl.onReceiveHeader(void(const(char[]), 
const(char[])) delegate).__lambda2(const(char[])) ()

std.net.curl.Curl._receiveHeaderCallback(const(char*), ulong, 
ulong, void*) ()

/usr/lib/x86_64-linux-gnu/libcurl.so

/usr/lib/x86_64-linux-gnu/libcurl.so

/usr/lib/x86_64-linux-gnu/libcurl.so

/usr/lib/x86_64-linux-gnu/libcurl.so

/usr/lib/x86_64-linux-gnu/libcurl.so

/usr/lib/x86_64-linux-gnu/libcurl.so

std.net.curl.HTTP.perform(std.typecons.Flag!("throwOnError").Flag) ()

std.net.curl._basicHTTP!(char)._basicHTTP(const(char)[], 
const(void)[], std.net.curl.HTTP) ()

char).get(const(char)[], std.net.curl.HTTP) ()

std.net.curl.get!(std.net.curl.AutoProtocol, 
char).get(const(char)[], std.net.curl.AutoProtocol) ()


Since I'm not super-familiar with libcurl, it's a bit un-obvious 
what's going on here.  If I understand right, libcurl is lazily 
loaded at runtime rather than explicitly linked.  
libcurl4-gnutls-dev is installed on my machine, so it can't just 
be lack of availability.  (I've also tried uninstalling 
libcurl4-gnutls-dev and installing libcurl4-openssl-dev instead, 
with no difference in the result.)

The segfault would suggest to me that either the loading of the 
library fails or that there's some resource phobos expects to 
find which it can't access.  Can anyone advise what could be 
going on here?

In particular, is there anything that happens at compile-time 
which could affect the ability of the resulting executable to 
locate libcurl.so or any other resources it needs?  My suspicion 
is that the snap-package containerization of LDC could mean that 
expectations about system-path locations are interfered with.

Can anyone advise?

Thanks & best wishes,

     -- Joe
Sep 22 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 22 September 2016 at 23:19:27 UTC, Joseph Rushton 
Wakeling wrote:

 The segfault would suggest to me that either the loading of the 
 library fails or that there's some resource phobos expects to 
 find which it can't access.  Can anyone advise what could be 
 going on here?
     -- Joe



 /usr/lib/x86_64-linux-gnu/libcurl.so

 /usr/lib/x86_64-linux-gnu/libcurl.so

 /usr/lib/x86_64-linux-gnu/libcurl.so


This suggests that libcurl is loaded.
could you compile with -g ?
and then post the output ?

Also the call seems to fail during allocation.
Sep 22 2016
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Friday, 23 September 2016 at 00:55:43 UTC, Stefan Koch wrote:

 This suggests that libcurl is loaded.
 could you compile with -g ?
 and then post the output ?


Thanks Stefan!  It was compiled with -g, but I was missing the 
libcurl3-dbg package.  Here's the results:


uint) ()

const(TypeInfo)) ()


const(TypeInfo)) ()

std.array.Appender!(char[]).Appender.ensureAddable(ulong) ()

std.uni.toCase!(std.uni.toLowerIndex(dchar), 1043, 
std.uni.toLowerTab(ulong), 
char[]).toCase(char[]).__foreachbody2(ref ulong, ref dchar) ()


std.net.curl.HTTP.Impl.onReceiveHeader(void(const(char[]), 
const(char[])) delegate).__lambda2(const(char[])) ()

std.net.curl.Curl._receiveHeaderCallback(const(char*), ulong, 
ulong, void*) ()

(conn=conn entry=0x741fd0, type=type entry=2,
     ptr=0x72d3a0 "Server: Apache/2.4.17 (FreeBSD) OpenSSL/1.0.2d 
PHP/5.6.16\r\n", len=59) at sendf.c:454

(conn=conn entry=0x741fd0, type=type entry=2,
     ptr=<optimised out>, len=<optimised out>) at sendf.c:511

(data=data entry=0x72fb60,
     conn=conn entry=0x741fd0, nread=nread entry=0x7fffffffd370,
     stop_reading=stop_reading entry=0x7fffffffd36f) at http.c:3739

didwhat=<synthetic pointer>, k=0x72fbd8,
     conn=0x741fd0, data=0x72fb60) at transfer.c:492

done=done entry=0x7fffffffd40b)
     at transfer.c:1074

(multi=multi entry=0x7388b0, now=...,
     data=data entry=0x72fb60) at multi.c:1544

(multi_handle=multi_handle entry=0x7388b0,
     running_handles=running_handles entry=0x7fffffffd5a4) at 
multi.c:1821

easy.c:724



std.net.curl.HTTP.perform(std.typecons.Flag!("throwOnError").Flag) ()

std.net.curl._basicHTTP!(char)._basicHTTP(const(char)[], 
const(void)[], std.net.curl.HTTP) ()

char).get(const(char)[], std.net.curl.HTTP) ()

std.net.curl.get!(std.net.curl.AutoProtocol, 
char).get(const(char)[], std.net.curl.AutoProtocol) ()




 Also the call seems to fail during allocation.


Yup.  And that allocation seems to occur during use of an 
Appender while converting unicode text to lowercase ... ?
Sep 24 2016
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Saturday, 24 September 2016 at 19:11:52 UTC, Joseph Rushton 
Wakeling wrote:

 On Friday, 23 September 2016 at 00:55:43 UTC, Stefan Koch wrote:

 This suggests that libcurl is loaded.
 could you compile with -g ?
 and then post the output ?




Further to earlier remarks: I now think this may be a general 
problem of LDC 1.0.0 and not a problem of the snap package.

I tried building my simple curl-using program using an LDC 1.0.0 
build and installed from source in the standard cmake && make && 
make install fashion.  The same segfault occurs.
Sep 24 2016
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Saturday, 24 September 2016 at 19:27:31 UTC, Joseph Rushton 
Wakeling wrote:

 Further to earlier remarks: I now think this may be a general 
 problem of LDC 1.0.0 and not a problem of the snap package.

 I tried building my simple curl-using program using an LDC 
 1.0.0 build and installed from source in the standard cmake && 
 make && make install fashion.  The same segfault occurs.


More on this: ldc 0.17.1 (the version packaged with Ubuntu 16.04) 
is based on dmd 2.068.2, which still linked against libcurl.  
It's only from v2.069.0+ that libcurl is loaded dynamically:
https://dlang.org/changelog/2.069.0.html#curl-dynamic-loading

ldc 1.0.0 is based on 2.070.2.
Sep 24 2016
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Saturday, 24 September 2016 at 19:42:11 UTC, Joseph Rushton 
Wakeling wrote:

 On Saturday, 24 September 2016 at 19:27:31 UTC, Joseph Rushton 
 Wakeling wrote:

 Further to earlier remarks: I now think this may be a general 
 problem of LDC 1.0.0 and not a problem of the snap package.

 I tried building my simple curl-using program using an LDC 
 1.0.0 build and installed from source in the standard cmake && 
 make && make install fashion.  The same segfault occurs.


 More on this: ldc 0.17.1 (the version packaged with Ubuntu 
 16.04) is based on dmd 2.068.2, which still linked against 
 libcurl.  It's only from v2.069.0+ that libcurl is loaded 
 dynamically:
 https://dlang.org/changelog/2.069.0.html#curl-dynamic-loading

 ldc 1.0.0 is based on 2.070.2.


Downloaded a pre-built copy of ldc 1.0.0 from here (I used the 
Linux x86_64 tarball):
https://github.com/ldc-developers/ldc/releases/tag/v1.0.0

... and this runs my little curl-based program without any 
trouble.  Maybe something about how the build was carried out?
Sep 24 2016