www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.ldc - std.net.curl and ldc v1.0.0+

reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
Hello all,

I've run into a nasty little issue with `std.net.curl` and ldc 
v1.0.0.  If I build v1.0.0 freshly from source, and then use it 
to compile the following tiny program:

////////////////////////////////////////////////

void main ()
{
     import std.net.curl : get;
     auto website = "http://dlang.org/".get;
}

////////////////////////////////////////////////

... the program compiles correctly, but when I run it, I get a 
segfault.  A stacktrace is given here:
https://forum.dlang.org/post/cxfrbiwccxdnmrkuufwe forum.dlang.org

This doesn't however occur if I build my program with the 
pre-built ldc v1.0.0 downloadable from GitHub.

I'm guessing this is in some way related to the fact that as of 
ldc v1.0.0, libcurl.so is dynamically loaded at runtime rather 
than being linked to phobos at build time.  So this makes me 
wonder: is there something extra required when building LDC that 
I might be missing?  I note that dmd now automatically links 
against libdl in order to support dynamic loading of libraries:
https://github.com/dlang/dmd/commit/541d9b61cd1792535cd4c99c418b42273bb9baa5

... so could I be missing something similar when building LDC?

I note that if I manually link in -ldl when compiling my program 
above, it makes no difference to the segfault.

Can anyone advise what could be wrong?

Thanks & best wishes,

     -- Joe
Sep 26 2016
next sibling parent reply Johan Engelen <j j.nl> writes:
On Monday, 26 September 2016 at 18:52:04 UTC, Joseph Rushton 
Wakeling wrote:

 So this makes me wonder: is there something extra required when 
 building LDC that I might be missing?


Just a very quick reply, perhaps it can help you:
https://github.com/ldc-developers/ldc-scripts/tree/master/ldc2-packaging
we use this git repo to store the scripts that are used to build 
the binary packages.

(which platform are you on?)
Sep 26 2016
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Monday, 26 September 2016 at 19:03:37 UTC, Johan Engelen wrote:

 Just a very quick reply, perhaps it can help you:
 https://github.com/ldc-developers/ldc-scripts/tree/master/ldc2-packaging
 we use this git repo to store the scripts that are used to 
 build the binary packages.


Thanks!  I notice that on Linux you use a 
-DCMAKE_EXE_LINKER_FLAGS='-static-libstdc++' that I'm not using.  
I'll give that a try.


 (which platform are you on?)


Ubuntu 16.04.  I'm using the distro-packaged dependencies, with 
llvm 3.8.
Sep 26 2016
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Monday, 26 September 2016 at 19:20:04 UTC, Joseph Rushton 
Wakeling wrote:

 On Monday, 26 September 2016 at 19:03:37 UTC, Johan Engelen 
 wrote:

 Just a very quick reply, perhaps it can help you:
 https://github.com/ldc-developers/ldc-scripts/tree/master/ldc2-packaging
 we use this git repo to store the scripts that are used to 
 build the binary packages.


 Thanks!  I notice that on Linux you use a 
 -DCMAKE_EXE_LINKER_FLAGS='-static-libstdc++' that I'm not 
 using.  I'll give that a try.


Hmm, makes no apparent difference :-(
Sep 26 2016
next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Monday, 26 September 2016 at 19:49:46 UTC, Joseph Rushton 
Wakeling wrote:

 On Monday, 26 September 2016 at 19:20:04 UTC, Joseph Rushton 
 Wakeling wrote:

 On Monday, 26 September 2016 at 19:03:37 UTC, Johan Engelen 
 wrote:

 Just a very quick reply, perhaps it can help you:
 https://github.com/ldc-developers/ldc-scripts/tree/master/ldc2-packaging
 we use this git repo to store the scripts that are used to 
 build the binary packages.


 Thanks!  I notice that on Linux you use a 
 -DCMAKE_EXE_LINKER_FLAGS='-static-libstdc++' that I'm not 
 using.  I'll give that a try.


 Hmm, makes no apparent difference :-(


Just to share more details on my system:

   * Ubuntu 16.04
   * llvm-3.8-dev
   * libconfig++9v5 1.5-02
   * gcc 5.4.0
   * libedit 3.1
   * zlib 1.2.8
Sep 26 2016
parent reply kinke <noone nowhere.com> writes:
I don't really have an idea here, but I'd try a few things to 
narrow it down - e.g., replace your druntime/Phobos libs by the 
pre-built ones and see if that makes it work, then try using the 
ldc2.conf from the prebuilt one, then the ldc2 executable itself 
etc.
Sep 27 2016
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Tuesday, 27 September 2016 at 17:49:58 UTC, kinke wrote:

 I don't really have an idea here, but I'd try a few things to 
 narrow it down - e.g., replace your druntime/Phobos libs by the 
 pre-built ones and see if that makes it work, then try using 
 the ldc2.conf from the prebuilt one, then the ldc2 executable 
 itself etc.


Good call.  If I replace druntime with the pre-built one, then 
things work OK with the built curl-using program.  Switching out 
phobos makes no difference.
Sep 27 2016
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Tuesday, 27 September 2016 at 18:34:41 UTC, Joseph Rushton 
Wakeling wrote:

 Good call.  If I replace druntime with the pre-built one, then 
 things work OK with the built curl-using program.  Switching 
 out phobos makes no difference.


This is quite interesting, because the backtrace suggests the 
segfault is happening in gc.gc.Gcx.smallAlloc:
https://forum.dlang.org/post/cxfrbiwccxdnmrkuufwe forum.dlang.org
Sep 27 2016
prev sibling parent reply kinke <noone nowhere.com> writes:
On Tuesday, 27 September 2016 at 18:34:41 UTC, Joseph Rushton 
Wakeling wrote:

 If I replace druntime with the pre-built one, then things work 
 OK with the built curl-using program.  Switching out phobos 
 makes no difference.


Oh this may make things clearer. druntime features C parts, so 
the pre-built one and yours probably differ in the used C(++) 
runtime; the malloc issue also points in this direction. As we 
seem to be using libstdc++, are you sure you're using that one 
too?
Sep 27 2016
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Tuesday, 27 September 2016 at 21:00:03 UTC, kinke wrote:

 Oh this may make things clearer. druntime features C parts, so 
 the pre-built one and yours probably differ in the used C(++) 
 runtime; the malloc issue also points in this direction. As we 
 seem to be using libstdc++, are you sure you're using that one 
 too?


That would make sense.  How would I determine whether or not I'm 
using libstdc++?

I note that the build scripts for Linux pass cmake a flag:

     -DCMAKE_EXE_LINKER_FLAGS='-static-libstdc++'

when I tried this, I had to pass instead:

     -DCMAKE_EXE_LINKER_FLAGS='-L-static-libstdc++'

... otherwise I got a linker failure for the ldc2 build with the 
message that there was no such option as 'tic-libstdc++' (!).

Since in any case I'm building on Ubuntu 16.04 rather than the 
12.04 used to create the prebuilt package, I presume libstdc++ 
would be a different version (I can't verify exactly which right 
now, I'm afraid, as I'm currently away from my computer), which I 
suppose could reasonably be the reason for the problem.
Sep 27 2016
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Tuesday, 27 September 2016 at 21:00:03 UTC, kinke wrote:

 On Tuesday, 27 September 2016 at 18:34:41 UTC, Joseph Rushton 
 Wakeling wrote:

 If I replace druntime with the pre-built one, then things work 
 OK with the built curl-using program.  Switching out phobos 
 makes no difference.


 Oh this may make things clearer. druntime features C parts, so 
 the pre-built one and yours probably differ in the used C(++) 
 runtime; the malloc issue also points in this direction. As we 
 seem to be using libstdc++, are you sure you're using that one 
 too?


I've come back to this after a little while of not being able to 
work on it.  Anyway, a few questions.

First, the build scripts for LDC:
https://github.com/ldc-developers/ldc-scripts/tree/master/ldc2-packaging

There are 3 noticeable things about them:

(i) a backported g++-4.9 is used (the default gcc/g++ version on 
Ubuntu 16.04 is 5.4)

(ii) LLVM is built from source

(iii) the build script uses two cmake config flags related to the 
choice of standard library:

         -DCMAKE_EXE_LINKER_FLAGS='-stdlib=libc++'
         -DCMAKE_EXE_LINKER_FLAGS='-static-libstdc++

My own builds fail for LDC 1.0.0 if I use these exact cmake 
flags, because they are not recognized as valid flags by LDC 
itself.  Using instead:

         -DCMAKE_EXE_LINKER_FLAGS='-L-stdlib=libc++'
         -DCMAKE_EXE_LINKER_FLAGS='-L-static-libstdc++

... builds, but makes no difference.

So, first, can I confirm that these scripts are really used as-is 
to build LDC 1.0.0 and later LDC versions?

Second, since I use Ubuntu 16.04's native ldc 0.17.1 package as 
the D compiler to build ldc 1.0.0, how is the need for an 
existing D compiler dealt with via LDC's own build scripts?  Is 
there an automated bootstrapping process?

Third, is it possible that the use of a pre-built LLVM could be 
affecting things?  Are there parts of LLVM used in LDC's 
druntime, particularly related to the GC?

Fourth, are there any known issues related to the libc/libstdc++ 
version and druntime?  Is there a reason why you didn't backport 
gcc 5 for building LDC?

Thanks & best wishes,

     -- Joe
Oct 15 2016
parent reply David Nadlinger <code klickverbot.at> writes:
On Saturday, 15 October 2016 at 19:22:42 UTC, Joseph Rushton 
Wakeling wrote:

 Third, is it possible that the use of a pre-built LLVM could be 
 affecting things?  Are there parts of LLVM used in LDC's 
 druntime, particularly related to the GC?


It shouldn't affect druntime directly any more than any other 
code (i.e. only through any potential codegen issues). The reason 
why we build LLVM from source is just one of reproducibility – 
several years ago now, we had seen a number of issues with broken 
distro packages, and it seemed prudent to build the binary 
packages with something entirely under our control.


 Fourth, are there any known issues related to the 
 libc/libstdc++ version and druntime?  Is there a reason why you 
 didn't backport gcc 5 for building LDC?


I believe it was simply the latest available version at the time, 
or maybe the oldest one required by LLVM (after switching to 
C++11, etc.). For the binary packages, the main goal is maximum 
compatibility, so we build on an old Ubuntu version – although, 
of course, these issues have been somewhat alleviated by linking 
glibc statically now.

There are no known issues with particular runtime versions, and 
if there were, we would try to work around them as well as 
possible.

  — David
Oct 15 2016
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Saturday, 15 October 2016 at 20:48:09 UTC, David Nadlinger 
wrote:

 There are no known issues with particular runtime versions, and 
 if there were, we would try to work around them as well as 
 possible.


Just for the record, I tried building DMD 2.070.2 (the 
frontend/druntime/phobos version used with LDC 1.0.0) with my 
usual GCC/libc/libstdc++ setup, and my little curl-based program 
builds and runs just fine with it.  Which suggests it's something 
to do with LDC's particular setup.  Are there any custom patches 
in druntime which might affect memory allocation, for example?

BTW when you say "known issues with particular runtime versions", 
I was thinking of issues with particular versions of GCC, but I 
take it there are none here either.
Oct 16 2016
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Sunday, 16 October 2016 at 15:32:15 UTC, Joseph Rushton 
Wakeling wrote:

 On Saturday, 15 October 2016 at 20:48:09 UTC, David Nadlinger 
 wrote:

 There are no known issues with particular runtime versions, 
 and if there were, we would try to work around them as well as 
 possible.




Further: I tried building ldc v1.1.0-beta3, and the issue doesn't 
show up.  So it looks ldc v1.0.0-specific.
Oct 16 2016
prev sibling parent reply David Nadlinger <code klickverbot.at> writes:
On Monday, 26 September 2016 at 19:49:46 UTC, Joseph Rushton 
Wakeling wrote:

 On Monday, 26 September 2016 at 19:20:04 UTC, Joseph Rushton 
 Wakeling wrote:

 Thanks!  I notice that on Linux you use a 
 -DCMAKE_EXE_LINKER_FLAGS='-static-libstdc++' that I'm not 
 using.  I'll give that a try.


 Hmm, makes no apparent difference :-(


That's not surprising – it only causes the LDC executable itself 
to be linked statically, which shouldn't affect code generation. 
(Then again, the whole bug is rather strange to begin with, so 
it's good to have that ruled out.)

  — David
Oct 15 2016
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Saturday, 15 October 2016 at 20:52:05 UTC, David Nadlinger 
wrote:

 That's not surprising – it only causes the LDC executable 
 itself to be linked statically, which shouldn't affect code 
 generation. (Then again, the whole bug is rather strange to 
 begin with, so it's good to have that ruled out.)


Yes, I wasn't expecting it to make a difference either, but I 
thought I'd try.  I wondered if the `-stdlib=libstdc++` option 
might also make a difference but again, doesn't seem to (which 
again, I'm ultimately not too surprised by, since I would imagine 
that on any Debian-based system, libstdc++6 just dynamically 
links against libc6 for the C standard library parts.

I guess I could have a go at building using g++-4.9, which 
requires a few extra cmake flags to account for the naming of the 
GCC executable:

     -DCMAKE_C_COMPILER=gcc-4.9
     -DCMAKE_CXX_COMPILER=g++-4.9

but that runs into a problem related to the build scripts that 
looks like it assumes that the GCC executable will always be 
called `gcc`:

make -j4
Scanning dependencies of target gen_gccbuiltins
Scanning dependencies of target FileCheck
Scanning dependencies of target LDMD_CXX_LIB
[  0%] Generating ddmd/idgen
[  0%] Building CXX object 
CMakeFiles/gen_gccbuiltins.dir/utils/gen_gccbuiltins.cpp.o
[  0%] Building CXX object 
CMakeFiles/LDMD_CXX_LIB.dir/driver/exe_path.cpp.o
[  0%] Building CXX object 
CMakeFiles/FileCheck.dir/utils/FileCheck-3.8.cpp.o
Error: failed to locate gcc
CMakeFiles/LDCShared.dir/build.make:68: recipe for target 
'ddmd/idgen' failed
make[2]: *** [ddmd/idgen] Error 1
CMakeFiles/Makefile2:141: recipe for target 
'CMakeFiles/LDCShared.dir/all' failed
make[1]: *** [CMakeFiles/LDCShared.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Oct 15 2016
parent reply David Nadlinger <code klickverbot.at> writes:
On Saturday, 15 October 2016 at 22:09:35 UTC, Joseph Rushton 
Wakeling wrote:

 but that runs into a problem related to the build scripts that 
 looks like it assumes that the GCC executable will always be 
 called `gcc`:


I don't think that would be the build scripts, but the host D 
compiler trying to link the executable.

  — David
Oct 15 2016
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Saturday, 15 October 2016 at 22:20:12 UTC, David Nadlinger 
wrote:

 I don't think that would be the build scripts, but the host D 
 compiler trying to link the executable.


Oh, damn, of course.  I got fooled by the fact that it showed up 
right after a bunch of stuff related to CMakeFiles.

Is there a convenient cmake flag to pass some flags to the D 
compiler so that I can specify the GCC executable?
Oct 15 2016
parent reply David Nadlinger <code klickverbot.at> writes:
On Saturday, 15 October 2016 at 23:31:33 UTC, Joseph Rushton 
Wakeling wrote:

 Is there a convenient cmake flag to pass some flags to the D 
 compiler so that I can specify the GCC executable?


IIRC, you can set either D_COMPILER or the DMD environment 
variable.

  — David
Oct 15 2016
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Saturday, 15 October 2016 at 23:34:41 UTC, David Nadlinger 
wrote:

 IIRC, you can set either D_COMPILER or the DMD environment 
 variable.


That unfortunately runs straight into a problem that I think we 
already encountered earlier: because the build scripts check for 
the existence of the D compiler, it finds that "ldc -gcc=gcc-4.9" 
does not exist :-\

There's DDMD_DFLAGS, which I tried setting with 
-DDDMD_DFLAGS=-gcc=gcc-4.9 and -DDDMD_DFLAGS="-gcc=gcc-4.9", and 
neither works; the cmake call generates output:

     -- Found D compiler /usr/bin/ldmd2, with default flags ''
     -- D compiler version: LDC - the LLVM D compiler (0.17.1)

... followed by the same old 'gcc not found' error, so it's 
somehow not being picked up.  I'd suspect that is down to these 
lines in CMakeLists.txt:

     set(DDMD_DFLAGS "-wi")
     set(DDMD_LFLAGS "")

which presumably override anything set by the command-line flags.

I tried tweaking the CMakeLists.txt file directly to set the 
-gcc=gcc-4.9 flag, and instead I get a linker error:

[  0%] Linking CXX executable bin/gen_gccbuiltins
CMakeFiles/gen_gccbuiltins.dir/utils/gen_gccbuiltins.cpp.o: In 
function `attributes(llvm::ListInit*)':
/home/joseph/code/D/ldc/utils/gen_gccbuiltins.cpp:91: undefined 
reference to `llvm::Record::getName() const'
CMakeFiles/gen_gccbuiltins.dir/utils/gen_gccbuiltins.cpp.o: In 
function `processRecord(llvm::raw_ostream&, llvm::Record&, 
std::string)':
/home/joseph/code/D/ldc/utils/gen_gccbuiltins.cpp:104: undefined 
reference to `llvm::Record::getValueAsString(llvm::StringRef) 
const'
/home/joseph/code/D/ldc/utils/gen_gccbuiltins.cpp:105: undefined 
reference to `llvm::Record::getName() const'
/home/joseph/code/D/ldc/utils/gen_gccbuiltins.cpp:125: undefined 
reference to `llvm::Record::getName() const'
collect2: error: ld returned 1 exit status

... which I would suspect is down to the prebuilt llvm-3.8 being 
built with g++ 5.4 and its corresponding libstdc++6 ... ?
Oct 16 2016
prev sibling parent reply Suliman <evermind live.ru> writes:
On Monday, 26 September 2016 at 18:52:04 UTC, Joseph Rushton 
Wakeling wrote:

 Hello all,

 I've run into a nasty little issue with `std.net.curl` and ldc 
 v1.0.0.  If I build v1.0.0 freshly from source, and then use it 
 to compile the following tiny program:

 ////////////////////////////////////////////////

 void main ()
 {
     import std.net.curl : get;
     auto website = "http://dlang.org/".get;
 }

 ////////////////////////////////////////////////

 ... the program compiles correctly, but when I run it, I get a 
 segfault.  A stacktrace is given here:
 https://forum.dlang.org/post/cxfrbiwccxdnmrkuufwe forum.dlang.org

 This doesn't however occur if I build my program with the 
 pre-built ldc v1.0.0 downloadable from GitHub.

 I'm guessing this is in some way related to the fact that as of 
 ldc v1.0.0, libcurl.so is dynamically loaded at runtime rather 
 than being linked to phobos at build time.  So this makes me 
 wonder: is there something extra required when building LDC 
 that I might be missing?  I note that dmd now automatically 
 links against libdl in order to support dynamic loading of 
 libraries:
 https://github.com/dlang/dmd/commit/541d9b61cd1792535cd4c99c418b42273bb9baa5

 ... so could I be missing something similar when building LDC?

 I note that if I manually link in -ldl when compiling my 
 program above, it makes no difference to the segfault.

 Can anyone advise what could be wrong?

 Thanks & best wishes,

     -- Joe


I suggest you to try https://github.com/ikod/dlang-requests it's 
much better than curl
Sep 27 2016
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Wednesday, 28 September 2016 at 06:23:48 UTC, Suliman wrote:

 I suggest you to try https://github.com/ikod/dlang-requests 
 it's much better than curl


Thanks for the suggestion, but it doesn't address my problem in 
this case.  I'm not concerned about being able to perform an HTTP 
GET request per se, but about being able to guarantee that the 
LDC + libs that I build will appropriately build programs based 
on any part of the standard library.
Sep 28 2016