digitalmars.D - Linking D code into existing C programs
- Walter Bright (38/38) Sep 26 2016 Linking C libraries and object code into D programs has always worked ea...
- Jonathan M Davis via Digitalmars-d (21/59) Sep 26 2016 Well, in this case, you're basically trying to use D without druntime, w...
- Kagamin (10/15) Sep 28 2016 Ugh, no, you can't use C as well as D. Dare I say, C is only for
- Nick Sabalausky (6/7) Sep 26 2016 I don't know anything about the technical side of this issue, but I do
- Basile B. (6/27) Sep 26 2016 This reminds me a huge clusterfuck encountered when trying to
- Jacob Carlborg (9/49) Sep 27 2016 I don't see how that can be expected to work. You're not linking
- Walter Bright (3/4) Sep 27 2016 That's one issue. The other one is druntime needs to be initialized, and...
- Claude (9/13) Sep 27 2016 We've got to consider that when it is statically linked, and also
- Jacob Carlborg (5/12) Sep 27 2016 In will include module and type info. It might include some runtime
- Jacob Carlborg (9/11) Sep 27 2016 Long time ago I suggested doing that [1]. Not sure if that file is
- Jacob Carlborg (4/13) Sep 27 2016 "doing that", I meant to say "automatically initialize druntime".
- Steven Schveighoffer (6/10) Sep 27 2016 Does it? your function seems to not require any druntime features.
- Kagamin (5/11) Sep 28 2016 LDC only emits one unresolved symbol _Dmodule_ref from module
- Ilya Yaroshenko (3/6) Sep 27 2016 Yes, this is what Mir need to replace Eigen and OpenBLAS. Mir do
- Dicebot (4/4) Sep 27 2016 Don't think it can be made much better for object files but it
- Johan Engelen (23/28) Sep 27 2016 We've been toying with this in setting up LDC's build such that
- David Nadlinger (28/41) Sep 27 2016 To clarify: What we are doing in LDC is of course not what was
- Atila Neves (6/9) Sep 27 2016 I think it should work as well as it does in C++, if possible.
- ZombineDev (11/53) Sep 28 2016 Many users that want/need to use D without druntime, which
- David Soria Parra (14/16) Sep 29 2016 I've run into this problem a few times and it took me a while to
- Walter Bright (5/7) Sep 29 2016 I fixed this for Linux and OSX:
Linking C libraries and object code into D programs has always worked easily in
D. The other way around, not so well.
Consider the following C program:
---- main.c ----
extern int foo();
void main(int argc, char* argv[]) {
foo();
}
---- foo.c ----
int foo() { return 7; }
---------------
Compile/link in the usual way:
gcc main.c -c
gcc foo.c -c
gcc main.o foo.o
No problem.
Now replace foo.c with foo.d:
---- foo.d ----
extern (C) int foo() { return 7; }
---------------
Compile/link with:
gcc main.c -c
dmd foo.d -c
gcc main.o foo.o
Produces:
bar.o:(.eh_frame+0x13): undefined reference to `__dmd_personality_v0'
bar.o: In function `_D3bar7__arrayZ':
bar.d:(.text._D3bar7__arrayZ+0x21): undefined reference to `_d_arraybounds'
bar.o: In function `_D3bar8__assertFiZv':
bar.d:(.text._D3bar8__assertFiZv+0x21): undefined reference to `_d_assert'
bar.o: In function `_D3bar15__unittest_failFiZv':
bar.d:(.text._D3bar15__unittest_failFiZv+0x21): undefined reference to
`_d_unittest'
bar.o: In function `__d_dso_init':
bar.d:(.text.d_dso_init[.data.d_dso_rec]+0x28): undefined reference to
`_d_dso_registry'
collect2: error: ld returned 1 exit status
How much of an issue is this with D? Is it something we need to address?
Sep 26 2016
On Monday, September 26, 2016 16:32:05 Walter Bright via Digitalmars-d wrote:
Linking C libraries and object code into D programs has always worked easily
in D. The other way around, not so well.
Consider the following C program:
---- main.c ----
extern int foo();
void main(int argc, char* argv[]) {
foo();
}
---- foo.c ----
int foo() { return 7; }
---------------
Compile/link in the usual way:
gcc main.c -c
gcc foo.c -c
gcc main.o foo.o
No problem.
Now replace foo.c with foo.d:
---- foo.d ----
extern (C) int foo() { return 7; }
---------------
Compile/link with:
gcc main.c -c
dmd foo.d -c
gcc main.o foo.o
Produces:
bar.o:(.eh_frame+0x13): undefined reference to `__dmd_personality_v0'
bar.o: In function `_D3bar7__arrayZ':
bar.d:(.text._D3bar7__arrayZ+0x21): undefined reference to
`_d_arraybounds' bar.o: In function `_D3bar8__assertFiZv':
bar.d:(.text._D3bar8__assertFiZv+0x21): undefined reference to
`_d_assert' bar.o: In function `_D3bar15__unittest_failFiZv':
bar.d:(.text._D3bar15__unittest_failFiZv+0x21): undefined reference to
`_d_unittest'
bar.o: In function `__d_dso_init':
bar.d:(.text.d_dso_init[.data.d_dso_rec]+0x28): undefined reference to
`_d_dso_registry'
collect2: error: ld returned 1 exit status
How much of an issue is this with D? Is it something we need to address?
Well, in this case, you're basically trying to use D without druntime, which
generally is a total non-starter anyway. If you're not using druntime, then
you might as well be using C, since you're not even going to get basic stuff
like array bounds checking. So, I don't think that that really matters.
That being said, I would hope that it wouldn't be all that hard to properly
link D code into a C/C++ program and use it (complete with druntime and
Phobos). There's no guarantee that D is going to get to be what's driving
the program even if you can use D for a lot of it. But isn't the fix for
that to link in libphobos.a? I'm pretty sure that I've linked D into a C
program before, but it's been a while, so I don't recall exactly what you
have to do. But it doesn't seem all that unreasonable to me that you'd have
to explicitly link in D's runtime library. It's not like the C linker is
going to know about it auto-magically.
As long as there's a reasonably easy and properly documented way to link D
code into a C/C++ program, I think that we're fine. But I don't think that
your exact example needs to work exactly as-is. It just shouldn't be much
harder than that. e.g. something like
gcc main.o foo.o -lphobos.a
should probably work.
- Jonathan M Davis
Sep 26 2016
Kagamin <spam here.lot> On Monday, 26 September 2016 at 23:47:45 UTC, Jonathan M Davis wrote:Well, in this case, you're basically trying to use D without druntime, which generally is a total non-starter anyway. If you're not using druntime, then you might as well be using C, since you're not even going to get basic stuff like array bounds checking.Ugh, no, you can't use C as well as D. Dare I say, C is only for maintenance of legacy code; if you have a choice, use D, it's just better. Non-starters are GC and exceptions. Asserts and bounds checking are one-liners (and for really minimal code even they are not needed), string switch is another couple of lines and you get the type system and all the compile time capabilities of D that you're never going to have in C (that modern convenience and modelling power).
Sep 28 2016
On 09/26/2016 07:32 PM, Walter Bright wrote:How much of an issue is this with D? Is it something we need to address?I don't know anything about the technical side of this issue, but I do think it's an important thing to have working well, since I believe it's important for us to start marketing our D-based libraries as easily usable from C: https://semitwist.com/articles/article/view/we-re-overlooking-a-key-part-of-c-c-d-user-migration
Sep 26 2016
On Monday, 26 September 2016 at 23:32:05 UTC, Walter Bright wrote:Linking C libraries and object code into D programs has always worked easily in D. The other way around, not so well. [...] Produces: bar.o:(.eh_frame+0x13): undefined reference to `__dmd_personality_v0' bar.o: In function `_D3bar7__arrayZ': bar.d:(.text._D3bar7__arrayZ+0x21): undefined reference to `_d_arraybounds' bar.o: In function `_D3bar8__assertFiZv': bar.d:(.text._D3bar8__assertFiZv+0x21): undefined reference to `_d_assert' bar.o: In function `_D3bar15__unittest_failFiZv': bar.d:(.text._D3bar15__unittest_failFiZv+0x21): undefined reference to `_d_unittest' bar.o: In function `__d_dso_init': bar.d:(.text.d_dso_init[.data.d_dso_rec]+0x28): undefined reference to `_d_dso_registry' collect2: error: ld returned 1 exit status How much of an issue is this with D? Is it something we need to address?This reminds me a huge clusterfuck encountered when trying to link an interfaced version of libdparse in Object Pascal: http://forum.lazarus.freepascal.org/index.php/topic,32340.msg208499.html#msg208499 Soon or later this will happen again. But the next time the guy could completely give up and say "D is shitty".
Sep 26 2016
On 2016-09-27 01:32, Walter Bright wrote:
Linking C libraries and object code into D programs has always worked
easily in D. The other way around, not so well.
Consider the following C program:
---- main.c ----
extern int foo();
void main(int argc, char* argv[]) {
foo();
}
---- foo.c ----
int foo() { return 7; }
---------------
Compile/link in the usual way:
gcc main.c -c
gcc foo.c -c
gcc main.o foo.o
No problem.
Now replace foo.c with foo.d:
---- foo.d ----
extern (C) int foo() { return 7; }
---------------
Compile/link with:
gcc main.c -c
dmd foo.d -c
gcc main.o foo.o
Produces:
bar.o:(.eh_frame+0x13): undefined reference to `__dmd_personality_v0'
bar.o: In function `_D3bar7__arrayZ':
bar.d:(.text._D3bar7__arrayZ+0x21): undefined reference to
`_d_arraybounds'
bar.o: In function `_D3bar8__assertFiZv':
bar.d:(.text._D3bar8__assertFiZv+0x21): undefined reference to
`_d_assert'
bar.o: In function `_D3bar15__unittest_failFiZv':
bar.d:(.text._D3bar15__unittest_failFiZv+0x21): undefined reference to
`_d_unittest'
bar.o: In function `__d_dso_init':
bar.d:(.text.d_dso_init[.data.d_dso_rec]+0x28): undefined reference to
`_d_dso_registry'
collect2: error: ld returned 1 exit status
How much of an issue is this with D? Is it something we need to address?
I don't see how that can be expected to work. You're not linking
druntime. The easiest is to link with DMD instead of GCC. An alternative
is to compile the D code with -betterC to avoid the runtime (not sure
what happens with the personality function), although that's currently
broken [1].
[1] https://issues.dlang.org/show_bug.cgi?id=16547
--
/Jacob Carlborg
Sep 27 2016
On 9/27/2016 12:03 AM, Jacob Carlborg wrote:You're not linking druntime.That's one issue. The other one is druntime needs to be initialized, and calling a random D function won't do that.
Sep 27 2016
On Tuesday, 27 September 2016 at 08:12:58 UTC, Walter Bright wrote:On 9/27/2016 12:03 AM, Jacob Carlborg wrote:We've got to consider that when it is statically linked, and also dynamically as well. Also, if it's statically linked, will the linker just use parts of druntime that are actually used by the D libtrary? For instance, if the D library does not use exceptions, ideally it should not link any related code in druntime (it's important, especially in embedded software).You're not linking druntime.That's one issue. The other one is druntime needs to be initialized, and calling a random D function won't do that.
Sep 27 2016
On 2016-09-27 10:50, Claude wrote:We've got to consider that when it is statically linked, and also dynamically as well. Also, if it's statically linked, will the linker just use parts of druntime that are actually used by the D libtrary? For instance, if the D library does not use exceptions, ideally it should not link any related code in druntime (it's important, especially in embedded software).In will include module and type info. It might include some runtime functions for exceptions unless all functions are marked with "nothrow". -- /Jacob Carlborg
Sep 27 2016
On 2016-09-27 10:12, Walter Bright wrote:That's one issue. The other one is druntime needs to be initialized, and calling a random D function won't do that.Long time ago I suggested doing that [1]. Not sure if that file is currently compiled and linked into the runtime. But, IIRC, there was complains when I suggested doing this because, as I was told, if you're using a D library in a C application you most likely want to have more control of the initialization of druntime. [1] https://github.com/dlang/druntime/blob/master/src/rt/dylib_fixes.c -- /Jacob Carlborg
Sep 27 2016
On 2016-09-27 11:18, Jacob Carlborg wrote:On 2016-09-27 10:12, Walter Bright wrote:"doing that", I meant to say "automatically initialize druntime". -- /Jacob CarlborgThat's one issue. The other one is druntime needs to be initialized, and calling a random D function won't do that.Long time ago I suggested doing that [1]. Not sure if that file is currently compiled and linked into the runtime. But, IIRC, there was complains when I suggested doing this because, as I was told, if you're using a D library in a C application you most likely want to have more control of the initialization of druntime. [1] https://github.com/dlang/druntime/blob/master/src/rt/dylib_fixes.c
Sep 27 2016
On 9/27/16 4:12 AM, Walter Bright wrote:On 9/27/2016 12:03 AM, Jacob Carlborg wrote:Does it? your function seems to not require any druntime features. I would say -betterC is the way to fix this, as conservatively not emitting druntime requirements is probably a lot more involved than a directive from the build that no druntime is required. -SteveYou're not linking druntime.That's one issue. The other one is druntime needs to be initialized, and calling a random D function won't do that.
Sep 27 2016
On Tuesday, 27 September 2016 at 12:58:36 UTC, Steven Schveighoffer wrote:Does it? your function seems to not require any druntime features. I would say -betterC is the way to fix this, as conservatively not emitting druntime requirements is probably a lot more involved than a directive from the build that no druntime is required.LDC only emits one unresolved symbol _Dmodule_ref from module initializer and that's all, so it can be linked with, say, ld --defsym _Dmodule_ref=16
Sep 28 2016
On Monday, 26 September 2016 at 23:32:05 UTC, Walter Bright wrote:Linking C libraries and object code into D programs has always worked easily in D. The other way around, not so well. [...]Yes, this is what Mir need to replace Eigen and OpenBLAS. Mir do not need DRuntime.
Sep 27 2016
Don't think it can be made much better for object files but it should be relatively simple to enable creating "C-friendly" static D libraries that include all required runtime bits as part of the library.
Sep 27 2016
On Monday, 26 September 2016 at 23:32:05 UTC, Walter Bright wrote:Linking C libraries and object code into D programs has always worked easily in D. The other way around, not so well. [snip] How much of an issue is this with D? Is it something we need to address?We've been toying with this in setting up LDC's build such that it works on different platforms. Wait, "toying" implies "fun". It was anything but. At first, we used the D compiler to do the final linking, but it resulted in troubles when special linker flags are needed. We've now moved to using the system linker separately to do the linking. https://github.com/ldc-developers/ldc/pull/1594 For this, we use `-v` to figure out what the system linker is and what linker flags are passed by the D compiler (link with phobos, druntime, etc.). But it needs "parsing" of `-v` output, quite annoying. See https://github.com/ldc-developers/ldc/blob/master/cmake/Modules/ExtractDMDSystemLinker.cmake The idea to add a cmdline flag `-ldflags` that would just output the linking flags, did not get a response. https://forum.dlang.org/post/gqaujnbgbpauirbezjki forum.dlang.org I think it will save people a lot of time and frustration. (possible improvements can be made such that the output of `-ldflags` also includes extra link flags when special compiler flags are passed, such as `-fprofile-instr-generate`.) -Johan
Sep 27 2016
On Tuesday, 27 September 2016 at 10:46:23 UTC, Johan Engelen wrote:On Monday, 26 September 2016 at 23:32:05 UTC, Walter Bright wrote:To clarify: What we are doing in LDC is of course not what was suggested in the initial post, i.e. linking D object files into C executables without any (explicit) druntime dependencies and expecting this to work. The problem we were facing in particular is that many C/C++ libraries (such as LLVM) come with a list of linker flags to use in the form of pkg-config or a similar tool. These will usually be formatted for the default way of linking C on the system (i.e. through a gcc-compatible interface). Thus, you can't just prefix them with -L and forward them to the D compiler since that forwards them directly to the low-level linker (-Xlinker, …). This is a problem which *every* D project that wants to seamlessly link against system C libraries on Linux needs to solve. There are two obvious ways to improve the situation: - Like Johan suggested, add a D compiler flag that prints the default linker flags used to create D executables. - Add a D compiler flag to forward flags directly to the linker driver used, without using -Xlinker. Unfortunately, the LDC build system of course needs to work with existing host compilers anyway, so these wouldn't have helped us with shipping the first DDMD-based version. The most robust solution I could come up with was to extract the linker command line used from `$DMD -v some_test_executable.d` to then pass the same flags directly to the system gcc when building the D program. — DavidLinking C libraries and object code into D programs has always worked easily in D. The other way around, not so well. [snip] How much of an issue is this with D? Is it something we need to address?We've been toying with this in setting up LDC's build such that it works on different platforms. […] We've now moved to using the system linker separately to do the linking.
Sep 27 2016
On Monday, 26 September 2016 at 23:32:05 UTC, Walter Bright wrote:Linking C libraries and object code into D programs has always worked easily in D. The other way around, not so well. [...]I think it should work as well as it does in C++, if possible. The equivalent to the example above links fine for foo.cpp. It's only if you use the standard library or throw exceptions that you need to link with g++ instead. Atila
Sep 27 2016
On Monday, 26 September 2016 at 23:32:05 UTC, Walter Bright wrote:
Linking C libraries and object code into D programs has always
worked easily in D. The other way around, not so well.
Consider the following C program:
---- main.c ----
extern int foo();
void main(int argc, char* argv[]) {
foo();
}
---- foo.c ----
int foo() { return 7; }
---------------
Compile/link in the usual way:
gcc main.c -c
gcc foo.c -c
gcc main.o foo.o
No problem.
Now replace foo.c with foo.d:
---- foo.d ----
extern (C) int foo() { return 7; }
---------------
Compile/link with:
gcc main.c -c
dmd foo.d -c
gcc main.o foo.o
Produces:
bar.o:(.eh_frame+0x13): undefined reference to
`__dmd_personality_v0'
bar.o: In function `_D3bar7__arrayZ':
bar.d:(.text._D3bar7__arrayZ+0x21): undefined reference to
`_d_arraybounds'
bar.o: In function `_D3bar8__assertFiZv':
bar.d:(.text._D3bar8__assertFiZv+0x21): undefined reference
to `_d_assert'
bar.o: In function `_D3bar15__unittest_failFiZv':
bar.d:(.text._D3bar15__unittest_failFiZv+0x21): undefined
reference to `_d_unittest'
bar.o: In function `__d_dso_init':
bar.d:(.text.d_dso_init[.data.d_dso_rec]+0x28): undefined
reference to `_d_dso_registry'
collect2: error: ld returned 1 exit status
How much of an issue is this with D? Is it something we need to
address?
Many users that want/need to use D without druntime, which
implies nothrow nogc, --boundscheck=off, no unittests (and
probably a couple of other things) and we need to support these
use cases. In other words,
the compiler should detect if a given module does not use any
druntime features and in such cases it should not emit calls/
linktime references to druntime. I don't know if in general the
compiler can tell if certain runtime features (e.g. exception
handling support) are not used needed, so at first we may require
the -betterC switch to supplied.
Sep 28 2016
On Monday, 26 September 2016 at 23:32:05 UTC, Walter Bright wrote:How much of an issue is this with D? Is it something we need to address?I've run into this problem a few times and it took me a while to understand how to correctly initialize the druntime (including attaching pthreads), when I was implementing dfuse. This is not necessarily only limited to using D from C but also from using C libraries that create threads or do other funky things that the druntime needs to be aware of. It would be great to have a section in the documentation on how to correctly initialize the druntime, which are typical pitfalls (threads created in C land, callbacks into C, etc) and how to correctly link druntime. If we don't do this already we should offer a phobos.lib on windows, and phobos.a on MacOS/Linux (the latter one i think we do).
Sep 29 2016
On 9/26/2016 4:32 PM, Walter Bright wrote:Produces: bar.o:(.eh_frame+0x13): undefined reference to `__dmd_personality_v0'I fixed this for Linux and OSX: https://github.com/dlang/dmd/pull/6159 There is apparently some issue with the gnu toolchain on FreeBSD which prevents this improvement from working there (had to disable it for FreeBSD).
Sep 29 2016