www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - DPP: Linker issue with functions implemented in C header files

reply Andre Pany <andre s-e-a-p.de> writes:
Hi,

I try to get wrap the "Azure SDK for C" using DPP and have 
following issue.
Functions, which are actually implemented in C header files will 
cause
linker errors:

https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/inc/az_span.h#L91

Example:
AZ_NODISCARD AZ_INLINE az_span az_span_init(uint8_t * ptr, 
int32_t length, int32_t capacity) {
   return (az_span){ ._internal = { .ptr = ptr, .length = length, 
.capacity = capacity, }, };
}

Error message:
/tmp/app.o:az_storage_blobs.d:function 
_D20blobs_client_example__T19AZ_SPAN_FROM_BUFFERTG4096hZQBdFNbQnZS16az_st
rage_blobs7az_span: error: undefined reference to 'az_span_init'

I do not know C well, is this the expected behavior and should I 
ask the Azure SDK developers to not implement functions within C 
header files?

Kind regards
André
Feb 17 2020
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Tuesday, 18 February 2020 at 05:41:38 UTC, Andre Pany wrote:
 Hi,

 I try to get wrap the "Azure SDK for C" using DPP and have 
 following issue.
 Functions, which are actually implemented in C header files 
 will cause
 linker errors:

 https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/inc/az_span.h#L91

 Example:
 AZ_NODISCARD AZ_INLINE az_span az_span_init(uint8_t * ptr, 
 int32_t length, int32_t capacity) {
   return (az_span){ ._internal = { .ptr = ptr, .length = 
 length, .capacity = capacity, }, };
 }

 Error message:
 /tmp/app.o:az_storage_blobs.d:function 
 _D20blobs_client_example__T19AZ_SPAN_FROM_BUFFERTG4096hZQBdFNbQnZS16az_st
rage_blobs7az_span: error: undefined reference to 'az_span_init'

 I do not know C well, is this the expected behavior and should 
 I ask the Azure SDK developers to not implement functions 
 within C header files?

 Kind regards
 André
I think the problem is that you haven't actually linked in the Azure SDK C library. Dpp translates the header declarations from C to D, but the actual definitions (function bodies) are not part of the process. The executable code for the function definitions should be inside either a static or dynamic library provided by the SDK. From the the project's readme file, it looks like they're using CMake as the build system generator (afterwards both make and ninja should be valid choices for building): mkdir build cd build cmake ../ make In cases like this, it's best to checkout the CMakeLists.txt files of the individual sub project, like this one: https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/CMakeLists.txt As you can see, there are several outputs of the build process, among which: - add_library(az_core ...) This defines a library named az_core which can produce either a static (.a on Linux, .lib on Windows) or dynamic library file (.so on Linux, .dll on Windows). (If no configuration is specified, I think it's static by default). So the final file name would be libaz_core.{a,so} on Linux. For the .c files to be built, a list of include directories must be specified, where the various .h would located (containing function and type declarations). This done like so: target_include_directories (az_core PUBLIC ...) The 'PUBLIC' argument to the target_include_directories specifies that if you want to use the library, you need to use the same include directories, as those needed for building it. - add_executable(az_core_test ..) This defines an executable build output, which looks is only used for testing, so it's not interesting to us, except that it can serve as an example app using the az_core library. --- So in summary, if you want to use the az_core library, you need to: 1. Build it 2. Run Dpp like so: d++ \ --include-path <specify the same as what was given to target_include_directories> <your D file(s) here) <path to libaz_core.a> You will need to repeat the same steps for any other part of the Azure C SDK. -------- TL;DR After I went through all those steps I got a similar linker error for az_http_response_init. After looking where is the actual function definition, it turned out that it's not defined in a .c file, but it is an inline function part of a header file. Searching for az_span_init revealed the same (I could have saved myself some time by reading your message more carefully :D). So, to answer your original question, the problem is that dpp translates only declarations, not function definitions (such as inline functions like that). For now, your best course of action is to translate all inline function definition by hand. Since in C inline functions are mostly short and simple functions (a better alternative to macros), hopefully that won't be too much work. Also, looking at macros like AZ_SPAN_FROM_STR, there's really very little chance that they could be correctly translated automatically. As the things they do are likely not valid even in system D code (without additional casts), so it's better to write your own D functions by hand anyway. Here's what I tried: test.dpp: #include <az_http.h> #include <az_span.h> import std.stdio; void main() { char[] resp = "HTTP/1.2 404 We removed the\tpage!\r\n" ~ "\r\n" ~ "But there is somebody. :-)".dup; az_span response_span = {{ ptr: cast(ubyte*)resp.ptr, length: cast(int)resp.length, capacity: cast(int)resp.length }}; az_http_response response; az_result result = az_http_response_init( &response, response_span); writeln(result); } d++ --compiler ldmd2 --include-path ./inc test.dpp ./build/libaz_core.a
Feb 18 2020
parent reply Andre Pany <andre s-e-a-p.de> writes:
On Tuesday, 18 February 2020 at 08:32:47 UTC, Petar Kirov 
[ZombineDev] wrote:
 On Tuesday, 18 February 2020 at 05:41:38 UTC, Andre Pany wrote:
 Hi,

 I try to get wrap the "Azure SDK for C" using DPP and have 
 following issue.
 Functions, which are actually implemented in C header files 
 will cause
 linker errors:

 https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/inc/az_span.h#L91

 Example:
 AZ_NODISCARD AZ_INLINE az_span az_span_init(uint8_t * ptr, 
 int32_t length, int32_t capacity) {
   return (az_span){ ._internal = { .ptr = ptr, .length = 
 length, .capacity = capacity, }, };
 }

 Error message:
 /tmp/app.o:az_storage_blobs.d:function 
 _D20blobs_client_example__T19AZ_SPAN_FROM_BUFFERTG4096hZQBdFNbQnZS16az_st
rage_blobs7az_span: error: undefined reference to 'az_span_init'

 I do not know C well, is this the expected behavior and should 
 I ask the Azure SDK developers to not implement functions 
 within C header files?

 Kind regards
 André
I think the problem is that you haven't actually linked in the Azure SDK C library. Dpp translates the header declarations from C to D, but the actual definitions (function bodies) are not part of the process. The executable code for the function definitions should be inside either a static or dynamic library provided by the SDK. From the the project's readme file, it looks like they're using CMake as the build system generator (afterwards both make and ninja should be valid choices for building): mkdir build cd build cmake ../ make In cases like this, it's best to checkout the CMakeLists.txt files of the individual sub project, like this one: https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/CMakeLists.txt As you can see, there are several outputs of the build process, among which: - add_library(az_core ...) This defines a library named az_core which can produce either a static (.a on Linux, .lib on Windows) or dynamic library file (.so on Linux, .dll on Windows). (If no configuration is specified, I think it's static by default). So the final file name would be libaz_core.{a,so} on Linux. For the .c files to be built, a list of include directories must be specified, where the various .h would located (containing function and type declarations). This done like so: target_include_directories (az_core PUBLIC ...) The 'PUBLIC' argument to the target_include_directories specifies that if you want to use the library, you need to use the same include directories, as those needed for building it. - add_executable(az_core_test ..) This defines an executable build output, which looks is only used for testing, so it's not interesting to us, except that it can serve as an example app using the az_core library. --- So in summary, if you want to use the az_core library, you need to: 1. Build it 2. Run Dpp like so: d++ \ --include-path <specify the same as what was given to target_include_directories> <your D file(s) here) <path to libaz_core.a> You will need to repeat the same steps for any other part of the Azure C SDK. -------- TL;DR After I went through all those steps I got a similar linker error for az_http_response_init. After looking where is the actual function definition, it turned out that it's not defined in a .c file, but it is an inline function part of a header file. Searching for az_span_init revealed the same (I could have saved myself some time by reading your message more carefully :D). So, to answer your original question, the problem is that dpp translates only declarations, not function definitions (such as inline functions like that). For now, your best course of action is to translate all inline function definition by hand. Since in C inline functions are mostly short and simple functions (a better alternative to macros), hopefully that won't be too much work. Also, looking at macros like AZ_SPAN_FROM_STR, there's really very little chance that they could be correctly translated automatically. As the things they do are likely not valid even in system D code (without additional casts), so it's better to write your own D functions by hand anyway. Here's what I tried: test.dpp: #include <az_http.h> #include <az_span.h> import std.stdio; void main() { char[] resp = "HTTP/1.2 404 We removed the\tpage!\r\n" ~ "\r\n" ~ "But there is somebody. :-)".dup; az_span response_span = {{ ptr: cast(ubyte*)resp.ptr, length: cast(int)resp.length, capacity: cast(int)resp.length }}; az_http_response response; az_result result = az_http_response_init( &response, response_span); writeln(result); } d++ --compiler ldmd2 --include-path ./inc test.dpp ./build/libaz_core.a
Hi Petar, thank you very much for the explanation and the code sample. Filling the az_span anonymous member was the tricky part, I thought it would be not possible to do so, but you showed me the trick. I will do it like you have proposed but had also already created a ticket for the Azure SDK developer: https://github.com/Azure/azure-sdk-for-c/issues/359 There should be a more convenient way to fill a az_span structure. For reference, here is my dockerfile which does the DPP call and linking: ``` dockerfile FROM dlang2/ldc-ubuntu:1.20.0 as ldc RUN apt-get install -y git libssl-dev uuid-dev libcurl4-openssl-dev curl RUN curl -OL https://cmake.org/files/v3.12/cmake-3.12.4-Linux-x86_64.sh \ && mkdir /opt/cmake \ && sh /cmake-3.12.4-Linux-x86_64.sh --prefix=/opt/cmake --skip-license \ && ln -s /opt/cmake/bin/cmake /usr/local/bin/cmake RUN git clone https://github.com/Azure/azure-sdk-for-c.git \ && cd azure-sdk-for-c \ && git submodule update --init --recursive RUN cd azure-sdk-for-c \ && mkdir build \ && cd build \ && cmake ../ \ && make RUN apt-get install -y clang-9 libclang-9-dev RUN ln -s /usr/bin/clang-9 /usr/bin/clang COPY az_storage_blobs.dpp /tmp/ RUN DFLAGS="-L=-L/usr/lib/llvm-9/lib/" dub run dpp -- --help RUN DFLAGS="-L=-L/usr/lib/llvm-9/lib/" dub run dpp -- /tmp/az_storage_blobs.dpp \ --include-path /azure-sdk-for-c/sdk/core/core/inc \ --include-path /azure-sdk-for-c/sdk/core/core/internal \ --include-path /azure-sdk-for-c/sdk/storage/blobs/inc \ --include-path /azure-sdk-for-c/sdk/transport_policies/curl/inc \ --preprocess-only ADD blobs_client_example.d /tmp/blobs_client_example.d RUN ldc2 /tmp/blobs_client_example.d /tmp/az_storage_blobs.d \ /azure-sdk-for-c/build/sdk/core/core/libaz_core.a \ /azure-sdk-for-c/build/sdk/storage/blobs/libaz_storage_blobs.a \ /azure-sdk-for-c/build/sdk/transport_policies/curl/libaz_curl.a \ -of=/tmp/app ``` Kind regards André
Feb 18 2020
parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Tuesday, 18 February 2020 at 09:20:08 UTC, Andre Pany wrote:
 Hi Petar,
Hi Andre, I'm happy to help :)
 thank you very much for the explanation and the code sample.
 Filling the az_span anonymous member was the tricky part,
 I thought it would be not possible to do so, but you showed me
 the trick.
I wouldn't call it a trick, I was using standard struct literal initialization (the very syntax that DIP1031 proposes to deprecate). For example: struct Inner { int x, y; } struct Outer { Inner inner; } // You can initialize Outer in various ways: // 1) auto o1 = Outer(Inner(1, 2)); // 2) Outer o2 = { inner: Inner(1, 2) }; // 3) Outer o3 = { Inner(1, 2) }; // 4) Outer o4 = { inner: { x: 1, y: 2} }; // 5) Outer o5 = { { x: 1, y: 2} }; // 6) Outer o6; o6.inner.x = 1; o6.inner.y = 1; For POD (plain old data) struct like that, all six variants are equivalent (of course there more possible variations). Since there's no `private` protection modifier in C, the only thing C library authors can do is make it inconvenient to access struct fields (by prefixing them with underscores), but they can't really prevent it. For example, without this syntax, in pure C you can initialize a span like this: char my_string[] = "Hey"; az_span span; span._internal.ptr = my_string; span._internal.length = sizeof(my_string) - 1; span._internal.capacity = sizeof(my_string) - 1; And with almost the same syntax you can do this in D: string my_string = "Hey"; az_span span; span._internal.ptr = cast(ubyte*)my_string.ptr; // note: I think this should be safe, because of [1] span._internal.length = my_string.length; span._internal.capacity = my_string.length; It's just that that author wanted to prevent accidental bugs by pushing you to use the inline helper functions or macros (which are technically not needed). [1]: https://github.com/Azure/azure-sdk-for-c/blob/25f8a0228e5f250c02e389f19d88c064c93959c1/sdk/core/core/inc/az_span.h#L22
 I will do it like you have proposed but had also already created
 a ticket for the Azure SDK developer:
 https://github.com/Azure/azure-sdk-for-c/issues/359
 There should be a more convenient way to fill a az_span 
 structure.
To be honest, I don't think the authors will agree to change this, as putting inline functions in headers files is a pretty common practice in both C and C++. There are two benefits to that: 1) Potentially better performance, because the code is easier to inline 2) It's possible to provide header-only libraries (not the case here), that don't require build steps.
 For reference, here is my dockerfile which does the DPP call 
 and linking:
Cool, I'll check it later!
 ``` dockerfile
 FROM dlang2/ldc-ubuntu:1.20.0 as ldc

 RUN apt-get install -y git libssl-dev uuid-dev 
 libcurl4-openssl-dev curl

 RUN curl -OL 
 https://cmake.org/files/v3.12/cmake-3.12.4-Linux-x86_64.sh \
     && mkdir /opt/cmake \
     && sh /cmake-3.12.4-Linux-x86_64.sh --prefix=/opt/cmake 
 --skip-license \
     && ln -s /opt/cmake/bin/cmake /usr/local/bin/cmake

 RUN git clone https://github.com/Azure/azure-sdk-for-c.git \
     && cd azure-sdk-for-c \
     && git submodule update --init --recursive

 RUN cd azure-sdk-for-c \
     && mkdir build \
     && cd build \
     && cmake ../ \
     && make

 RUN apt-get install -y clang-9 libclang-9-dev
 RUN ln -s /usr/bin/clang-9 /usr/bin/clang
 COPY az_storage_blobs.dpp /tmp/

 RUN DFLAGS="-L=-L/usr/lib/llvm-9/lib/" dub run dpp -- --help

 RUN DFLAGS="-L=-L/usr/lib/llvm-9/lib/" dub run dpp -- 
 /tmp/az_storage_blobs.dpp \
     --include-path /azure-sdk-for-c/sdk/core/core/inc \
     --include-path /azure-sdk-for-c/sdk/core/core/internal \
     --include-path /azure-sdk-for-c/sdk/storage/blobs/inc \
     --include-path 
 /azure-sdk-for-c/sdk/transport_policies/curl/inc \
     --preprocess-only

 ADD blobs_client_example.d /tmp/blobs_client_example.d
 RUN  ldc2 /tmp/blobs_client_example.d /tmp/az_storage_blobs.d \
     /azure-sdk-for-c/build/sdk/core/core/libaz_core.a \
     
 /azure-sdk-for-c/build/sdk/storage/blobs/libaz_storage_blobs.a \
     
 /azure-sdk-for-c/build/sdk/transport_policies/curl/libaz_curl.a 
 \
     -of=/tmp/app
 ```

 Kind regards
 André
Cheers, Petar
Feb 18 2020