www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How to build a static lib properly?

reply ryuukk_ <ryuukk.dev gmail.com> writes:
Hello,

I am trying to build: https://github.com/dlang-community/DCD/ as 
a static lib

In the DCD's dub i changed targetType to "staticLibrary"

Then i build:

``dub build -c library`` for the library config


But now the weird part

On my program i try to link it and i get bunch of unreloved 
symbols (druntime stuff)

Even thought i do nothing fancy on the program

I build it this way:

``dmd -m64 -g -debug dcd.lib app.d``

Nothing fancy:


```D
void main(string[] args)
{
	init_dcd();
}

extern(C) void init_dcd();
```


That init_dcd is declared this way on the dll:

```D
extern(C) export void init_dcd()
{

}
```

I get this long error full of 'unresolved external symbosl'


https://gist.github.com/ryuukk/fae57614aab40638bbe00c8d3fbba708

Anyone know what the hell is happening?


Windows10 x64
DMD32 D Compiler v2.102.1
Mar 05 2023
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
 In the DCD's dub i changed targetType to "staticLibrary"


You didn't need to do that, the default was correct.


 Then i build:
 
 ``dub build -c library`` for the library config


Just ``$ dub build`` would've done it (first configuration is default).

Now for the errors, I think its a simple case of you are not linking 
everything in. Use dub, don't do this manually.

Here is the output for linking the client configuration, looks a bit 
different ;)

```
Linking dcd-client
C:\Tools\D\ldc2-1.31.0-windows-multilib\bin\ldc2.exe 
-of.dub\build\client-debug-windows-x86_64-ldc_v1.31.0-DF438D1296B6C9BC0E3D50D4EFA1EC5CE1C82F4ED9169185C58440A2BD5
D999\dcd-client.exe 
.dub\build\client-debug-windows-x86_64-ldc_v1.31.0-DF438D1296B6C9BC0E3D50D4EFA1EC5CE1C82F4ED9169185C58440A2BD5
D999\dcd-client.obj 
common\.dub\build\library-debug-windows-x86_64-ldc_v1.31.0-CF8574F0D3568FA37AB3781BE5A103C1AE1678C5ECADFFF15736AAEF48C
3F12\dcd_common.lib 
C:\Users\alpha\AppData\Local\dub\packages\msgpack-d-1.0.1\msgpack-d\.dub\build\default-debug-windows-x86_64-ldc_v1.31.0-BA2035194DC7872ACAEF6D406F56AF24986B46F8626E1B06E81C1964C3
4BB12\msgpack-d.lib 
dsymbol\.dub\build\library-debug-windows-x86_64-ldc_v1.31.0-3E3FE60C47A7B98F87E3453CE72C2FF075DB7AD708F08100D68AACD7D4A3
D87\dcd_dsymbol.lib 
C:\Users\alpha\AppData\Local\dub\packages\emsi_containers-0.9.0\emsi_containers\.dub\build\library-debug-windows-x86_64-ldc_v1.31.0-04B3937CA3A0BD6CBBA647E8D9AB53AB78AB124AFAD7E416E7433F06812D4D00
emsi_containers.lib 
C:\Users\alpha\AppData\Local\dub\packages\libdparse-0.21.1\libdparse\.dub\build\library-debug-windows-x86_64-ldc_v1.31.0-E2B54AF26AC9977B374E024724763A7CD14ED2B981E84AA640F8193
06B71D90\dparse.lib 
-g
```
Mar 05 2023
next sibling parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Monday, 6 March 2023 at 01:46:22 UTC, Richard (Rikki) Andrew 
Cattermole wrote:

 In the DCD's dub i changed targetType to "staticLibrary"


 You didn't need to do that, the default was correct.


 Then i build:
 
 ``dub build -c library`` for the library config


 Just ``$ dub build`` would've done it (first configuration is 
 default).

 Now for the errors, I think its a simple case of you are not 
 linking everything in. Use dub, don't do this manually.

 Here is the output for linking the client configuration, looks 
 a bit different ;)

 ```
 Linking dcd-client
 C:\Tools\D\ldc2-1.31.0-windows-multilib\bin\ldc2.exe 
 -of.dub\build\client-debug-windows-x86_64-ldc_v1.31.0-DF438D1296B6C9BC0E3D50D4EFA1EC5CE1C82F4ED9169185C58440A2BD5
D999\dcd-client.exe .dub\build\client-debug-windows-x86_64-ldc_v1.31.0-DF438D1296B6C9BC0E3D50D4EFA1EC5CE1C82F4ED9169185C58440A2BD5
D999\dcd-client.obj common\.dub\build\library-debug-windows-x86_64-ldc_v1.31.0-CF8574F0D3568FA37AB3781BE5A103C1AE1678C5ECADFFF15736AAEF48C
3F12\dcd_common.lib C:\Users\alpha\AppData\Local\dub\packages\msgpack-d-1.0.1\msgpack-d\.dub\build\default-debug-windows-x86_64-ldc_v1.31.0-BA2035194DC7872ACAEF6D406F56AF24986B46F8626E1B06E81C1964C3
4BB12\msgpack-d.lib dsymbol\.dub\build\library-debug-windows-x86_64-ldc_v1.31.0-3E3FE60C47A7B98F87E3453CE72C2FF075DB7AD708F08100D68AACD7D4A3
D87\dcd_dsymbol.lib C:\Users\alpha\AppData\Local\dub\packages\emsi_containers-0.9.0\emsi_containers\.dub\build\library-debug-windows-x86_64-ldc_v1.31.0-04B3937CA3A0BD6CBBA647E8D9AB53AB78AB124AFAD7E416E7433F06812D4D00
emsi_containers.lib C:\Users\alpha\AppData\Local\dub\packages\libdparse-0.21.1\libdparse\.dub\build\library-debug-windows-x86_64-ldc_v1.31.0-E2B54AF26AC9977B374E024724763A7CD14ED2B981E84AA640F8193
06B71D90\dparse.lib -g
 ```


Are you saying dub doesn't build a static dcd.lib?

What to do to make it so i get a static dcd.lib file that 
contains all the code it needs?

This comfort me in my desire to no longer use dub ever
Mar 05 2023
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 06/03/2023 2:52 PM, ryuukk_ wrote:

 Are you saying dub doesn't build a static dcd.lib?


It did, but it doesn't contain any of the dependencies.


 What to do to make it so i get a static dcd.lib file that contains all 
 the code it needs?


$ dub build --combined

That may work. No guarantees. Dub isn't setup for distribution like this.
Mar 05 2023
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Monday, 6 March 2023 at 01:57:21 UTC, Richard (Rikki) Andrew 
Cattermole wrote:

 On 06/03/2023 2:52 PM, ryuukk_ wrote:

 Are you saying dub doesn't build a static dcd.lib?


 It did, but it doesn't contain any of the dependencies.


 What to do to make it so i get a static dcd.lib file that 
 contains all the code it needs?


 $ dub build --combined

 That may work. No guarantees. Dub isn't setup for distribution 
 like this.


combined didn't work, i still get a bunch of unresolved symbols 
(although less long):

https://gist.github.com/ryuukk/d59eb89979f1666139ffe0ac95307f20

there needs to be a better solution than to hunt down 
dependencies, then how to build the dependencies and to guess 
what version was used to build said dependencies for the whole 
tree of dependencies one project will use

this is very very poor design

the fact that this is apparently an issue for this long doesn't 
give me hope it's going to be solved anytime soon

if someone from the D team can build a usable static library for 
DCD and its dependencies, then please share to me how, something 
that doesn't involve playing a hunting game on github

that can't be worse than other languages, can it?
Mar 05 2023
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 06/03/2023 3:49 PM, ryuukk_ wrote:

 On Monday, 6 March 2023 at 01:57:21 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 On 06/03/2023 2:52 PM, ryuukk_ wrote:

 Are you saying dub doesn't build a static dcd.lib?


 It did, but it doesn't contain any of the dependencies.


 What to do to make it so i get a static dcd.lib file that contains 
 all the code it needs?


 $ dub build --combined

 That may work. No guarantees. Dub isn't setup for distribution like this.


 
 combined didn't work, i still get a bunch of unresolved symbols 
 (although less long):
 
 https://gist.github.com/ryuukk/d59eb89979f1666139ffe0ac95307f20


I've got it to work via:

$ cd DCD
$ dub build --build-mode=allAtOnce --combined
$ cd ..
$ ldc2 app.d DCD/dcd.lib

```d
void main(string[] args)
{
     init_dcd();
}

extern(C) void init_dcd() {}
```

ldc2 on Windows.

DCD/dcd.lib should be about 23mb.

I did have to hunt some files down, it seems some build commands in DCD 
isn't setup for this (almost certainly not using the environment 
variables that dub offers to handle this situation).
Mar 05 2023
prev sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Monday, 6 March 2023 at 01:52:06 UTC, ryuukk_ wrote:
6B71D90\dparse.lib -g

 ```


 Are you saying dub doesn't build a static dcd.lib?

 What to do to make it so i get a static dcd.lib file that 
 contains all the code it needs?

 This comfort me in my desire to no longer use dub ever


This is not dub's fault. When building a shared library, there's 
a link step, so any external dependencies are linked into the 
shared library just as they are with an executable. There is no 
link step with a static library. That means when you build your 
executable, you need to also link the static library's 
dependencies along with it.

Most of the symbols in your list appear to come from the packages 
listed in the dependencies section of DCD's dub.json:

https://github.com/dlang-community/DCD/blob/master/dub.json

So if you want to do this manually, with DCD as a static library, 
then you also need to build all of those dependencies and link 
them with your executable.
Mar 05 2023
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Monday, 6 March 2023 at 02:00:16 UTC, Mike Parker wrote:

 On Monday, 6 March 2023 at 01:52:06 UTC, ryuukk_ wrote:
 6B71D90\dparse.lib -g

 ```


 Are you saying dub doesn't build a static dcd.lib?

 What to do to make it so i get a static dcd.lib file that 
 contains all the code it needs?

 This comfort me in my desire to no longer use dub ever


 This is not dub's fault. When building a shared library, 
 there's a link step, so any external dependencies are linked 
 into the shared library just as they are with an executable. 
 There is no link step with a static library. That means when 
 you build your executable, you need to also link the static 
 library's dependencies along with it.

 Most of the symbols in your list appear to come from the 
 packages listed in the dependencies section of DCD's dub.json:

 https://github.com/dlang-community/DCD/blob/master/dub.json

 So if you want to do this manually, with DCD as a static 
 library, then you also need to build all of those dependencies 
 and link them with your executable.


This is a bug in DCD in my book

Let's take this example:

Let's pretend Mr A is a developer of a widely used Editor


"A: let's build this tool D people use to get code completion, so 
we can support D"
"A: git clone dcd"
"A: let's install D so we can build it"
"A: dub build -c library"
"A: sweet a dcd.lib file"
"A: feed it into my ABC language compiler"
"A: hmm i am getting unresolved symbols, i thought i compiled a 
static library of that project"
"A: sorry guys, no D support, that's what their build too gave me 
but it doesn't work"

"D people: yeah sorry it doesn't give you a static library of the 
project, you need to individually compile each dependencies it 
uses, some are in the project directory, some are referenced as a 
DUB package so you need to lookup for the source, close it and 
compile it individually too, then you need to do that recursively 
for each dependencies each project will use"


dub should build a static library for the project i build, that 
includes each library it uses that are referenced as "library" 
since the default is "staticLibrary" according to rikki
Mar 05 2023
next sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 06/03/2023 3:09 PM, ryuukk_ wrote:

 This is a bug in DCD in my book


This has nothing to do with DCD. This is on DUB, it was never designed 
for anything related to distribution.

Nobody has attempted to change this. All the information is available 
for it to do it.

Just a matter of adding another cli command that inherits from build.


 dub should build a static library for the project i build, that includes 
 each library it uses that are referenced as "library" since the default 
 is "staticLibrary" according to rikki


So the targetType library was meant to devolve into either staticLibrary 
or dynamicLibrary as determined at a later point in time circa 2012.

Problem is it will never be able to devolve into a shared library. That 
requires code to be modified to work with it.

Right now targetType library is the default, but should really be 
removed at some point in the future.

Its good to remember that shared library support has only really gotten 
to the point in ldc where we can start to see a glimmer of hope in the 
last 2-3 years. Back in 2012 it was practically something very few 
people could even hope to use.

dmd is still a complete write off, but work is being done to now resolve 
that and I'm on the solving the language level issues with a DIP :)
Mar 05 2023
prev sibling next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 3/5/23 9:09 PM, ryuukk_ wrote:

 On Monday, 6 March 2023 at 02:00:16 UTC, Mike Parker wrote:



 This is not dub's fault. When building a shared library, there's a 
 link step, so any external dependencies are linked into the shared 
 library just as they are with an executable. There is no link step 
 with a static library. That means when you build your executable, you 
 need to also link the static library's dependencies along with it.

 Most of the symbols in your list appear to come from the packages 
 listed in the dependencies section of DCD's dub.json:

 https://github.com/dlang-community/DCD/blob/master/dub.json

 So if you want to do this manually, with DCD as a static library, then 
 you also need to build all of those dependencies and link them with 
 your executable.


 


Well, you can create a library that contains all the static libraries, 
you just can't use dub to do it for you automatically, or even manually 
(dub ignores `.a` files when specified as source files).


 dub should build a static library for the project i build, that includes 
 each library it uses that are referenced as "library" since the default 
 is "staticLibrary" according to rikki
 


Dub expects to have dub-built artifacts available to link. This problem 
only occurs if you are building outside of dub, without proper 
dependency management.

Dub indeed does not play nice with exporting artifacts for other build 
systems. It's something that should be addressed.

Keep in mind though, that you still need to specify any shared library 
dependencies, since static libs cannot contain shared libraries or 
automatic references to them.

-Steve
Mar 05 2023
prev sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Monday, 6 March 2023 at 02:09:23 UTC, ryuukk_ wrote:


 dub should build a static library for the project i build, that 
 includes each library it uses that are referenced as "library" 
 since the default is "staticLibrary" according to rikki


What you're asking for is a different use case. `staticLibrary` 
is for compiling a specific dub package as a static library. It 
does not imply that all of that package's dependencies should 
also be compiled into a single static library. Nor should it.

You're asking for a package and its dependencies to be bundled 
for use outside of the dub ecosystem as a single static library. 
I won't say it's not a legitimate use case, it's just not one 
that dub currently supports, nor was it originally intended to (I 
wouldn't expect it to be a common one either, but then again 
common expectations are always changing).

As a potential dub enhancement, e.g., `staticLibraryBundle`, I do 
agree it's worth exploring. I don't believe you can expect every 
package to "just work" in that environment. As Steve mentioned, 
there will always be link-time dependencies to any shared 
libraries on which those bundled libraries depend. And some 
packages may be set up in a way that causes issues, as Rikki 
noted when he said he had to make some tweaks on his successful 
build.

But for the time being, dealing with static libraries in D is 
just the same as dealing with them in the C and C++ world. They 
always have been a PITA to deal with, and that's why the trend in 
recent years has been to move away from them. Build tools like 
dub hide them from you when used as intended. It's when you mix 
build systems that you run into trouble.

Still, I suggest you send an email to social dlang.org as part of 
the Gripes and Wishes campaign so that this gets added into the 
dataset. Anything that enhances dub's usability should be looked 
at.
Mar 05 2023
parent Hipreme <msnmancini hotmail.com> writes:
On Monday, 6 March 2023 at 05:59:09 UTC, Mike Parker wrote:

 On Monday, 6 March 2023 at 02:09:23 UTC, ryuukk_ wrote:


 dub should build a static library for the project i build, 
 that includes each library it uses that are referenced as 
 "library" since the default is "staticLibrary" according to 
 rikki


 What you're asking for is a different use case. `staticLibrary` 
 is for compiling a specific dub package as a static library. It 
 does not imply that all of that package's dependencies should 
 also be compiled into a single static library. Nor should it.

 You're asking for a package and its dependencies to be bundled 
 for use outside of the dub ecosystem as a single static 
 library. I won't say it's not a legitimate use case, it's just 
 not one that dub currently supports, nor was it originally 
 intended to (I wouldn't expect it to be a common one either, 
 but then again common expectations are always changing).

 As a potential dub enhancement, e.g., `staticLibraryBundle`, I 
 do agree it's worth exploring. I don't believe you can expect 
 every package to "just work" in that environment. As Steve 
 mentioned, there will always be link-time dependencies to any 
 shared libraries on which those bundled libraries depend. And 
 some packages may be set up in a way that causes issues, as 
 Rikki noted when he said he had to make some tweaks on his 
 successful build.

 But for the time being, dealing with static libraries in D is 
 just the same as dealing with them in the C and C++ world. They 
 always have been a PITA to deal with, and that's why the trend 
 in recent years has been to move away from them. Build tools 
 like dub hide them from you when used as intended. It's when 
 you mix build systems that you run into trouble.

 Still, I suggest you send an email to social dlang.org as part 
 of the Gripes and Wishes campaign so that this gets added into 
 the dataset. Anything that enhances dub's usability should be 
 looked at.



Having a way to define a static library bundle is really great. 
It does help a lot when you create a package where it is 
initialized in non D code.

Right now, this is the third time I'm going to need it for my 
engine, for MacOS I need to manually assign my dependencies, and 
having only the main code as a library doesn't work for that 
purpose. I have done before a simple program which executes dub 
describe for taking the dependencies then I build the program as 
a staticLibrary, problem is that this requires a 2-pass build and 
it is not standard.
Mar 08 2023
prev sibling parent ryuukk_ <ryuukk.dev gmail.com> writes:
  Use dub, don't do this manually.


i build DCD with DUB, not manually


i ask dub to give me a static lib, dcd.lib is the file he gave 
me, that should be all i


yet i get the errors
Mar 05 2023
prev sibling parent reply Ruby The Roobster <rubytheroobster yandex.com> writes:
On Monday, 6 March 2023 at 00:55:04 UTC, ryuukk_ wrote:

 Hello,

 I am trying to build: https://github.com/dlang-community/DCD/ 
 as a static lib

 [...]


Try ```dub build --build-mode=allAtOnce```.
Mar 05 2023
parent ryuukk_ <ryuukk.dev gmail.com> writes:
On Monday, 6 March 2023 at 02:54:42 UTC, Ruby The Roobster wrote:

 On Monday, 6 March 2023 at 00:55:04 UTC, ryuukk_ wrote:

 Hello,

 I am trying to build: https://github.com/dlang-community/DCD/ 
 as a static lib

 [...]


 Try ```dub build --build-mode=allAtOnce```.


That didn't work

However, changing 
https://github.com/dlang-community/DCD/blob/master/dsymbol/dub.json#L7 into
"sourceLibrary" worked, along with: "--combined"

I maintain what i said, this is poorly designed and seems to be 
also buggy too
Mar 05 2023