www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How to make dub recursively build subPackages?

reply Vijay Nayar <madric gmail.com> writes:
Greetings folks,

In my project, I have a parent package with several sub-packages, 
each of which builds into either a library or an executable.

I first started observing odd problems when I was running `dub 
build`, it would complain about different versions of vibe-d 
present, and it suggested running `dub upgrade`. After doing 
this, I noticed that most subPackages were not actually being 
upgraded.

The only thing I have found thus far is to manually run each 
subPackage one at a time, e.g. `dub :proto; dub :common; ...`.

Is it possible to get `dub upgrade` to recursively work on all 
sub-packages?

My parent package dub.sdl file:
```
name "funnel"
description "An in-memory queryable database for processing 
extreme loads of current data."
authors "Vijay Nayar"
copyright "Copyright © 2019, Vijay Nayar"
license "proprietary"
targetType "none"
targetPath "target"
dependency "funnel:proto" version=">=0.0.0"
dependency "funnel:spout" version=">=0.0.0"
dependency "funnel:stem" version=">=0.0.0"
dependency "funnel:mouth" version=">=0.0.0"
dependency "funnel:common" version=">=0.0.0"
subPackage "./common"
subPackage "./proto"
subPackage "./mouth"
subPackage "./stem"
subPackage "./spout"
```

Each subPackage is structured in the same way, for example, the 
common subPackage:
```
authors "Vijay Nayar"
copyright "Copyright © 2019, Vijay Nayar"
description "Common logic between the mouth and spout components."
license "proprietary"
name "common"
targetType "library"
targetPath "target"
dependency "funnel:proto" version="*"
dependency "poodinis" version="~>8.0.3"
dependency "vibe-d" version="~>0.9.4"
```

I mostly followed the dub documentation in setting up my project. 
https://dub.pm/package-format-sdl.html
Feb 02 2022
next sibling parent Vijay Nayar <madric gmail.com> writes:
On Wednesday, 2 February 2022 at 10:14:25 UTC, Vijay Nayar wrote:

 Greetings folks,

 In my project, I have a parent package with several 
 sub-packages, each of which builds into either a library or an 
 executable.

 [...]


I should point out there is 1 exception to subPackages not being 
recursive. The `dub clean` command does recurse, and there is 
source code for it here: 
https://github.com/dlang/dub/blob/3abaa4d5b7c3b2c21ab75370cd5330e9ae7bbd12/source/dub/dub.d#L880
Feb 02 2022
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 2/2/22 5:14 AM, Vijay Nayar wrote:

 Greetings folks,
 
 In my project, I have a parent package with several sub-packages, each 
 of which builds into either a library or an executable.
 
 I first started observing odd problems when I was running `dub build`, 
 it would complain about different versions of vibe-d present, and it 
 suggested running `dub upgrade`. After doing this, I noticed that most 
 subPackages were not actually being upgraded.
 
 The only thing I have found thus far is to manually run each subPackage 
 one at a time, e.g. `dub :proto; dub :common; ...`.
 
 Is it possible to get `dub upgrade` to recursively work on all 
 sub-packages?
 
 My parent package dub.sdl file:
 ```
 name "funnel"
 description "An in-memory queryable database for processing extreme 
 loads of current data."
 authors "Vijay Nayar"
 copyright "Copyright © 2019, Vijay Nayar"
 license "proprietary"
 targetType "none"
 targetPath "target"
 dependency "funnel:proto" version=">=0.0.0"
 dependency "funnel:spout" version=">=0.0.0"
 dependency "funnel:stem" version=">=0.0.0"
 dependency "funnel:mouth" version=">=0.0.0"
 dependency "funnel:common" version=">=0.0.0"
 subPackage "./common"
 subPackage "./proto"
 subPackage "./mouth"
 subPackage "./stem"
 subPackage "./spout"
 ```
 
 Each subPackage is structured in the same way, for example, the common 
 subPackage:
 ```
 authors "Vijay Nayar"
 copyright "Copyright © 2019, Vijay Nayar"
 description "Common logic between the mouth and spout components."
 license "proprietary"
 name "common"
 targetType "library"
 targetPath "target"
 dependency "funnel:proto" version="*"
 dependency "poodinis" version="~>8.0.3"
 dependency "vibe-d" version="~>0.9.4"
 ```
 
 I mostly followed the dub documentation in setting up my project. 
 https://dub.pm/package-format-sdl.html


If you have them in the same repository, my recommendation is to use 
path dependencies instead of versions.

So, e.g.:

```sdl
dependency "funnel:proto" path="./proto" // in main dub.sdl
dependency "funnel:proto" path="../proto" // in sibling package
```

Because otherwise, dub is going to try and fetch the appropriate version 
from an online repository and not use your local files.

Honestly, for packages in the same repository, I'm not sure why you 
would version them separately from the main package. I don't even know 
if that works.

-Steve
Feb 02 2022
parent reply Vijay Nayar <madric gmail.com> writes:
On Wednesday, 2 February 2022 at 14:07:08 UTC, Steven 
Schveighoffer wrote:

 On 2/2/22 5:14 AM, Vijay Nayar wrote:
 If you have them in the same repository, my recommendation is 
 to use path dependencies instead of versions.

 So, e.g.:

 ```sdl
 dependency "funnel:proto" path="./proto" // in main dub.sdl
 dependency "funnel:proto" path="../proto" // in sibling package
 ```

 Because otherwise, dub is going to try and fetch the 
 appropriate version from an online repository and not use your 
 local files.

 Honestly, for packages in the same repository, I'm not sure why 
 you would version them separately from the main package. I 
 don't even know if that works.

 -Steve


I made this change and it did indeed work correctly, thank you 
for that! Truthfully, it was not entirely clear to me how dub was 
deciding where to go to build. I had assumed that this was being 
done via the `subPackage` lines.

The examples given in the offical documentation were also using 
versions, and I was following that: 
https://dub.pm/package-format-sdl.html#sub-packages
Feb 02 2022
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 2/2/22 1:42 PM, Vijay Nayar wrote:


 
 I made this change and it did indeed work correctly, thank you for that! 
 Truthfully, it was not entirely clear to me how dub was deciding where 
 to go to build. I had assumed that this was being done via the 
 `subPackage` lines.


Like many things in dub, there are multiple ways to do things. I've done 
some of the other things in some projects, and I found this works the 
best for intra-repository dependencies.

If you are depending on a subproject from outside the repository, then I 
think you have to use the version of the parent repository as a 
specified version, since the whole repo is tagged the same.


 
 The examples given in the offical documentation were also using 
 versions, and I was following that: 
 https://dub.pm/package-format-sdl.html#sub-packages
 
 


Yeah, that example shows a repository which exists purely to combine 
multiple subprojects into one repo (note the targetType of none). Also 
note the `version="*"`, which is basically "I don't care, whatever is 
there". I think you could also do a path as well, but I think even if 
you use the `*` version, you might have dub trying to fetch the 
dependency from the internet. I'm more comfortable with the `path` 
directive, since I know it can't possibly think that it has to go 
elsewhere to resolve that dependency.

Dub is kind of a hot mess in terms of the dependency resolution and ways 
to specify projects. I would love to see it cleaned up/reimplemented.

-Steve
Feb 02 2022
parent reply Vijay Nayar <madric gmail.com> writes:
On Wednesday, 2 February 2022 at 19:03:35 UTC, Steven 
Schveighoffer wrote:

 On 2/2/22 1:42 PM, Vijay Nayar wrote:


 Dub is kind of a hot mess in terms of the dependency resolution 
 and ways to specify projects. I would love to see it cleaned 
 up/reimplemented.

 -Steve


For your larger more complex projects, what build system do you 
use? So far I've been having good luck with dub, but I haven't 
done any mixed-language projects lately either.
Feb 02 2022
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 2/2/22 4:42 PM, Vijay Nayar wrote:

 On Wednesday, 2 February 2022 at 19:03:35 UTC, Steven Schveighoffer wrote:

 On 2/2/22 1:42 PM, Vijay Nayar wrote:


 Dub is kind of a hot mess in terms of the dependency resolution and 
 ways to specify projects. I would love to see it cleaned 
 up/reimplemented.


 
 For your larger more complex projects, what build system do you use? So 
 far I've been having good luck with dub, but I haven't done any 
 mixed-language projects lately either.
 


I only use dub for my D projects.

I'm not saying dub is completely bad. Just that the dependency 
resolution needs a lot of work, and probably a rewrite.

-Steve
Feb 03 2022