www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Secure dependency management

reply Chris Piker <chris hoopjump.com> writes:
Hi D

So one of the projects I've been working on is moving closer to 
production. Currently, the pull/build/test/install cycle is 
handled by git, dub and GNU make.

Currently I let dub fetch dependencies off the Internet, but for 
mission reliability I would like to be able to handle the process 
without Internet access.  Dub looks like it supports local 
repositories, but before just start "Doing something" are there 
any practices the community would recommend for D supply-chain 
management?

Thanks for any links and tips,
Jan 04
parent reply evilrat <evilrat666 gmail.com> writes:
On Saturday, 4 January 2025 at 20:33:55 UTC, Chris Piker wrote:

 Hi D

 So one of the projects I've been working on is moving closer to 
 production. Currently, the pull/build/test/install cycle is 
 handled by git, dub and GNU make.

 Currently I let dub fetch dependencies off the Internet, but 
 for mission reliability I would like to be able to handle the 
 process without Internet access.  Dub looks like it supports 
 local repositories, but before just start "Doing something" are 
 there any practices the community would recommend for D 
 supply-chain management?

 Thanks for any links and tips,


I don't think there is D specific rules on dependency management, 
as most people use it for small scale personal/hobby projects 
only.

If your project can't take the risk of losing online dependencies 
you might just want to put and commit them under your project's 
version control system - this is sometimes done in Go, people 
there justify it that unlike JS and some other languages with 
tons of generated stuff Go packages are relatively small so it is 
just a natural choice to place them next to your code.

And in Git for example submodules (other git repos linked to your 
repo) are also a thing, however by default it won't clone them 
without explicit recursive flag, and even you have git experience 
it is somewhat confusing to upgrade them (at least for me).

As for the dub itself, it has very scarce info on that, it has 
local overrides and stuff, but I'd say this is mostly for quick 
fixing the build issues, not a production solution.
So just having `dub.selections.json` in your repository pointing 
to a local (committed dependencies in that same repo) packages is 
viable option, even if something goes wrong you can always change 
it in dev environment to fix problems and commit back.
Unless you have license issues with dependencies this is probably 
the most secure one.
Jan 04
parent reply Chris Piker <chris hoopjump.com> writes:
On Sunday, 5 January 2025 at 06:21:15 UTC, evilrat wrote:

 If your project can't take the risk of losing online 
 dependencies you might just want to put and commit them under 
 your project's version control system


That's nice and simple, so maybe a good idea, but it does make it 
more difficult to get upstream changes when desired.  For example 
one of my dependencies `dpq2` actually reduced it's dependency 
count over the last year, so and I definitely wanted those 
changes.


 And in Git for example submodules (other git repos linked to 
 your repo) are also a thing, however by default it won't clone 
 them without explicit recursive flag, and even you have git 
 experience it is somewhat confusing to upgrade them (at least 
 for me).


This is the route I'll probably take, i.e. make git submodules 
for my dependencies, (there's only 7 of them).  Even though 
submodules can be a pain, we use them a lot around here so I'm 
committed to dealing with their idiosyncrasies.


 So just having `dub.selections.json` in your repository 
 pointing to a local (committed dependencies in that same repo)


This is very useful advice, thanks!  I didn't know about 
dub.selections.json, especially since it's [manual 
page](https://dub.pm/dub-guide/selections/) is blank. That could 
work well with sub modules.

...now to find out how to use dub.selections.json.
Jan 05
parent reply Chris Piker <chris hoopjump.com> writes:
Here's what I ended up doing in case it's either useful for 
others, or a really bad idea and serves as an anti-example.

0) For context, the working directory has roughly this setup 
(after git submodule calls):
```
git-root
  |-- main_project
  |-- deps/
  |    |-- vibe-serialization
  |    |-- ... more here
  |
  |-- makefile
```

1) Add dependencies as git submodules, for example:
```bash
    git submodule add https://github.com/vibe-d/vibe-serialization 
deps/vibe-serialization

```

2) In the top level build file (makefile in my case) the 
following happens:
```bash
    dub add-local vibe-serialization 1.0.7


    cd main_project && dub --skip-registry=all build

    dub remove-local vibe-serialization

```

3) On a regular basis:
```bash
    rm -r $HOME/.dub
```
   You never know what could be hiding in there.

A person can just `cd main_project && dub build` and standard 
things will happen, i.e. dependencies will be downloaded from the 
internet.  However in a production setting `make` is run from the 
top level and only the dependencies specified in the sub modules 
are used.

This should work *unless* two builds are happening at the same 
time in the same account since `add-local` and `remove-local` are 
**user-wide** and affect an entire user's home directory at a 
time.  Not too bad if the user is a real person, *terrible* if 
it's the account used by a build host.

It works, but I'm sure there's a better way.
Jan 05
parent evilrat <evilrat666 gmail.com> writes:
On Monday, 6 January 2025 at 03:04:40 UTC, Chris Piker wrote:

 Here's what I ended up doing in case it's either useful for 
 others, or a really bad idea and serves as an anti-example.

 0) For context, the working directory has roughly this setup 
 (after git submodule calls):
 ```
 git-root
  |-- main_project
  |-- deps/
  |    |-- vibe-serialization
  |    |-- ... more here
  |
  |-- makefile
 ```

 1) Add dependencies as git submodules, for example:
 ```bash
    git submodule add 
 https://github.com/vibe-d/vibe-serialization 
 deps/vibe-serialization

 ```

 2) In the top level build file (makefile in my case) the 
 following happens:
 ```bash
    dub add-local vibe-serialization 1.0.7


    cd main_project && dub --skip-registry=all build

    dub remove-local vibe-serialization

 ```

 3) On a regular basis:
 ```bash
    rm -r $HOME/.dub
 ```
   You never know what could be hiding in there.

 A person can just `cd main_project && dub build` and standard 
 things will happen, i.e. dependencies will be downloaded from 
 the internet.  However in a production setting `make` is run 
 from the top level and only the dependencies specified in the 
 sub modules are used.

 This should work *unless* two builds are happening at the same 
 time in the same account since `add-local` and `remove-local` 
 are **user-wide** and affect an entire user's home directory at 
 a time.  Not too bad if the user is a real person, *terrible* 
 if it's the account used by a build host.

 It works, but I'm sure there's a better way.



There is nothing wrong with your example, having two ways to 
build artifacts maybe confusing but this is just fine, except 
maybe that you should clearly state your intentions and name 
scripts/makefile accordingly (e.g. build_prod.sh/build_dev.sh or 
something, meaning that is it only going to work in specific 
environment because this is a part of specific procedure)

There is a problem with `dub add-local` though, it is only useful 
for developer on that same machine while tinkering with 
problematic packages. But this is an extra step that is easily 
can be forgotten leading to confusion.
I think it can be avoided entirely by simply setting explicit 
paths to dependencies in `dub.selections.json` (not suitable for 
library projects as you can't freeze version for your users), not 
sure if it will actually work with relative paths but I guess it 
should.

__dub.selection.json__:
```json
{
	"fileVersion": 1,
	"versions": {
		"godot-dlang": {"path":"C:/godot/bin/godot-dlang"},
		"intel-intrinsics": "1.11.18",
		"pyd": "~master"
	}
}
```
Jan 05