• Sebastiaan Koppe (22/22) Nov 19 2019 After having had another day where my CI can't build because of
• Sebastiaan Koppe (26/36) Jan 30 2020 I have updated the script to cache all routes. This means the
• Robert Schadek (1/1) Jan 30 2020 very nice, thank you

Sebastiaan Koppe <mail skoppe.eu> writes:

After having had another day where my CI can't build because of 
gateway errors, this morning I had enough. (I setup an uptime 
robot so we can see how bad it actually is [1], it checks every 5 
min and I just started it.)

I know, I can setup my CI to cache dub packages. But that is just 
an optimisation.

Things ought to just work.

So this morning I started on a quick fix. I took a look at the 
source and concluded that for dub packages hosted on gitlab and 
github you can simply redirect to gitlab's and github's 
respective archives. That is what the dub registry does anyway.

To that end I setup that logic in a cloudflare worker [2]. So for 
metadata it redirects to the api endpoint on code.dlang.org 
(which is pretty stable), and for to actual packages directly to 
gitlab or github.

Use it like so:

`dub build --registry=https://dub.bytecraft.nl`

Of course, this is a bandaid solution. Probably dub can be made 
to do this as well. And likely should. If it were up to me I 
would go serverless.

[1] https://stats.uptimerobot.com/6mQX4Crw2L
[2] https://github.com/skoppe/dub-registry-mirror

Sebastiaan Koppe <mail skoppe.eu> writes:

On Tuesday, 19 November 2019 at 13:27:31 UTC, Sebastiaan Koppe 
wrote:
 After having had another day where my CI can't build because of 
 gateway errors, this morning I had enough. (I setup an uptime 
 robot so we can see how bad it actually is [1], it checks every 
 5 min and I just started it.)

 [...]

 Of course, this is a bandaid solution. Probably dub can be made 
 to do this as well. And likely should. If it were up to me I 
 would go serverless.

 [1] https://stats.uptimerobot.com/6mQX4Crw2L
 [2] https://github.com/skoppe/dub-registry-mirror

I have updated the script to cache all routes. This means the 
registry mirror at dub.bytecraft.nl will also work when the 
registry at code.dlang.org is completely down, **if** your 
dependencies have been cached previously.

The cache works in the following manner:

1. if a cache entry exists for a given route and it is no longer 
than 5 minutes old: return that
2. if a cache entry is older than 5 minutes: fetch from upstream, 
cache it and return that
3. if a cache entry is older than 5 minutes and upstream is down: 
reset cache expiry and return from cache
4. if a cache entry doesn't exist: fetch from upstream, cache it 
and return that
5. if a cache entry doesn't exist and upstream is down: return 
error

On top of that there is a 5 second timeout on fetches from 
upstream.

Essentially a cached entry is kept up-to-date (5 minute lag) 
whenever upstream is alive, otherwise stale data is returned if 
available.

Use it like so:

`dub build --registry=https://dub.bytecraft.nl`

Code at 
https://github.com/skoppe/dub-registry-mirror/blob/master/worker.js

Robert Schadek <rschadek symmetryinvestments.com> writes:

very nice, thank you