www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Alternatives to travis-ci

reply Mathias LANG <geod24 gmail.com> writes:
As many of you might be aware already, travis-ci is killing their 
open-source plans.
They haven't sunset it yet,, but they introduced a set of 
restrictions that makes them completely uncompetitive and 
impractical to use compared to other services on the market.

It started with an announcement 
(https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing) 
after they got bought, and pretty quickly people started to 
realize it was much less work to migrate to another service than 
to send an email every other days.

It looks like they don't even provide extra credits anymore: 
https://news.ycombinator.com/item?id=25338983

So if you are maintaining an open-source library, you should 
seriously consider switching away from it. Personally, I favor 
Github Actions, as it's multi-platform (Linux/Mac/Windows), free 
for open source, and has good community-maintained support for D 
(https://github.com/dlang-community/setup-dlang). For Linux only, 
CircleCI seems like a decent alternative.

If you're on buildkite, be aware that we might be using your 
travis-ci script to run your tests (for example: 
https://github.com/dlang/ci/blob/e26bf0cca636394a90ea56652cae445609032d14/buildkite/build_project.sh#L163-L164).

If you're thinking about using Github Actions, it's as simple as 
using `- uses: dlang-community/setup-dlang v1`. That will gives 
you `dmd-latest` to get started. If you want a more advanced 
usage, DMD's workflow file contains a lot of comment which might 
help you with some patterns, for example how to add an extra row 
to a matrix (spoiler: you can't, you can only remove rows): 
https://github.com/dlang/dmd/blob/master/.github/workflows/runnable_cxx.yml

If you plan of using CircleCI, I can recommend to look at our 
configuration: 
https://github.com/bpfkorea/agora/blob/ea9f26c3d09527f79c16365633923b472393571e/.circleci/config.yml

As you can see, the only tricky part is to make sure you get the 
merge commit generated by Github, and not the HEAD of the PR.

Happy hacking!
Dec 10 2020
next sibling parent reply M.M. <matus email.cz> writes:
On Thursday, 10 December 2020 at 11:42:41 UTC, Mathias LANG wrote:

 As many of you might be aware already, travis-ci is killing 
 their open-source plans.
 They haven't sunset it yet,, but they introduced a set of 
 restrictions that makes them completely uncompetitive and 
 impractical to use compared to other services on the market.

 [...]


Thank you for sharing and suggesting alternatives. Is GitLab CI 
similarly easy/friendly/suitable for D? Does anyone use it?
Dec 10 2020
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Thursday, 10 December 2020 at 11:52:41 UTC, M.M. wrote:

 On Thursday, 10 December 2020 at 11:42:41 UTC, Mathias LANG 
 wrote:

 As many of you might be aware already, travis-ci is killing 
 their open-source plans.
 They haven't sunset it yet,, but they introduced a set of 
 restrictions that makes them completely uncompetitive and 
 impractical to use compared to other services on the market.

 [...]


 Thank you for sharing and suggesting alternatives. Is GitLab CI 
 similarly easy/friendly/suitable for D? Does anyone use it?


Gitlab.org (so not self-hosted version) CI is only free for 
linux. osx and windows runners require to buy "minutes".

Then it is as suitable a any other CI service that supports 
docker images, meaning that D is not supported natively (but 
really the docker image solve this issue).

On top of that you have facilities to release binaries, deploy 
packages or publish a static website.

It has always been stable and reliable for me but I run it let's 
say 5 times per day top.

To make you an idea, see https://gitlab.com/basile.b/dlang-ci.
Dec 10 2020
parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 10 December 2020 at 16:30:43 UTC, Basile B. wrote:

 On Thursday, 10 December 2020 at 11:52:41 UTC, M.M. wrote:

 On Thursday, 10 December 2020 at 11:42:41 UTC, Mathias LANG 
 wrote:

 As many of you might be aware already, travis-ci is killing 
 their open-source plans.
 They haven't sunset it yet,, but they introduced a set of 
 restrictions that makes them completely uncompetitive and 
 impractical to use compared to other services on the market.

 [...]


 Thank you for sharing and suggesting alternatives. Is GitLab 
 CI similarly easy/friendly/suitable for D? Does anyone use it?


 Gitlab.org (so not self-hosted version) CI is only free for 
 linux. osx and windows runners require to buy "minutes".

 Then it is as suitable a any other CI service that supports 
 docker images, meaning that D is not supported natively (but 
 really the docker image solve this issue).

 On top of that you have facilities to release binaries, deploy 
 packages or publish a static website.

 It has always been stable and reliable for me but I run it 
 let's say 5 times per day top.

 To make you an idea, see https://gitlab.com/basile.b/dlang-ci.


And for a bit more advanced example, at work we have a project 
that includes two micro-services written in D and hosted on 
GitLab with the following features / techniques:
* Using a custom docker+buildx docker image: [0]
* Mono-repo project organization via dub subpackages
* Each service is built and deployed as a separate runner image, 
but the build dependencies are shared in a common dependencies 
builder image
* We use GitLab Container Registry to push images and to pull 
cache from
* Multi-stage Dockerfile build which has caching of dub 
dependencies and separate builder and run-time environments [1]
* docker buildx bake is used to automate the docker image build, 
tag and push processes [2]
* docker buildx bake targets + GitLab CI matrix allows defining 
jobs for each micro-service docker image without repetition [3]

I want to thank the LDC team and especially Mathias LANG for 
their work on making D on Alpine docker images possible.

[0]: 
https://gitlab.com/jarvis-network/base/container-images/docker-buildx

[1]: 
https://gitlab.com/jarvis-network/apps/exchange/price-feed/-/blob/1fa5624964165b34b987ede0728eb71bdb59d403/Dockerfile

[2]: 
https://gitlab.com/jarvis-network/apps/exchange/price-feed/-/blob/1fa5624964165b34b987ede0728eb71bdb59d403/docker-bake.hcl

[3]: 
https://gitlab.com/jarvis-network/apps/exchange/price-feed/-/blob/1fa5624964165b34b987ede0728eb71bdb59d403/.gitlab-ci.yml
Dec 11 2020
prev sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 10 December 2020 at 11:52:41 UTC, M.M. wrote:

 On Thursday, 10 December 2020 at 11:42:41 UTC, Mathias LANG 
 wrote:

 As many of you might be aware already, travis-ci is killing 
 their open-source plans.
 They haven't sunset it yet,, but they introduced a set of 
 restrictions that makes them completely uncompetitive and 
 impractical to use compared to other services on the market.

 [...]


 Thank you for sharing and suggesting alternatives. Is GitLab CI 
 similarly easy/friendly/suitable for D? Does anyone use it?


GitHub Actions has the least mature UI (compared to every other 
CI service), but for most open-source D projects hosted on GitHub 
it is easily the best choice, thanks to 
https://github.com/dlang-community/setup-dlang which makes it 
trivial to add CI for Windows, Linux and macOS to your project. 
Probably the biggest disadvantage is that they don't offer the 
option to restart individual jobs, which is partial offset by 
offering noticeably faster machines than Travis CI did.

If you have more experience and you want more control, than the 
other services are good choices as well - now CircleCI and Azure 
Pipelines also have Windows/Linux and macOS support. SemaphoreCI 
is also good, but only has macOS and Linux support.
With services like that you have three good options:
* Use the official install.sh script (https://dlang.org/install) 
on the default build environment. The script recently gained 
Windows support (in addition to Linux and macOS).
* Use an existing Docker image with D support:
   * https://hub.docker.com/r/dlang2/ldc-ubuntu
   * https://hub.docker.com/r/bpfk/agora-builder (you'd have to 
ask Mathias whether they're going to support it for third-party 
projects :P)
* Build your own Docker image, by either:
   * Using compilers package by the Linux distributions
   * Using the install.sh script
   * Manually downloading and installing dmd/ldc packages
   * Building a compiler from source

Outside of the GitHub monoculture, GitLab, as mentioned by 
Basile, is the best choice as they offer the most integrated 
development environment.
Dec 11 2020
next sibling parent reply drug <drug2004 bk.ru> writes:
On 12/11/20 1:34 PM, Petar Kirov [ZombineDev] wrote:

 On Thursday, 10 December 2020 at 11:52:41 UTC, M.M. wrote:
 [snip] 
 Outside of the GitHub monoculture, GitLab, as mentioned by Basile, is 
 the best choice as they offer the most integrated development environment.


It is also can be self-hosted, that really a big plus.

But what push me off is those continuous changes of GUI that sometimes 
makes me thinking they just try to sure investors that money has been 
spent to reasons.
Dec 11 2020
parent Basile B. <b2.temp gmx.com> writes:
On Friday, 11 December 2020 at 11:30:39 UTC, drug wrote:

 On 12/11/20 1:34 PM, Petar Kirov [ZombineDev] wrote:

 On Thursday, 10 December 2020 at 11:52:41 UTC, M.M. wrote:
 [snip] Outside of the GitHub monoculture, GitLab, as mentioned 
 by Basile, is the best choice as they offer the most 
 integrated development environment.


 It is also can be self-hosted, that really a big plus.

 But what push me off is those continuous changes of GUI that 
 sometimes makes me thinking they just try to sure investors 
 that money has been spent to reasons.


Well on the last deployment they pushed a change that has for 
effect to display a tree view of the files changed in a MR on the 
left main area and the diff on the right main area. I would have 
rather choose to keep a full diff in the main area and added a 
combobox on the top that display the tree, but apart that, no UI 
changes were anoying so far.

Otherwise annoyances i've encountered:
- two few very specific or minor bugs/problems
- a big bug in their link system but it is fixed now as it was 
"epic"

It's still good for your own stuff, and apparently even for big 
companies (even if they self host gitlab in this case).

In the worst of the cases you can use external CI for windows and 
osx. For example the binary release of dexed for windows is build 
on appveyor, triggered by tag pushes on the main repo that's 
hosted on gitlab.org.
Dec 11 2020
prev sibling next sibling parent reply kinke <noone nowhere.com> writes:
On Friday, 11 December 2020 at 10:34:34 UTC, Petar Kirov 
[ZombineDev] wrote:

 GitHub Actions has the least mature UI (compared to every other 
 CI service), but for most open-source D projects hosted on 
 GitHub it is easily the best choice, thanks to 
 https://github.com/dlang-community/setup-dlang which makes it 
 trivial to add CI for Windows, Linux and macOS to your project. 
 Probably the biggest disadvantage is that they don't offer the 
 option to restart individual jobs, which is partial offset by 
 offering noticeably faster machines than Travis CI did.


I tend to agree. But if your CI scripts are complex, there's one 
big catch for the MS solutions (GitHub Actions / Azure Pipelines) 
- they don't use a regular YAML parser and don't support YAML 
anchors for reusable pieces. Azure Pipelines offers custom 
'templates' instead (essentially, importing a file). That's not 
possible for GitHub Actions and the main reason it's not an 
option for LDC CI.
Dec 11 2020
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 11 December 2020 at 13:18:52 UTC, kinke wrote:

 On Friday, 11 December 2020 at 10:34:34 UTC, Petar Kirov 
 [ZombineDev] wrote:

 GitHub Actions has the least mature UI (compared to every 
 other CI service), but for most open-source D projects hosted 
 on GitHub it is easily the best choice, thanks to 
 https://github.com/dlang-community/setup-dlang which makes it 
 trivial to add CI for Windows, Linux and macOS to your 
 project. Probably the biggest disadvantage is that they don't 
 offer the option to restart individual jobs, which is partial 
 offset by offering noticeably faster machines than Travis CI 
 did.


 I tend to agree. But if your CI scripts are complex, there's 
 one big catch for the MS solutions (GitHub Actions / Azure 
 Pipelines) - they don't use a regular YAML parser and don't 
 support YAML anchors for reusable pieces. Azure Pipelines 
 offers custom 'templates' instead (essentially, importing a 
 file). That's not possible for GitHub Actions and the main 
 reason it's not an option for LDC CI.


Wow, apparently I haven't used Azure Pipelines / GitHub Actions 
in sufficient depth, as we use YAML anchors in almost every 
GitLab CI pipeline at work :D

I guess this big disadvantage is offset GitHub Actions 
Marketplace, which makes reusable CI components much more 
scalable to use. Though it's still annoying as I wouldn't bother 
creating a separate repo for something that would otherwise be a 
YAML anchor, unless we use it more than a few projects.
Dec 11 2020
parent reply Jacob Carlborg <doob me.com> writes:
On 2020-12-11 14:48, Petar Kirov [ZombineDev] wrote:


 Wow, apparently I haven't used Azure Pipelines / GitHub Actions in 
 sufficient depth, as we use YAML anchors in almost every GitLab CI 
 pipeline at work :D


Really? `extends` [1] is the preferred way reuse "code" in GitLab CI ;)


 I guess this big disadvantage is offset GitHub Actions Marketplace, 
 which makes reusable CI components much more scalable to use. Though 
 it's still annoying as I wouldn't bother creating a separate repo for 
 something that would otherwise be a YAML anchor, unless we use it more 
 than a few projects.



[1] https://docs.gitlab.com/ee/ci/yaml/#extends

-- 
/Jacob Carlborg
Dec 11 2020
parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 11 December 2020 at 19:43:43 UTC, Jacob Carlborg wrote:

 On 2020-12-11 14:48, Petar Kirov [ZombineDev] wrote:


 Wow, apparently I haven't used Azure Pipelines / GitHub 
 Actions in sufficient depth, as we use YAML anchors in almost 
 every GitLab CI pipeline at work :D


 Really? `extends` [1] is the preferred way reuse "code" in 
 GitLab CI ;)
 
 [1] https://docs.gitlab.com/ee/ci/yaml/#extends


I remember reading the docs a while ago and not seeing a 
compelling reason for using this feature. I re-read the page 
again and I still don't think we're going to use it, as in all of 
our use cases the yaml snippet reuse is limited to a single file. 
`includes` are much more useful though ;)
Dec 11 2020
prev sibling parent Mathias LANG <geod24 gmail.com> writes:
On Friday, 11 December 2020 at 10:34:34 UTC, Petar Kirov 
[ZombineDev] wrote:

 GitHub Actions has the least mature UI (compared to every other 
 CI service), but for most open-source D projects hosted on 
 GitHub it is easily the best choice, thanks to 
 https://github.com/dlang-community/setup-dlang which makes it 
 trivial to add CI for Windows, Linux and macOS to your project. 
 Probably the biggest disadvantage is that they don't offer the 
 option to restart individual jobs, which is partial offset by 
 offering noticeably faster machines than Travis CI did.

 [...]


Funnily, the UI just got a big update thanks to Github universe.
Another thing that irks me about Github is how unstable their 
environment is. From experience: 
https://github.com/actions/virtual-environments/issues/1498
If you look at their announcements 
(https://github.com/actions/virtual-environments/issues?q=is%3Aissue+is%3Aopen+l
bel%3AAnnouncement) you'll see that they move fast. It might be a good thing
for some use cases, but I definitely don't want to have to catchup to updates
to my Docker images when there's work to be done.

Hopefully they'll see the light there too, at some point in the 
near future.
Dec 11 2020
prev sibling next sibling parent Guillaume Piolat <first.name guess.com> writes:
On Thursday, 10 December 2020 at 11:42:41 UTC, Mathias LANG wrote:

 If you're thinking about using Github Actions, it's as simple 
 as using `- uses: dlang-community/setup-dlang v1`. That will 
 gives you `dmd-latest` to get started. If you want a more 
 advanced usage, DMD's workflow file contains a lot of comment 
 which might help you with some patterns, for example how to add 
 an extra row to a matrix (spoiler: you can't, you can only 
 remove rows): 
 https://github.com/dlang/dmd/blob/master/.github/workflows/runnable_cxx.yml


Hello,

Thanks for doing this.
If anything I can recommend against Azure CI, which is much 
harder to setup than Travis CI for D, the UI of Azure is 
confusing, also there is also a token limit which is easy.

Goong to switch to Github Actions whenever possible.
Dec 10 2020
prev sibling next sibling parent Gregor =?UTF-8?B?TcO8Y2ts?= <gregormueckl gmx.de> writes:
On Thursday, 10 December 2020 at 11:42:41 UTC, Mathias LANG wrote:

 As many of you might be aware already, travis-ci is killing 
 their open-source plans.
 They haven't sunset it yet,, but they introduced a set of 
 restrictions that makes them completely uncompetitive and 
 impractical to use compared to other services on the market.

 [...]


Just want to put out there that I have started to use AppVeyor to 
create Windows CI builds for some OSS projects that host on my 
private instance of Heptapod (Gitlab fork aupportimg Mercurial). 
The experience has been great so far. UI is nice and powerful. 
Doxumentation is good.

It is a reasonable option in my book if you want to avoid the 
git/Github monoculture.
Dec 11 2020
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2020-12-10 12:42, Mathias LANG wrote:


 If you want a more advanced usage, DMD's 
 workflow file contains a lot of comment which might help you with some 
 patterns, for example how to add an extra row to a matrix (spoiler: you 
 can't, you can only remove rows): 
 https://github.com/dlang/dmd/blob/master/.github/workflows/runnable_cxx.yml


What do you mean with: can't add extra rows? That's what `include` is 
for, which is even being used in the above link [1].

[1] 
https://github.com/dlang/dmd/blob/37b93260e697a1a31b97d046f18cf32f78ef3b0d/.github/workflows/runnable_cxx.yml#L104

-- 
/Jacob Carlborg
Dec 11 2020
parent Mathias LANG <geod24 gmail.com> writes:
On Friday, 11 December 2020 at 19:50:06 UTC, Jacob Carlborg wrote:

 On 2020-12-10 12:42, Mathias LANG wrote:


 If you want a more advanced usage, DMD's workflow file 
 contains a lot of comment which might help you with some 
 patterns, for example how to add an extra row to a matrix 
 (spoiler: you can't, you can only remove rows): 
 https://github.com/dlang/dmd/blob/master/.github/workflows/runnable_cxx.yml


 What do you mean with: can't add extra rows? That's what 
 `include` is for, which is even being used in the above link 
 [1].

 [1] 
 https://github.com/dlang/dmd/blob/37b93260e697a1a31b97d046f18cf32f78ef3b0d/.github/workflows/runnable_cxx.yml#L104


I just checked the documentation. It used to be that you could 
not add a new entry to a matrix, at least when this file was 
originally written. Looking again today, it seems to be possible, 
so I'm glad they fixed this!
Dec 11 2020