www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - [Semi-OT] Cross-Platform GitHub Action

reply Jacob Carlborg <doob me.com> writes:


I would like to announce the first version of a project I've been 
working on for a while. It's not anything D specific or 
implemented in D, but it can be used with D projects. This 
project provides a GitHub action for running GitHub Action 
workflows on multiple platforms. This includes platforms that 
GitHub Actions don't natively support. It currently supports 
FreeBSD and OpenBSD.

https://github.com/cross-platform-actions/action



Some of the features that are supported include:

* Multiple operating system with one single action
* Multiple versions of each operating system
* Allows to use default shell or Bash shell
* Low boot overhead
* Fast execution

Compared to 
[vmactions/freebsd-vm](https://github.com/vmactions/freebsd-vm), 
the boot time is around a fifth and the full execution time for 
the same job is around half of freebsd-vm.



Here's a sample workflow file which will setup a matrix resulting 
in two jobs.
One which will run on FreeBSD 12.2 and one which runs on OpenBSD 
6.8.

```yaml
name: CI

on: [push]

jobs:
   test:
     runs-on: macos-10.15
     strategy:
       matrix:
         os:
           - name: freebsd
             version: 12.2
           - name: openbsd
             version: 6.8

     steps:
       - uses: actions/checkout v2

       - name: Test on ${{ matrix.os.name }}
         uses: cross-platform-actions/action v0.0.1
         env:
           MY_ENV1: MY_VALUE1
           MY_ENV2: MY_VALUE2
         with:
           environment_variables: MY_ENV1 MY_ENV2
           operating_system: ${{ matrix.os.name }}
           version: ${{ matrix.os.version }}
           shell: bash
           run: |
             uname -a
             echo $SHELL
             pwd
             ls -lah
             whoami
             env | sort
```

I've been using this action for one of my own projects 
([DLP](https://github.com/jacob-carlborg/dlp/runs/2759807903)) 
for  now close to a week and it works fine. It's mostly FreeBSD 
that has been tested.

If you're interested in how the sausage is made, read on. Also 
see the readmes of the builder repositories:

https://github.com/cross-platform-actions/freebsd-builder
https://github.com/cross-platform-actions/openbsd-builder



GitHub Actions currently only support the following platforms: 
macOS, Linux and
Windows. To be able to run other platforms, this GitHub action 
runs the commands
inside a virtual machine (VM). macOS is used as the host platform 
because it
supports nested virtualization.

The VMs run on the [xhyve][xhyve] hypervisor, which is built on 
top of Apple's
[Hypervisor][hypervisor_framework] framework. The Hypervisor 
framework allows
to implement hypervisors with support for hardware acceleration 
without the
need for kernel extensions. xhyve is a lightweight hypervisor 
that boots the
guest operating systems quickly and requires no dependencies 
outside of what's
provided by the system.

The VM images running inside the hypervisor are built using 
[Packer][packer].
It's a tool for automatically creating VM images, installing the 
guest
operating system and doing any final provisioning.

The GitHub action uses SSH to communicate and execute commands 
inside the VM.
It uses [rsync][rsync] to share files between the guest VM and 
the host. xhyve
does not have any native support for sharing files. To 
authenticate the SSH
connection a unique key pair is used. This pair is generated each 
time the
action is run. The public key is added to the VM image and the 
private key is
stored on the host. Since xhyve does not support file sharing, a 
secondar hard
drive, which is backed by a file, is created. The public key is 
stored on this
hard drive, which is then mounted by the VM. At boot time, the 
secondary hard
drive will be identified and the public key will be copied to the 
appropriate
location.

To reduce the time it takes for the GitHub action to start 
executing the
commands specified by the user, it aims to boot the guest 
operating systems as
fast as possible. This is achieved in a couple of ways:

* By downloading [resources][resources], like xhyve and a few 
other tools,
     instead of installing them through a package manager

* No compression is used for the resources that are downloaded. 
The size is
     small enough anyway and it's faster to download the 
uncompressed data than
     it is to download compressed data and then uncompress it.

* It leverages `async`/`await` to perform tasks asynchronously. 
Like
     downloading the VM image and other resources at the same time

* It performs as much as possible of the setup ahead of time when 
the VM image
     is provisioned

[xhyve]: https://github.com/machyve/xhyve
[hypervisor_framework]: 
https://developer.apple.com/library/mac/documentation/DriversKernelHardware/Reference/Hypervisor/index.html
[rsync]: https://en.wikipedia.org/wiki/Rsync
[resources]: https://github.com/cross-platform-actions/resources
[packer]: https://www.packer.io
[openbsd_builder]: 
https://github.com/cross-platform-actions/openbsd-builder
[freebsd_builder]: 
https://github.com/cross-platform-actions/freebsd-builder
Jun 08 2021
next sibling parent reply kinke <noone nowhere.com> writes:
Thx for sharing! Interesting; I've recently worked on something 
similar, but on Linux hosts and using a kvm/qemu/libvirt stack 
for running CI jobs in Windows VMs.
Jun 08 2021
parent reply Jacob Carlborg <doob me.com> writes:
On Tuesday, 8 June 2021 at 19:40:01 UTC, kinke wrote:

 Thx for sharing! Interesting; I've recently worked on something 
 similar, but on Linux hosts and using a kvm/qemu/libvirt stack 
 for running CI jobs in Windows VMs.


Yeah, this is running on macOS instead because the Linux and the 
Windows runners on GitHub actions don't support nested 
virtualization. The Hypervisor framework is something similar to 
KVM. The VM images are actually created using QEMU (on Linux 
hosts), because Packer doesn't have any support for Xhyve. Packer 
will create a qcow2 VM image. At run time, the qcow2 image will 
be converted to the "raw" format, which is the only format that 
Xhyve supports. qcow2 is used up until runtime because it 
natively supports compression.

I do want to support other operating systems going forward, but 
unfortunately, it's only FreeBSD and OpenBSD that work in Xhyve. 
For other operating systems I will have to use QEMU. QEMU does 
support the Hypervisor framework as an accelerator, but I don't 
think it will be as fast as Xhyve.

When QEMU is supported, it will hopefully be trivial to add 
support for non-native architectures. I've already built the 
OpenBSD image for ARM64.

--
/Jacob Carlborg
Jun 08 2021
parent reply evilrat <evilrat666 gmail.com> writes:
On Wednesday, 9 June 2021 at 05:20:14 UTC, Jacob Carlborg wrote:

 On Tuesday, 8 June 2021 at 19:40:01 UTC, kinke wrote:

 Thx for sharing! Interesting; I've recently worked on 
 something similar, but on Linux hosts and using a 
 kvm/qemu/libvirt stack for running CI jobs in Windows VMs.


 Yeah, this is running on macOS instead because the Linux and 
 the Windows runners on GitHub actions don't support nested 
 virtualization.


Just a note from terms of service:
you get 2000 minutes available for Github Actions every month for 
free, however for using Windows hosts it takes 2x minutes and Mac 
hosts takes 5x minutes.
Jun 09 2021
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/9/21 4:17 AM, evilrat wrote:

 On Wednesday, 9 June 2021 at 05:20:14 UTC, Jacob Carlborg wrote:

 On Tuesday, 8 June 2021 at 19:40:01 UTC, kinke wrote:

 Thx for sharing! Interesting; I've recently worked on something 
 similar, but on Linux hosts and using a kvm/qemu/libvirt stack for 
 running CI jobs in Windows VMs.


 Yeah, this is running on macOS instead because the Linux and the 
 Windows runners on GitHub actions don't support nested virtualization.


 
 Just a note from terms of service:
 you get 2000 minutes available for Github Actions every month for free, 
 however for using Windows hosts it takes 2x minutes and Mac hosts takes 
 5x minutes.


I think this only applies to private repositories:

```
*GitHub Actions usage is free for both public repositories and 
self-hosted runners.* For private repositories, each GitHub account 
receives a certain amount of free minutes and storage, depending on the 
product used with the account.
```

(emphasis mine)

I do not see any balance of minutes spent on github actions on 
mysql-native, and I'm pretty sure I would have exhausted 2000 minutes 
already.

-Steve
Jun 09 2021
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/9/21 6:49 AM, Steven Schveighoffer wrote:

 On 6/9/21 4:17 AM, evilrat wrote:

 Just a note from terms of service:
 you get 2000 minutes available for Github Actions every month for 
 free, however for using Windows hosts it takes 2x minutes and Mac 
 hosts takes 5x minutes.


 
 I think this only applies to private repositories:


Confirmed: 
https://github.community/t/for-public-repositories-is-there-a-monthly-limit-on-minutes/129017

-Steve
Jun 09 2021
parent evilrat <evilrat666 gmail.com> writes:
On Wednesday, 9 June 2021 at 14:05:33 UTC, Steven Schveighoffer 
wrote:

 On 6/9/21 6:49 AM, Steven Schveighoffer wrote:

 On 6/9/21 4:17 AM, evilrat wrote:

 Just a note from terms of service:
 you get 2000 minutes available for Github Actions every month 
 for free, however for using Windows hosts it takes 2x minutes 
 and Mac hosts takes 5x minutes.


 
 I think this only applies to private repositories:


 Confirmed: 
 https://github.community/t/for-public-repositories-is-there-a-monthly-limit-on-minutes/129017

 -Steve


Ok then, good to know. Must be overly paranoid ToS checking 
kicked in :(
Jun 09 2021
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/8/21 3:10 PM, Jacob Carlborg wrote:


 
 I would like to announce the first version of a project I've been 
 working on for a while. It's not anything D specific or implemented in 
 D, but it can be used with D projects. This project provides a GitHub 
 action for running GitHub Action workflows on multiple platforms. This 
 includes platforms that GitHub Actions don't natively support. It 
 currently supports FreeBSD and OpenBSD.
 
 https://github.com/cross-platform-actions/action
 


Very cool!

I might have a need for it. When I moved mysql-native to github actions, 
I could no longer run mysql integration tests on MacOS or Windows, since 
there is no docker support for a mysql instance on those platforms. I 
can probably install mysql manually at some point, but I haven't looked 
into it.

At least for MacOS, this sounds like a way I can run a mysql instance 
that the MacOS host can talk to.

At some point, I will give it a try!

-Steve
Jun 08 2021
parent Jacob Carlborg <doob me.com> writes:
On Tuesday, 8 June 2021 at 20:39:45 UTC, Steven Schveighoffer 
wrote:


 I might have a need for it. When I moved mysql-native to github 
 actions, I could no longer run mysql integration tests on MacOS 
 or Windows, since there is no docker support for a mysql 
 instance on those platforms. I can probably install mysql 
 manually at some point, but I haven't looked into it.

 At least for MacOS, this sounds like a way I can run a mysql 
 instance that the MacOS host can talk to.


I don't think that would work. The VM is only running during one 
step. When the step is done, the VM is terminated. Also, Docker 
doesn't support FreeBSD or OpenBSD. I don't plan to add platforms 
which GitHub Actions natively support.

On the other hand, it seems pretty straightforward to install 
MySQL natively on macOS:

```
brew install mysql
brew services start mysql
```

There are also several GitHub Actions that will setup MySQL:

https://github.com/marketplace?type=actions&query=mysql

--
/Jacob Carlborg
Jun 09 2021
prev sibling parent Brian <bcallah openbsd.org> writes:
On Tuesday, 8 June 2021 at 19:10:41 UTC, Jacob Carlborg wrote:



 I would like to announce the first version of a project I've 
 been working on for a while. It's not anything D specific or 
 implemented in D, but it can be used with D projects. This 
 project provides a GitHub action for running GitHub Action 
 workflows on multiple platforms. This includes platforms that 
 GitHub Actions don't natively support. It currently supports 
 FreeBSD and OpenBSD.

 [...]


Neat.

~Brian
Jun 09 2021