CI Test unusual projects with GitHub actions and self-hosted runners

I contribute to Armbian, a Linux distribution for 120+ bespoke ARM-based Single Board Computers (SBCs) such as the Odroid XU4, Libre Computer Le Potato and the Orange Pi 3.

Early this year, we introduced rudimentary CI testing with some basic integration between Jenkins and GitHub. Recently we chose to switch our Jenkins CI to GitHub Actions.

The Scope of Testing is a Challenge

Unlike x86 architecture, there are fewer standards for booting and system configuration SBCs. The Armbian Build System has to manage different configurations for dozens of System on Chip (SoC) architectures, sub-architectures, as well as config for the SBCs themselves.

We need to be able to build and test against those different configurations as contributions are submitted.

Why Switch to GitHub Actions

We were running Jenkins off of my home internet connection. Occasionally it would go off-line, crash, or I would need to do other system maintenance. Project maintainers are all over the world, so we needed to improve our uptime.

The integration plugins for GitHub were okay, but not intuitive. Given that GitHub actions are native to GitHub, we hoped integration would be simpler.

Runner configuration for Jenkins quite antiquated. The defacto method is still SSH-based, where the controller connects to client. This makes it difficult to distribute runners as SSH access is required. A beta websockets version of the Jenkins client has been recently introduced, but we didn't test it.

Add Custom Self-Hosted runner to Project

Why Self-Hosted?

The CI testing needs for Armbian are a bit unique. Since it compiles linux kernels and cross-compiles Debian packages, we have a need for large compute resources. The more cores the better.

The project doesn't have much sponsorship, so 40 core VMs in the cloud would not be a cost effective solution. Instead we leverage on-prem compute resources of the project maintainers' personal workstations and servers.

Caching is critical as the toolchains alone are 14G in size. When building full system images, we need to cache filesystems, kernel packages and more. This makes ephemeral instances impractical without complex volume management to keep tools and caches current and available.

The image build process also requires privileges system access. Filesystems are created by mounting loopback images in a chroot environment.

A test on a clean uncached 4-core EC2 runner takes over an hour to run, whereas a test on a properly cached 40-core runner takes less than 10 minutes.

Configuring the Runner

The runners consist of an Ubuntu VM with a few basic dependencies and the Github Runner installed.

I found a very helpful role on ansible galaxy that handles the runner installation, queries the Github API for access and attaches to the git repo.

I made an ansible playbook to install docker, create a privileged user account for the GitHub runner, and apply the GitHub actions runner ansible role.

## prereq: ansible-galaxy install monolithprojects.github_actions_runner
- hosts: github_runners
  gather_facts: no

    - github_account: example
    - github_repo: example
    - access_token: "{{ vaulted_example }}"
    - runner_user: github
    - name: install python
      raw: "apt install python3 -y"
      become: yes

    - gather_facts:

    - name: install git >= 1.18
       repo: ppa:git-core/ppa

    - name: install packages
          - git
        update_cache: yes
        state: latest
    - name: add privileged user account for github runner
        name: "{{ runner_user }}"
        shell: /bin/bash
        groups: sudo, docker
        append: yes

    - name: configure passwordless sudo
        dest: /etc/sudoers
        line: "%sudo  ALL=(ALL) NOPASSWD: ALL"
        regex: "^%sudo"
      validate: visudo -cf %s

    - name: github actions runner role
        name: monolithprojects.github_actions_runner

The new runner joined the repo after playbook execution.
ansible-playbook github-runner.yml --become

Implement CI-Testing

Fortunately our Jenkins CI-testing was primarily a library of bash functions. This made migrating fairly simple.

We created an action workflow consisting of 2 jobs, shellcheck and and compiling a changed kernel.

Notice the shellcheck job does not have a self-hosted label on its runs-on parameter. Since shellcheck is just a linter, we were able to use runners provided by Github.

There is a shellcheck GitHub action, but it was difficult to limit testing to the file globs we wanted to target. Manually running shellcheck proved to be simpler and more flexible.

The second job uses our CI library to test building an SBC kernel based on the files changes in the commit. This job runs on our self-hosted runners.

name: test pull request
# This workflow is triggered on pushes to the repository.
on: [pull_request]

    name: shellcheck
    runs-on: ubuntu-latest
       - name: Checkout repository
         uses: actions/checkout@v2
       - name: Environment variables
         run: sudo -E bash -c set
       - name: Install required packages
         run: sudo apt-get install shellcheck
       - name: "lint libraries via shellcheck"
         run: shellcheck --color -x -s bash lib/*.sh && echo "shellcheck lib/*.sh - OK";
    name: Compile changed kernel
    # This job runs on self hosted Linux machine, with public label
    runs-on: [self-hosted, public]
      - name: fix sudo ownership
        run: |
          stat build && sudo chown -R $(whoami) build || echo "fresh workspace detected"
      - uses: actions/checkout@v2
          fetch-depth: 0
          path: build
          clean: false
      - uses: actions/checkout@v2
          repository: armbian/ci-testing-tools
          path: ci-testing-tools
      - name: Find SBC target and build kernel
        shell: bash {0}         
        run: |  
         cd build
         GIT_COMMIT=$(echo $GITHUB_SHA)
         GIT_PREVIOUS_COMMIT=$(git rev-parse origin/$GITHUB_BASE_REF)
         cd ..
         source ci-testing-tools/
         mkdir -p build/userpatches
         cp -f ci-testing-tools/config-jenkins-kernel.conf build/userpatches/
         cd build
         git checkout ${GITHUB_SHA}
         git branch -v
         export GPG_TTY=$(tty)
         build_kernel jenkins-kernel

Determining Test Parameters in a Monolithic Codebase

Since configurations for all SBCs and SoC families reside in the same repo, we approach testing as a monorepo. The test target is dynamically determined based on what files have changed.

I adapted the code from slimm609's monorepo-gitwatcher into shell functions that identify files changed that correspond to SBCs and SoCs. The list of changed files is captured into vars for later processing.

detect_git_changes() {
  local watch_files=${1}
  IFS=$'\r\n' GLOBIGNORE='*' command eval 'IGNORE_FILES=($(cat $watch_files))'
  folders=$(git diff --name-only ${GIT_COMMIT} ${GIT_PREVIOUS_COMMIT} | sort -u | uniq)

  for component in ${changed_components}; do
    for file in ${IGNORE_FILES[@]}; do
      if echo ${component} | grep -q ${file}; then
        echo "${component} has changed"


get_files_changed() {
  ## these var values needed by detect_git_changes
  family_changed="$(detect_git_changes ../"
  board_changed="$(detect_git_changes ../"

The list of changed files is scored against a dynamically generated list of build targets. If there are no matches, a default value is used.

get_build_target() {
  for row in ${BOARD_TABLE[@]}; do
    _info "board row ${row[@]}" 
    family=$(echo $row|cut -d',' -f1)
    board=$(echo $row|cut -d',' -f4)
    for family_row in ${family_changed}; do
      current_score=$(echo $family_row | fgrep -o -e ${family} -e $(translate_family ${family})|wc -c)
      _info "score: ${current_score} | family_row: ${family_row}"
      if [[ $current_score -gt $board_score ]]; then
         _info "ARMBIAN_BOARD=${board}"
         for branch in current dev legacy; do
           if echo $family_row |fgrep -q $branch; then
              _info "ARMBIAN_BRANCH=${branch}"
    for board_row in ${board_changed}; do  
      current_score=$(echo $board_row | fgrep -o ${board}|wc -c)
      _info "score: ${current_score} | board_row: ${board_row}"
      if [[ $current_score -gt $board_score ]]; then
        _info "ARMBIAN_BOARD=${board}"
  _info "${ARMBIAN_BOARD} ${ARMBIAN_BRANCH} selected"

Once the target is identified, the build is executed.

build_kernel() {
  local build_config=${1}

Managing Secrets

Our next phase of CI/CD Jobs involves GPG signing packages and uploading artifacts to our mirrors via rsync-over-ssh.

GitHub's access control around secrets isn't very robust. Any user with commit access to master branch can retrieve GitHub secrets. This made storing our private keys directly in GitHub secrets unappealing.

Since we consider our runners to be untrusted hosts, having the secrets stored on the filesystem was also unappealing.

The compromise was to encrypt the secrets with ccrypt and store the encrypted files on runners selected as designated hosts for uploading and signing. The decryption keyword is stored in GitHub secrets. A little bash scripting magic lets us only store the decrypted values in memory, and never on disk--or on GitHub.

eval $(ssh-agent) causes the script to launch ssh-agent and automatically export the environment variables ssh-agent outputs during runtime. These values are needed for an ssh client to use the agent.

ssh-add <(ccat $PWD/${SSH_KEYFILE} -K ${{secrets.keyword}}) decrypts the ssh private key into memory using the GitHub secret keyword.

The ${{secret.variable}} causes GitHubb actions (not bash) to perform a substitution during run time. The secret is automatically obfuscated by GitHub actions.

The <(command) redirects the output of a command into a file descriptor so a command can read it as a file stream, despite only being in memory. Additionally ssh-agent only stores private keys in memory.

Rsync uploads the test file via ssh because of the user@host: syntax. The ssh-private key is automatically used from ssh agent.

    name: do secrety things
    runs-on: [self-hosted]
      - uses: actions/checkout@v2
          fetch-depth: 0
          path: ci-test
          clean: true
      - name: Do some tests
        shell: bash {0}
        run: |
         eval $(ssh-agent)
         cd ci-test
         ssh-add <(ccat $PWD/${SSH_KEYFILE} -K ${{secrets.keyword}})
         echo "Unique Data $(uuidgen)" > testfile
         rsync -auv testfile ${TEST_USER}@${TEST_HOST}:

Observations and Challenges

A workflow must exist before a worker can be added

The GitHub Runner ansible role was unable register to a repo that had no actions configured. First create a simple GitHub actions workflow and commit to repo. Once a basic workflow exist, a runner can be added.

Git >= 1.18 required

The GitHub Checkout Action requires the runner to have Git v1.18 or greater, otherwise it falls back to using the GitHub rest about to download a tar of a repo. This can be problematic, if your CI testing depends on reading information from an actual git repo. Our runners use Ubuntu 18.04 LTS, which comes with Git 1.17. We install a newer git from a PPA archive during runner provisioning as a workaround.

Documentation has Gaps

I was unable to find straightforward information about the filesystem topology and the workspace environment. My checkouts landed in weird paths with redundant names such as /full/path/to/workspace/repo_name/repo_name

Access Control

Access control does not seem very robust. Large organizations may want to stick with older more-proven platforms such as Jenkins.

Future Plans

It's been great to get quick feedback on Pull Requests without performing manual testing. The improved uptime with GitHub actions has also been beneficial to giving our CI testing credibility.

We look forward to enhancing our CI system, with more distributed image builds, testing running images against actual SBC hardware, and creating dashboards to track the trends of our challenges and successes.

Author image
Lane is a Site Reliability Engineer Consultant with Ippon Technologies. He is passionate about Linux, Ansible, and Hashicorp tools
Richmond, VA
Ippon Technologies is an international consulting firm that specializes in Agile Development, Big Data and DevOps / Cloud. Our 400+ highly skilled consultants are located in the US, France, Australia and Russia. Ippon technologies has a $42 million revenue.