Testing Lua projects with GitHub Actions

GitHub Actions is a free tool provided by GitHub for software automation. It enables you to run a command on a virtual machine in response to changes to a repository, like a new commit. It’s an ideal candidate for automated testing, or continuous integration, for your Lua modules or projects.

By setting up automated testing you can ensure that any contributors to your project have their changes verified against your test suite as well. All pull-request will be utilizes your testing workflow.

In this example I'm going to be using busted as my test suite runner, but you can use pretty much anything: it’s easy to run any shell command.

We'll be using the following two GitHub Actions:

• gh-actions-lua
• gh-actions-luarocks

Feeling kind? Star the two projects above to help other people find out how to run Lua in their GitHub Actions workflows

Before getting started

Before setting up GitHub Actions you'll want to make sure you have a test suite that runs successfully on your computer. Additionally, if you're developing a Lua module for LuaRocks, then you'll want to have a .rockspec file that lists all your project’s dependencies and build process.

Even if you aren’t planning on publishing your code as a module, you can still use a rockspec to declare your dependencies to make it easy to install them during testing

Sicne I'm using busted, I use the following command to run my all of my tests:

$ busted

Optional: I have a rockspec file named something like myproject-dev-1.rockspec that is checked into my repository. I recommend checking in a dev rockspec that can install your module by source from the repository. It might look something like this:

package = "myproject"
version = "dev-1"

source = {
  url = "git://github.com/leafo/myproject.git",
}

description = {
  summary = "Do something fancy",
  homepage = "https://leafo.net",
  license = "MIT"
}

dependencies = {
  "lua >= 5.1",
  "lpeg",
  "luasocket",
  "lua-cjson",
}

build = {
  type = "builtin",
  modules = {
    ["myproject"] = "myproject/init.lua"
  }
}

Configuring the Actions workflow

A workflow represents a collection of jobs you have created with GitHub Actions. A job is a list of shell commands & actions executed on a virtual machine. You can create a workflow with a specially named file in your repository.

For running a test suite on new commits we'll start by creating a new file in our repository: .github/workflows/test.yml

The name test.yml can be anything you like. Additionally you can create multiple workflows by creating multiple files in that directory

Insert the following into .github/workflows/test.yml to start out:

name: test

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@master

    - uses: leafo/gh-actions-lua@v5
      with:
        luaVersion: "5.1"

    - uses: leafo/gh-actions-luarocks@v2

    - name: setup 
      run: |
        luarocks install busted

    - name: test
      run: |
        busted -o utfTerminal

Most of the lines in this file should be relatively self explanatory, but here’s a quick summary:

Detailed documentation for the workflow syntax can be found on GitHub: Workflow syntax for GitHub Actions

1. on instructs the workflow to run for every push to the repository
2. We declare a job called test
3. steps lists everything that will happen to complete the job (each step is executed in order)
4. The actions/checkout@master action is used to checkout the code in our repository (This action is authored by GitHub)
5. The leafo/gh-actions-lua@v5 action will build and install Lua into the current environment
   • In this example we specify ask for Lua 5.1, examples with more Lua versions are shown below
6. The leafo/gh-actions-luarocks@v2 action will build and install LuaRocks, this must come after the Lua installation
7. We issue a command called setup that will install busted for use in the rest of our action
8. We issue a command called test that runs busted in the working directory (with color enabled!)

In this example we included a version number (eg. @v5) with each action, it may be necessary to update these versions number in the future. Generally you should specify a version number so prevent unexpected failures happening when an action’s source code is updated.

Head to Running the action/workflow to see how to run the workflow (hint: you just commit and push), or scroll down to see some ways to customize the workflow.

Installing dependencies with LuaRocks

The simplest way to install dependencies is to list them in the workflow file within the test job in a setup step. The gh-actions-luarocks action will automatically configure LuaRocks to install for the current version of Lua your test is running for.

Here’s how you would edit .github/workflows/test.yml to add a few more dependencies:

 - name: setup
   run: |
     luarocks install busted
+    luarocks install moonscript
+    luarocks install luaossl
     luarocks make

name: test

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@master

    - uses: leafo/gh-actions-lua@v5
      with:
        luaVersion: "5.1"

    - uses: leafo/gh-actions-luarocks@v2

    - name: setup 
      run: |
        luarocks install busted
        luarocks install moonscript
        luarocks install luaossl

    - name: test
      run: |
        busted -o utfTerminal

If you have a rockspec in your repository’s root folder you can use LuaRocks to automatically install the dependencies with the make command. (This has the side effect of also building your module)

If you have multiple rockspecs you will need to explicitly specify one like this: luarocks make myrockspec-dev-1.rockspec

In our example rockspec we have the following dependencies declared:

-- myproject-dev-1.rockspec

dependencies = {
  "lua >= 5.1",
  "lpeg",
  "luasocket",
  "lua-cjson",
}

Modified the setup step to look like this:

 - name: setup 
   run: |
+    luarocks make
     luarocks install busted

name: test

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@master

    - uses: leafo/gh-actions-lua@v5
      with:
        luaVersion: "5.1"

    - uses: leafo/gh-actions-luarocks@v2

    - name: setup 
      run: |
        luarocks make
        luarocks install busted

    - name: test
      run: |
        busted -o utfTerminal

luarocks make will use the rockspec located in your repository root directory, and install the dependencies specified. If anything fails to install then the job will fail.

busted is still installed manually because it’s not listed in the rockspec since it’s a a test dependency, and not a runtime dependency.

Testing multiple versions of Lua with a matrix

Testing code that is designed to run on multiple versions of Lua is easily accomplished by listing out the versions you want to test in a variable matrix.

Make the following change to .github/workflows/test.yml:

 jobs:
   test:
+    strategy:
+      fail-fast: false
+      matrix:
+        luaVersion: ["5.1", "5.2", "5.3", "luajit"]
+
     runs-on: ubuntu-latest
 
     steps:
     - uses: leafo/gh-actions-lua@v5
       with:
-        luaVersion: "5.1"
+        luaVersion: ${{ matrix.luaVersion }}
 
     - uses: leafo/gh-actions-luarocks@v2

name: test

on: [push]

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        luaVersion: ["5.1", "5.2", "5.3", "luajit"]

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@master

    - uses: leafo/gh-actions-lua@v5
      with:
        luaVersion: ${{ matrix.luaVersion }}

    - uses: leafo/gh-actions-luarocks@v2

    - name: build
      run: |
        luarocks install busted

    - name: test
      run: |
        busted -o utfTerminal

With this change, on push the workflow will run 4 instances of the test job for each version of Lua specified. You can see what versions are available on the gh-actions-lua documentation.

Although not necessary, the fail-fast: false option will let all versions run to completion, even if one of them fails early. You can see multiple failures and successes for every push, which can make it a lot easier to debug build issues.

Running the action/workflow

Commit .github/workflows/test.yml and push it to your repository and GitHub will automatically start running your workflow on every push. If any of the steps in your job fail, GitHub will report the commit as failed. Any merge requests will also automatically be passed through the workflow to verify changes.

If it doesn’t automatically run, check your repository Settings > Actions and enable it there.

Add a status badge to your repository

GitHub provides easy embeddable image to share the status of your build on your site or readme:

![build status](https://github.com/{USERNAME}/{REPO}/workflows/{WORKFLOW}/badge.svg)

You can generate the code for these directory by heading to your repository’s actions page and selecting the workflow you want.

Example from one of my projects:

I hope that helps you get started with GitHub Actions and Lua!