19 August 2022 — Comments
In 2019, GitHub published their own solution to run automated workflows called GitHub Actions, which allowed those hosting their code in GitHub, to be able to define and run their CI/CD pipelines in the same platform.
When it was released, one of the main pain points to use it was that defining pipelines required large yaml config files, where it was sometimes hard to avoid duplication.
However, during this time, and based on users' feedback, GitHub has introduced several improvements on this regard.
Recently, I have been refactoring and improving a pipeline in one of my projects, and I wanted to share the different approaches I used to reduce duplication.
Those approaches are:
This article assumes you have certain knowledge on how GitHub Actions works. If that's not the case, you probably want to take a look at its documentation.
Let's imagine we start with this continuous integration workflow for a PHP project:
name: Continuous integration
on:
pull_request: null
push:
branches:
- main
- develop
jobs:
coding-styles:
runs-on: ubuntu-22.04
env:
extensions: 'openswoole, gd, intl'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: '8.0'
extensions: ${{ env.extensions }}
key: coding-styles-extensions
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.0'
tools: composer
extensions: ${{ env.extensions }}
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
- name: Check coding styles
run: composer coding-styles
static-analysis:
runs-on: ubuntu-22.04
env:
extensions: 'openswoole, gd, intl'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: '8.0'
extensions: ${{ env.extensions }}
key: coding-styles-extensions
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.0'
tools: composer
extensions: ${{ env.extensions }}
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
- name: Static analysis
run: composer static-analysis
unit-tests-8-0:
runs-on: ubuntu-22.04
env:
extensions: 'openswoole, gd, intl'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: '8.0'
extensions: ${{ env.extensions }}
key: coding-styles-extensions
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.0'
tools: composer
extensions: ${{ env.extensions }}
coverage: pcov
ini-values: pcov.directory=module
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
- name: Unit tests
run: composer test:unit
unit-tests-8-1:
runs-on: ubuntu-22.04
env:
extensions: 'openswoole, gd, intl'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: '8.1'
extensions: ${{ env.extensions }}
key: coding-styles-extensions
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.1'
tools: composer
extensions: ${{ env.extensions }}
coverage: pcov
ini-values: pcov.directory=module
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
- name: Unit tests
run: composer test:unit
- name: Publish coverage
uses: actions/upload-artifact@v3
with:
name: coverage-unit
path: |
build/coverage-unit
e2e-tests-8-0:
runs-on: ubuntu-22.04
env:
extensions: 'openswoole, gd, intl'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: '8.0'
extensions: ${{ env.extensions }}
key: coding-styles-extensions
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.0'
tools: composer
extensions: ${{ env.extensions }}
coverage: pcov
ini-values: pcov.directory=module
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
- name: E2E tests
run: composer test:e2e
e2e-tests-8-1:
runs-on: ubuntu-22.04
env:
extensions: 'openswoole, gd, intl'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: '8.1'
extensions: ${{ env.extensions }}
key: coding-styles-extensions
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.1'
tools: composer
extensions: ${{ env.extensions }}
coverage: pcov
ini-values: pcov.directory=module
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
- name: E2E tests
run: composer test:e2e
- name: Publish coverage
uses: actions/upload-artifact@v3
with:
name: coverage-e2e
path: |
build/coverage-e2e
unit-mutation-tests:
runs-on: ubuntu-22.04
needs:
- unit-tests-8-1
env:
extensions: 'openswoole, gd, intl'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: '8.1'
extensions: ${{ env.extensions }}
key: coding-styles-extensions
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.1'
tools: composer
extensions: ${{ env.extensions }}
coverage: pcov
ini-values: pcov.directory=module
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
- name: Download coverage
uses: actions/download-artifact@v3
with:
name: coverage-unit
path: build
- name: Unit mutation tests
run: composer mutation-tests:unit
e2e-mutation-tests:
runs-on: ubuntu-22.04
needs:
- e2e-tests-8-1
env:
extensions: 'openswoole, gd, intl'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: '8.1'
extensions: ${{ env.extensions }}
key: coding-styles-extensions
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.1'
tools: composer
extensions: ${{ env.extensions }}
coverage: pcov
ini-values: pcov.directory=module
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
- name: Download coverage
uses: actions/download-artifact@v3
with:
name: coverage-unit
path: build
- name: E2E mutation tests
run: composer mutation-tests:e2e
In human language, this is what the pipeline does:
The first 4 jobs are all run in parallel, and the last two are run after the tests have finished (as they require the code coverage reports).
Also, for all the jobs, an environment needs to be set-up, with certain version of PHP (sometimes just 8.0, sometimes also 8.1) and some PHP extensions.
Now, let's see how to improve all those duplicated steps.
The first thing we can do is merge the jobs for all the PHP versions, and pass that as a matrix argument.
Also, coding styles and static analysis are both static code checks which require a very similar set-up. We can merge those two and pass them via matrix as well.
name: Continuous integration
on:
pull_request: null
push:
branches:
- main
- develop
jobs:
static-checks:
runs-on: ubuntu-22.04
env:
extensions: 'openswoole, gd, intl'
strategy:
matrix:
command: ['coding-styles', 'static-analysis']
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: '8.0'
extensions: ${{ env.extensions }}
key: coding-styles-extensions
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.0'
tools: composer
extensions: ${{ env.extensions }}
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
- name: Run ${{ matrix.command }} with composer
run: composer ${{ matrix.command }}
unit-tests:
runs-on: ubuntu-22.04
env:
extensions: 'openswoole, gd, intl'
strategy:
matrix:
php-version: ['8.0', '8.1']
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: ${{ matrix.php-version }}
extensions: ${{ env.extensions }}
key: coding-styles-extensions
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: ${{ matrix.php-version }}
tools: composer
extensions: ${{ env.extensions }}
coverage: pcov
ini-values: pcov.directory=module
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
- name: Unit tests
run: composer test:unit
- name: Publish coverage
if: ${{ matrix.php-version == '8.1' }}
uses: actions/upload-artifact@v3
with:
name: coverage-unit
path: |
build/coverage-unit
e2e-tests:
runs-on: ubuntu-22.04
env:
extensions: 'openswoole, gd, intl'
strategy:
matrix:
php-version: ['8.0', '8.1']
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: '8.0'
extensions: ${{ env.extensions }}
key: coding-styles-extensions
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.0'
tools: composer
extensions: ${{ env.extensions }}
coverage: pcov
ini-values: pcov.directory=module
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
- name: E2E tests
run: composer test:e2e
- name: Publish coverage
if: ${{ matrix.php-version == '8.1' }}
uses: actions/upload-artifact@v3
with:
name: coverage-e2e
path: |
build/coverage-e2e
unit-mutation-tests:
runs-on: ubuntu-22.04
needs:
- unit-tests
env:
extensions: 'openswoole, gd, intl'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: '8.1'
extensions: ${{ env.extensions }}
key: coding-styles-extensions
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.1'
tools: composer
extensions: ${{ env.extensions }}
coverage: pcov
ini-values: pcov.directory=module
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
- name: Download coverage
uses: actions/download-artifact@v3
with:
name: coverage-unit
path: build
- name: Unit mutation tests
run: composer mutation-tests:unit
e2e-mutation-tests:
runs-on: ubuntu-22.04
needs:
- e2e-tests
env:
extensions: 'openswoole, gd, intl'
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: '8.1'
extensions: ${{ env.extensions }}
key: coding-styles-extensions
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.1'
tools: composer
extensions: ${{ env.extensions }}
coverage: pcov
ini-values: pcov.directory=module
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
- name: Download coverage
uses: actions/download-artifact@v3
with:
name: coverage-unit
path: build
- name: E2E mutation tests
run: composer mutation-tests:e2e
With this change, we are down from 8 jobs to 5, with the only consideration that we now publish code coverage conditionally based on the PHP version.
The next more obvious thing is that there are a couple of steps that appear on each one of the jobs to set up the environment.
We can combine those into a local composite action that wraps all the individual steps and can be called as a whole by every job.
Local actions have to be located inside .github/actions, in a folder with the name we want, containing an action.yml file, which in our case, could look like this:
.github/actions/ci-setup/action.yml:
name: CI setup
description: 'Sets up the environment for jobs during CI workflow'
inputs:
php-version:
description: 'The PHP version to be setup'
required: true
php-extensions:
description: 'The PHP extensions to install'
required: false
default: ''
extensions-cache-key:
description: 'The key used to cache PHP extensions'
required: true
runs:
using: composite
steps:
- name: Setup cache environment
id: extcache
uses: shivammathur/cache-extensions@v1
with:
php-version: ${{ inputs.php-version }}
extensions: ${{ inputs.php-extensions }}
key: ${{ inputs.extensions-cache-key }}
- name: Cache extensions
uses: actions/cache@v2
with:
path: ${{ steps.extcache.outputs.dir }}
key: ${{ steps.extcache.outputs.key }}
restore-keys: ${{ steps.extcache.outputs.key }}
- name: Use PHP
uses: shivammathur/setup-php@v2
with:
php-version: ${{ inputs.php-version }}
tools: composer
extensions: ${{ inputs.php-extensions }}
coverage: pcov
ini-values: pcov.directory=module
- name: Install dependencies
run: composer install --no-interaction --prefer-dist
shell: bash
This action wraps the 4 steps that we have on every job. The only step we can't add here is the checkout step, as we need the code to have been checked out first in order to find the action file itself.
With this local composite action, we can refactor the workflow to look like this:
name: Continuous integration
on:
pull_request: null
push:
branches:
- main
- develop
jobs:
static-checks:
runs-on: ubuntu-22.04
strategy:
matrix:
command: ['coding-styles', 'static-analysis']
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup
uses: './.github/actions/ci-setup'
with:
php-version: '8.1'
php-extensions: 'openswoole, gd, intl'
extensions-cache-key: e2e-tests-${{ matrix.command }}
- name: Run ${{ matrix.command }} with composer
run: composer ${{ matrix.command }}
unit-tests:
runs-on: ubuntu-22.04
strategy:
matrix:
php-version: ['8.0', '8.1']
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup
uses: './.github/actions/ci-setup'
with:
php-version: ${{ matrix.php-version }}
php-extensions: 'openswoole, gd, intl'
extensions-cache-key: unit-tests-${{ matrix.php-version }}
- name: Unit tests
run: composer test:unit
- name: Publish coverage
if: ${{ matrix.php-version == '8.1' }}
uses: actions/upload-artifact@v3
with:
name: coverage-unit
path: |
build/coverage-unit
e2e-tests:
runs-on: ubuntu-22.04
strategy:
matrix:
php-version: ['8.0', '8.1']
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup
uses: './.github/actions/ci-setup'
with:
php-version: ${{ matrix.php-version }}
php-extensions: 'openswoole, gd, intl'
extensions-cache-key: e2e-tests-${{ matrix.php-version }}
- name: E2E tests
run: composer test:e2e
- name: Publish coverage
if: ${{ matrix.php-version == '8.1' }}
uses: actions/upload-artifact@v3
with:
name: coverage-e2e
path: |
build/coverage-e2e
unit-mutation-tests:
runs-on: ubuntu-22.04
needs:
- unit-tests
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup
uses: './.github/actions/ci-setup'
with:
php-version: '8.1'
php-extensions: 'openswoole, gd, intl'
extensions-cache-key: unit-mutation-tests
- name: Download coverage
uses: actions/download-artifact@v3
with:
name: coverage-unit
path: build
- name: Unit mutation tests
run: composer mutation-tests:unit
e2e-mutation-tests:
runs-on: ubuntu-22.04
needs:
- e2e-tests
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup
uses: './.github/actions/ci-setup'
with:
php-version: '8.1'
php-extensions: 'openswoole, gd, intl'
extensions-cache-key: e2e-mutation-tests
- name: Download coverage
uses: actions/download-artifact@v3
with:
name: coverage-unit
path: build
- name: E2E mutation tests
run: composer mutation-tests:e2e
Every job is now much shorter, with almost all duplicated code moved to the composite action.
Also, as a side effect, we got rid of defining the PHP extensions as an env var, since we now pass them as an arg to the action only in one place.
But that's not it. There's still a lot of duplication between both tests jobs and both mutation tests jobs.
One way we can reduce even further the gap is by extracting them to reusable workflows.
They are similar to composite actions, with the difference that they do not wrap only a couple of steps, but they can have even multiple jobs that we then invoke from the main workflow.
Also, our reusable workflow can still use the composite action we created in previous step.
Let's define our ci-test reusable workflow:
.github/workflows/ci-test.yml:
name: Tests
on:
workflow_call:
inputs:
test-group:
type: string
required: true
description: unit or e2e
jobs:
tests:
runs-on: ubuntu-22.04
strategy:
matrix:
php-version: ['8.0', '8.1']
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup
uses: './.github/actions/ci-setup'
with:
php-version: ${{ matrix.php-version }}
php-extensions: 'openswoole, gd, intl'
extensions-cache-key: ${{ inputs.test-group }}-tests-${{ matrix.php-version }}
- name: Tests
run: composer test:${{ inputs.test-group }}
- name: Publish coverage
if: ${{ matrix.php-version == '8.1' }}
uses: actions/upload-artifact@v3
with:
name: coverage-${{ inputs.test-group }}
path: |
build/coverage-${{ inputs.test-group }}
mutation-tests:
runs-on: ubuntu-22.04
needs:
- tests
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup
uses: './.github/actions/ci-setup'
with:
php-version: '8.1'
php-extensions: 'openswoole, gd, intl'
extensions-cache-key: ${{ inputs.test-group }}-mutation-tests
- name: Download coverage
uses: actions/download-artifact@v3
with:
name: coverage-${{ inputs.test-group }}
path: build
- name: ${{ inputs.test-group }} mutation tests
run: composer mutation-tests:${{ inputs.test-group }}
And now, we can invoke this reusable workflow from our main CI workflow like this:
name: Continuous integration
on:
pull_request: null
push:
branches:
- main
- develop
jobs:
static-checks:
runs-on: ubuntu-22.04
strategy:
matrix:
command: ['coding-styles', 'static-analysis']
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup
uses: './.github/actions/ci-setup'
with:
php-version: '8.1'
php-extensions: 'openswoole, gd, intl'
extensions-cache-key: e2e-tests-${{ matrix.command }}
- name: Run ${{ matrix.command }} with composer
run: composer ${{ matrix.command }}
unit-tests:
uses: './.github/workflows/ci-test.yml'
with:
test-group: unit
e2e-tests:
uses: './.github/workflows/ci-test.yml'
with:
test-group: e2e
This reduces the duplication to the bare minimum, allowing us to reuse the tests + mutation-tests logic both for unit tests and E2E tests, keeping the benefit of making the later depend on the former.
You can see how the final result looks like in this example repository.
There are a couple of things to clarify from the examples above:
In here, we use local composite actions and reusable workflows.
However, GitHub Actions supports loading them from a different repository, and therefore using them in multiple projects if needed.
In the case of actions, it of course also allows to publish them in the marketplace so that you don't have to reference them via repository name and path.
It may seem as if unit and mutation tests could have been simplified with a matrix, as we did with the static checks.
However, that would not allow to make every mutation-tests job to depend only on its corresponding tests job, and they would have to wait for all the tests to finish.
That's why a reusable workflow is a better solution here.
It may also look like the unit-tests and e2e-tests jobs, which in the last version only invoke the ci-tests reusable workflow could have been merged using a matrix.
However, GitHub Actions does not currently allow to use a matrix with reusable workflows.
That's why they are defined as two independent jobs.
The example used in this article is made-up, but tries to cover a bit of everything to justify all the strategies presented on it.
If you want to see a real example, this has been based (on a very simplified way) on the continuous integration workflow from Shlink.