Introduction

It's very common to structure CI steps as a directed acyclic graph (DAG), and most CI platforms support this. Unfortunately, each one has a different way to configure this DAG, and dealing with these is often painful. Engage provides a way to configure a DAG of CI steps separately from CI platforms that can also be run locally.

Other pages about Engage

• Source code
• Bug reports and feature requests
• License

Tutorial

This section demonstrates common features of Engage. After understanding these sections, the reference-style documentation (e.g. command line help text and Engage file JSON Schema) should be sufficient for learning the other available features.

The basics

By default, Engage will search for a file called engage.toml in the current directory or all of its parent directories. This can also be overridden by command line options. Henceforth, the phrase "Engage file" means the file chosen by either of those strategies. Engage files are in the TOML format.

Choosing an interpreter

An Engage file must, at minimum, define the interpreter that will be used to execute the scripts. A common choice is to use the bash shell and to invoke it like this:

interpreter = ["bash", "-euo", "pipefail", "-c"]

The -c option is necessary because Engage passes the value of each script field to the interpreter as another argument at the end of the interpreter list. The other options will make the script exit immediately if an error is encountered, which is typically desirable in CI.

Adding a task

An Engage file is pointless without any tasks, so add one at the end of the file from the above section like so:

interpreter = ["bash", "-euo", "pipefail", "-c"]

[[task]]
name = "cargo"
group = "versions"
script = "cargo --version"

name, group, and script are the minimum required fields to define a task.

This task's name is cargo and it belongs to the versions group. Groups don't need to be "explicity" defined, just mentioning one in a task like this is enough to create it. Task and group names are useful for identifying which part of the pipeline is producing what output, visualizing the DAG, and selecting a subset of the DAG to run at the command line. It's idiomatic for task and group names to be nouns that don't contain characters that a shell would treat specially.

When Engage runs this task, it will run cargo --version as a command because the interpreter is bash, causing the version of Cargo to be printed.

Visualizing it

The engage dot subcommand can be used to convert an Engage file into Graphviz DOT Language, which can then be processed by other tools. For example, a graph like this can be produced:

This illustrates the order and parallelization with which Engage will execute the DAG defined by the Engage file. In this case, there is only one group and one task, so the graph is pretty simple.

Running it

The engage command, when run with no subcommand, will execute the entire DAG. While doing so, Engage will forward stdout and stderr of the tasks, alongside an indication of which task is causing the output, whether the output is coming from the task's stdout or stderr (marked with an O or E, respectively), and some extra fluff to make it look pretty. At the end, Engage will print out whether the run succeeded or failed and exit with an appropriate status code:

Note that the stdout and stderr of Engage is not considered stable.

Task dependencies

An Engage file with a handful of tasks (but only one group) might look like this:

interpreter = ["bash", "-euo", "pipefail", "-c"]

[[task]]
name = "cargo"
group = "versions"
script = "cargo --version"

[[task]]
name = "cargo-fmt"
group = "versions"
script = "cargo fmt --version"

[[task]]
name = "cargo-clippy"
group = "versions"
script = "cargo clippy --version"

[[task]]
name = "rustc"
group = "versions"
script = "rustc --version"

Which results in this graph:

In this configuration, Engage will run all four tasks at the same time. This is actually ideal for this example, since none of these tasks require side effects from any of the other tasks to happen first.

For the sake of demonstrating how Engage works though, pretend that cargo-fmt and cargo-clippy must be run after cargo, and rustc can only run after cargo-fmt and cargo-clippy have finished. This can be accomplished by making the following changes:

interpreter = ["bash", "-euo", "pipefail", "-c"]

[[task]]
name = "cargo"
group = "versions"
script = "cargo --version"

[[task]]
name = "cargo-fmt"
group = "versions"
script = "cargo fmt --version"
# New field!
depends = ["cargo"]

[[task]]
name = "cargo-clippy"
group = "versions"
script = "cargo clippy --version"
# New field!
depends = ["cargo"]

[[task]]
name = "rustc"
group = "versions"
script = "rustc --version"
# New field!
depends = ["cargo-fmt", "cargo-clippy"]

Which results in this graph:

Here, it can be seen that Engage will first run cargo, and when it's done, it will run cargo-fmt and cargo-clippy in parallel, and after those both finish, it will finally run rustc.

If a task with dependents fails, Engage will not start any of those dependent tasks. In the above Engage file, if cargo-fmt or cargo-clippy fail, then rustc is guaranteed to not run; if cargo fails, then all three other tasks are guaranteed to not run.

Group dependencies

An Engage file with a handful of groups and tasks might look like this:

interpreter = ["bash", "-euo", "pipefail", "-c"]

[[task]]
name = "cargo"
group = "versions"
script = "cargo --version"

[[task]]
name = "cargo-fmt"
group = "versions"
script = "cargo fmt --version"
depends = ["cargo"]

[[task]]
name = "cargo-clippy"
group = "versions"
script = "cargo clippy --version"
depends = ["cargo"]

[[task]]
name = "rustc"
group = "versions"
script = "rustc --version"
depends = ["cargo-fmt", "cargo-clippy"]

[[task]]
name = "wait"
group = "database"
# Pretend this is a command that exists
script = "wait-for-database-to-be-ready"

[[task]]
name = "migrate"
group = "database"
# Pretend this is a command that exists too
script = "run-database-migrations"
depends = ["wait"]

[[task]]
name = "cargo"
group = "tests"
script = "cargo test"

Which results in this graph:

This will correctly wait for the database to become ready before running the migrations, but it will also start the test suite before either of those have finished! This can be fixed by using group dependencies like so:

interpreter = ["bash", "-euo", "pipefail", "-c"]

[[task]]
name = "cargo"
group = "versions"
script = "cargo --version"

[[task]]
name = "cargo-fmt"
group = "versions"
script = "cargo fmt --version"
depends = ["cargo"]

[[task]]
name = "cargo-clippy"
group = "versions"
script = "cargo clippy --version"
depends = ["cargo"]

[[task]]
name = "rustc"
group = "versions"
script = "rustc --version"
depends = ["cargo-fmt", "cargo-clippy"]

[[task]]
name = "wait"
group = "database"
# Pretend this is a command that exists
script = "wait-for-database-to-be-ready"

[[task]]
name = "migrate"
group = "database"
# Pretend this is a command that exists too
script = "run-database-migrations"
depends = ["wait"]

# New section!
[[group]]
name = "tests"
depends = ["database"]

[[task]]
name = "cargo"
group = "tests"
script = "cargo test"

Which results in this graph:

Here, it can be seen that the database group must finish before the tests group can start.

Groups encapsulate multiple tasks into a single unit, and group dependencies allow depending upon those units as a whole. This makes it easy to reuse groups of tasks with their intended ordering by depending on the entire group. Without these features, reuse of mulitple tasks becomes difficult as one must recall which tasks belong to the "group" each time that "group" needs to be depended upon, which is error-prone.

File schema

Below is a JSON Schema document describing the format of Engage files. A copy of this document can also be obtained by running engage schema.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "File",
  "description": "An Engage file",
  "type": "object",
  "required": [
    "interpreter"
  ],
  "properties": {
    "interpreter": {
      "description": "The interpreter that will be used to run task scripts\n\nThe string given to `script` will be appended to the list given to this field, and the resulting list will be executed.",
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "task": {
      "description": "The list of tasks to run",
      "default": [],
      "type": "array",
      "items": {
        "$ref": "#/definitions/Task"
      }
    },
    "group": {
      "description": "Configuration of task groups",
      "default": [],
      "type": "array",
      "items": {
        "$ref": "#/definitions/Group"
      }
    }
  },
  "definitions": {
    "Task": {
      "description": "A task within an Engage file",
      "type": "object",
      "required": [
        "group",
        "name",
        "script"
      ],
      "properties": {
        "name": {
          "description": "Name of this task",
          "type": "string"
        },
        "group": {
          "description": "The group that this task belongs to",
          "type": "string"
        },
        "script": {
          "description": "The script to run\n\nThe string given to this field will be appended to the list given to the `interpreter` field, and the resulting list will be executed.",
          "type": "string"
        },
        "ignore": {
          "description": "Any extra status codes to treat as successful",
          "default": [],
          "type": "array",
          "items": {
            "type": "integer",
            "format": "int32"
          }
        },
        "depends": {
          "description": "List of tasks that need to complete before this one can start\n\nThe values given to this field must be a value of the `name` field of other tasks within the same group as this task.",
          "default": [],
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      }
    },
    "Group": {
      "description": "A group within an Engage file",
      "type": "object",
      "required": [
        "name"
      ],
      "properties": {
        "name": {
          "description": "Name of this group",
          "type": "string"
        },
        "depends": {
          "description": "List of groups that need to complete before this one can start\n\nThe values given to this field must be a value of the `group` field of a task or the `name` field of another group.",
          "default": [],
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      }
    }
  }
}