buildkite-agent pipeline

This page references the out-of-date Buildkite Agent v2.

For docs referencing the Buildkite Agent v3, see the latest version of this document.

The Buildkite Agent's pipeline command allows you to add and replace build steps in the running build. The steps are defined using YAML or JSON and can be read from a file or streamed from the output of a script.

See the Defining your pipeline steps guide for a step-by-step example and list of step types.

Uploading pipelines

Usage:

   buildkite-agent pipeline upload <file> [arguments...]

Description:

   Allows you to change the pipeline of a running build by uploading either a
   JSON or YAML configuration file. If no configuration file is provided,
   we look for the file in the following locations:

   - .buildkite/pipeline.yml
   - .buildkite/pipeline.json

   You can also pipe build pipelines to the command, allowing you to create scripts
   that generate dynamic pipelines.

Example:

   $ buildkite-agent pipeline upload
   $ buildkite-agent pipeline upload my-custom-steps.json
   $ ./script/dynamic_step_generator | buildkite-agent pipeline upload

Options:

   --replace                                    Replace the rest of the existing pipeline with the steps uploaded. Jobs that are already running are not removed. [$BUILDKITE_PIPELINE_REPLACE]
   --job                                        The job that is making the changes to its build [$BUILDKITE_JOB_ID]
   --agent-access-token                         The access token used to identify the agent [$BUILDKITE_AGENT_ACCESS_TOKEN]
   --endpoint "https://agent.buildkite.com/v3"  The Agent API endpoint [$BUILDKITE_AGENT_ENDPOINT]
   --no-color                                   Don't show colors in logging [$BUILDKITE_AGENT_NO_COLOR]
   --debug                                      Enable debug mode [$BUILDKITE_AGENT_DEBUG]
   --debug-http                                 Enable HTTP debug mode, which dumps all request and response bodies to the log [$BUILDKITE_AGENT_DEBUG_HTTP]

Pipeline format

The pipeline can be written as YAML or JSON, but YAML is more common for its readability. There are two top level properties you can specify:

Environment variable substitution

In Buildkite Agent versions 3.0 and above, the pipeline upload command supports environment variable substitution using the syntax $VAR and ${VAR}.

If you are unable to upgrade your agent to version 3.0 or above, it is possible (but not recommended) to emulate the agent's environment variable substitution using bash:

eval "echo \"$(cat pipeline.yml)\"" | tee /dev/stderr | buildkite-agent pipeline upload

Omit the tee command if you have no need to see the resulting YAML.

Escaping the $ character

If you need to prevent substitution, you can escape the $ character by using $$ or \$.

For example, using $$USD and \$USD will both result in the same value: $USD.

Requiring environment variables

You can set required environment variables using the syntax ${VAR?}. If VAR is not set, the pipeline upload command will print and error and exit with a status of 1.

For example, the following step will cause the pipeline upload to error if the SERVER environment variable has not been set:

- command: "deploy.sh \"${SERVER?}\""

You can set a custom error message after the ? character. For example, the following prints the error message SERVER: is not set. Please specify a server if the environment variable has not been set:

- command: "deploy.sh \"${SERVER?is not set. Please specify a server}\""

Default, blank and missing values

If an environment variable has not been set it will evaluate to a blank string. You can set a fallback value using the syntax ${VAR:-default-value}.

For example, the following step will run the command deploy.sh staging:

- command: "deploy.sh \"${SERVER:-staging}\""
Environment Variables Syntax Result
"${SERVER:-staging}" "staging"
SERVER="" "${SERVER:-staging}" "staging"
SERVER="staging-5" "${SERVER:-staging}" "staging-5"

If you need to substitute environment variables containing empty strings, you can use the syntax ${VAR-default-value} (notice the missing :).

Environment Variables Syntax Result
"${SERVER-staging}" "staging"
SERVER="" "${SERVER-staging}" ""
SERVER="staging-5" "${SERVER-staging}" "staging-5"

Extracting character ranges

You can substitute a subset of characters from an environment variable by specifying a start and end range using the syntax ${VAR:start:end}.

For example, the following step will echo the first 7 characters of the BUILDKITE_COMMIT environment variable:

- command: "echo \"Short commit is: ${BUILDKITE_COMMIT:0:7}\""

If the environment variable has not been set, the range will return a blank string.