Manage clusters

This page provides details on how to manage clusters within your Buildkite organization.

Learn more about on how to set up queues within a cluster in Manage queues.

Setting up clusters

When a new Buildkite organization is created, a single default cluster (initially named Default cluster) is also created.

For smaller organizations, working on smaller projects, this default cluster may be sufficient. However, it's usually more convenient for organizations to manage projects in separate clusters, when these projects require different:

Staged environments, for example, development, test, staging/pre-production and production
Source code visibility, such as open-source versus closed-source code projects
Target platforms, such as Linux, Android, macOS, Windows, etc.
Multiple projects, for example, different product lines

Once your clusters are set up, you can set up one or more queues within each cluster.

Create a cluster

New clusters can be created by a cluster maintainer using the Clusters page, as well as the REST API's or GraphQL API's create a cluster feature.

Using the Buildkite interface

To create a new cluster using the Buildkite interface:

Select Agents in the global navigation to access the Clusters page.
Select Create a Cluster.
On the New Cluster page, enter the mandatory Name for the new cluster.
Enter an optional Description for the cluster. This description appears under the name of the cluster in its tile on the Clusters page.
Enter an optional Emoji and Color using the recommended syntax. This emoji appears next to the cluster's name and the color (in hex code syntax, for example, #FFE0F1) provides the background color for this emoji.
Select Create Cluster.

The new cluster's page is displayed on its Queues page, indicating the cluster's name and its default queue, named queue. From this page, you can set up one or more additional queues within this cluster.

Using the REST API

To create a new cluster using the REST API, run the following example curl command:

curl -H "Authorization: Bearer $TOKEN" \
  -X POST "https://api.buildkite.com/v2/organizations/{org.slug}/clusters" \
  -H "Content-Type: application/json" \
  -d '{ 
    "name": "Open Source",
    "description": "A place for safely running our open source builds",
    "emoji": ":technologist:",
    "color": "#FFE0F1"
  }'

where:

$TOKEN is an API access token scoped to the relevant Organization and REST API Scopes that your agent needs access to in Buildkite.

{org.slug} can be obtained:
- From the end of your Buildkite URL, after accessing Pipelines in the global navigation of your organization in Buildkite.
- By running the List organizations REST API query to obtain this value from slug in the response. For example:
```
curl -H "Authorization: Bearer $TOKEN" \
  - X GET "https://api.buildkite.com/v2/organizations"
```

name (required) is the name for the new cluster.
description (optional) is the description that appears under the name of cluster in its tile on the Clusters page.
emoji (optional) is the emoji that appears next to the cluster's name in the Buildkite interface and uses the example syntax above.
color (optional) provides the background color for this emoji and uses hex code syntax (for example, #FFE0F1).

A default queue is not automatically created

Unlike creating a new cluster through the Buildkite interface, a default queue is not automatically created using this API call. To create a new/default queue for any new cluster created through an API call, you need to manually create a new queue.

Using the GraphQL API

To create a new cluster using the GraphQL API, run the following example mutation:

mutation {
  clusterCreate(
    input: {
      organizationId: "organization-id"
      name: "Open Source"
      description: "A place for safely running our open source builds"
      emoji: ":technologist:"
      color: "#FFE0F1"
    }
  ) {
    cluster {
      id
      uuid
      name
      description
      emoji
      color
      defaultQueue {
        id
      }
      createdBy {
        id
        uuid
        name
        email
        avatar {
          url
        }
      }
    }
  }
}

where:

organizationId (required) can be obtained:
- From the GraphQL API Integration section of your Organization Settings page, accessed by selecting Settings in the global navigation of your organization in Buildkite.
- By running a getCurrentUsersOrgs GraphQL API query to obtain the organization slugs for the current user's accessible organizations, followed by a getOrgId query, to obtain the organization's id using the organization's slug. For example:
  
  Step 1. Run getCurrentUsersOrgs to obtain the organization slug values in the response for the current user's accessible organizations:
```
query getCurrentUsersOrgs {
  viewer {
    organization {
      edges {
        node {
          name
          slug
        }
      }
    }
  }
}
```
  Step 2. Run getOrgId with the appropriate slug value above to obtain this organization's id in the response:
```
query getOrgId {
  organization(slug: "organization-slug") {
    id
    uuid
    slug
  }
}
```
  Note: The organization-slug value can also be obtained from the end of your Buildkite URL, by selecting Pipelines in the global navigation of your organization in Buildkite.

name (required) is the name for the new cluster.
description (optional) is the description that appears under the name of cluster in its tile on the Clusters page.
emoji (optional) is the emoji that appears next to the cluster's name in the Buildkite interface and uses the example syntax above.
color (optional) provides the background color for this emoji and uses hex code syntax (for example, #FFE0F1).

A default queue is not automatically created

Connect agents to a cluster

Agents are associated with a cluster through the cluster's agent tokens. Learn more about this in Agent tokens.

Once you have created your required agent token/s, use them with the relevant agents, along with an optional tag representing the relevant queue in your cluster.

You can also create, edit, and revoke other agent tokens from the cluster’s Agent tokens.

Move unclustered agents to a cluster

Unclustered agents are agents associated with the Unclustered area of the Clusters page in a Buildkite organization. Learn more about unclustered agents in Unclustered agent tokens.

Moving unclustered agents to a cluster will allow those agents to use agent tokens that connect to Buildkite via a cluster, and requires at least cluster maintainer privileges.

Organizations created after February 26, 2024

Buildkite organizations created after this date will not have an Unclustered area. Therefore, this process is not required for these newer organizations.

To move an unclustered agent across to using a cluster:

Stop the unclustered agent (from running). To do this, either terminate the agent's running process (for example, via Ctrl-C on the keyboard) or use the Buildkite interface:
1. Select Agents in the global navigation to access the Clusters page.
2. Select Unclustered.
3. From the Unclustered Agents page, select the agent to stop and on its page, select Stop Agent.
Create a new agent token for the cluster the agent will be moved to.
Start the Buildkite agent using the --token value is that of the agent token created in the previous step. Alternatively, configure this agent token's value in the Buildkite agent's configuration file before starting the agent.

If you migrate all your existing agents over to clusters, ensure that all of your pipelines have also been moved to their relevant clusters. Otherwise, any builds for those pipelines will never find agents to run them.

Restrict an agent token's access by IP address

As a security measure, each agent token has an optional Allowed IP Addresses setting that can be used to lock down access to the token. When this option is set on an agent token, only agents with an IP address that matches one this agent token's setting can use this token to connect to your Buildkite organization (through your cluster).

An agent token's Allowed IP Addresses setting can be set when the token is created, or this setting can be added to or modified on existing agent tokens by a cluster maintainer, using the Agent Tokens page of a cluster, as well as the REST API's or GraphQL API's update agent token feature.

For these API requests, the cluster ID value submitted in the request is the target cluster the token is associated with.

Changing the Allowed IP Addresses setting

Modifying an agent token's Allowed IP Addresses setting forcefully disconnects any existing agents (using this token) with an IP address that no longer matches one of the values of this updated setting. This will prevent the completion of any jobs in progress on those agents.

To remove this IP address restriction from an agent's token, explicitly set its Allowed IP Addresses value to its default value of 0.0.0.0/0.

Be aware that an agent token's Allowed IP Addresses setting also has the following limitations:

Access to the Metrics API for this agent token is not restricted.
There is a maximum of 24 CIDR blocks per agent token.
IPv6 is currently not supported.

Using the Buildkite interface

To restrict an existing agent token's access by IP address (via the token's Allowed IP Addresses setting) using the Buildkite interface:

Select Agents in the global navigation to access the Clusters page.
Select the cluster associated with the agent token.
Select Agent Tokens and expand the agent token whose Allowed IP Addresses setting is to be added or modified.
Select Edit.
Update the Allowed IP Addresses setting, using space-separated CIDR notation to the IP addresses which agents must be accessible through.
Select Save Token.

Using the REST API

To restrict an existing agent token's access by IP address using the REST API, run the following example curl command to update this agent token:

curl -H "Authorization: Bearer $TOKEN" \
  -X PUT "https://api.buildkite.com/v2/organizations/{org.slug}/clusters/{cluster.id}/tokens/{id}" \
  -H "Content-Type: application/json" \
  -d '{ "allowed_ip_addresses": "192.0.2.0/24 198.51.100.12" }'

where:

$TOKEN is an API access token scoped to the relevant Organization and REST API Scopes that your agent needs access to in Buildkite.

{org.slug} can be obtained:
- From the end of your Buildkite URL, after accessing Pipelines in the global navigation of your organization in Buildkite.
- By running the List organizations REST API query to obtain this value from slug in the response. For example:
```
curl -H "Authorization: Bearer $TOKEN" \
  - X GET "https://api.buildkite.com/v2/organizations"
```

{cluster.id} can be obtained:
- From the Cluster Settings page of your target cluster. To do this:
  1. Select Agents (in the global navigation) > the specific cluster > Settings.
  2. Once on the Cluster Settings page, copy the id parameter value from the GraphQL API Integration section, which is the {cluster.id} value.
- By running the List clusters REST API query and obtain this value from the id in the response associated with the name of your target cluster (specified by the name value in the response). For example:
```
curl -H "Authorization: Bearer $TOKEN" \
  - X GET "https://api.buildkite.com/v2/organizations/{org.slug}/clusters"
```

{id} is that of the agent token, whose value can be obtained:
- From the Buildkite URL path when editing the agent token. To do this:
  - Select Agents (in the global navigation) > the specific cluster > Agent Tokens > expand the agent token > Edit.
  - Copy the ID value between /tokens/ and /edit in the URL.
- By running the List tokens REST API query and obtain this value from the id in the response associated with the description of your token (specified by the description value in the response). For example:
```
curl -H "Authorization: Bearer $TOKEN" </span>
  - X GET "https://api.buildkite.com/v2/organizations/{org.slug}/clusters/{cluster.id}/tokens"
```

allowed_ip_addresses is/are the IP addresses which agents must be accessible through to access this agent token and be able to connect to Buildkite via your cluster. Use space-separated CIDR notation to enter IP addresses for this field value.

Using the GraphQL API

To restrict an existing agent token's access by IP address using the GraphQL API, run the following example mutation to update this agent token:

mutation {
  clusterAgentTokenUpdate(
    input: {
      organizationId: "organization-id"
      id: "token-id"
      description: "A description"
      allowedIpAddresses: "202.144.0.0/24 198.51.100.12"
    }
  ) {
    clusterAgentToken {
      id
      uuid
      description
      allowedIpAddresses
      cluster {
        id
        uuid
        organization {
          id
          uuid
        }
      }
      createdBy {
        id
        uuid
        email
      }
    }
  }
}

where:

organizationId (required) can be obtained:
- From the GraphQL API Integration section of your Organization Settings page, accessed by selecting Settings in the global navigation of your organization in Buildkite.
- By running a getCurrentUsersOrgs GraphQL API query to obtain the organization slugs for the current user's accessible organizations, followed by a getOrgId query, to obtain the organization's id using the organization's slug. For example:
  
  Step 1. Run getCurrentUsersOrgs to obtain the organization slug values in the response for the current user's accessible organizations:
```
query getCurrentUsersOrgs {
  viewer {
    organization {
      edges {
        node {
          name
          slug
        }
      }
    }
  }
}
```
  Step 2. Run getOrgId with the appropriate slug value above to obtain this organization's id in the response:
```
query getOrgId {
  organization(slug: "organization-slug") {
    id
    uuid
    slug
  }
}
```
  Note: The organization-slug value can also be obtained from the end of your Buildkite URL, by selecting Pipelines in the global navigation of your organization in Buildkite.

id (required) is that of the agent token, whose value can only be obtained using the APIs, by running a getClustersAgentTokenIds query, to obtain the organization's clusters and each of their agent tokens' id values in the response. For example:

  query getClustersAgentTokenIds {
    organization(slug: "organization-slug") {
      clusters(first: 10) {
        edges {
          node {
            name
            id
            agentTokens(first: 10) {
              edges {
                node {
                  description
                  id
                }
              }
            }
          }
        }
      }
    }
  }

description (required) should clearly identify the environment the token is intended to be used for (for example, Read-only token for static site generator), as it is listed on the Agent tokens page of your specific cluster the agent connects to. To access this page, select Agents (in the global navigation) > the specific cluster > Agent Tokens.

If you do not need to change the existing description value, specify the existing field value in the request.
allowedIpAddresses is/are the IP addresses which agents must be accessible through to access this agent token and be able to connect to Buildkite via your cluster. Use space-separated CIDR notation to enter IP addresses for this field value.

Manage maintainers on a cluster

Buildkite administrators or users with the change organization permission can create clusters.

As one of these types of users, you can add and manage other users or teams in your Buildkite organization as maintainers of a cluster in the organization. A cluster maintainer can:

Update or delete the cluster.
Manage agent tokens associated with the cluster.
Manage queues within the cluster.
Add pipelines to or remove them from the cluster.

To add a maintainer to a cluster:

Select Agents in the global navigation to access the Clusters page.
Select the cluster whose user or team is to be added as a maintainer of this cluster.
Select Maintainers > Add Maintainer.
Select if the maintainer will either be a specific User or Team of users.
Select the specific user or team from the drop-down list.
Click Add Maintainer and the user or team is listed on the Maintainers page.

To remove a maintainer from a cluster:

From the cluster's Maintainers page, select Remove from the user or team to be removed as a maintainer.
Select OK to confirm this action.

Move a pipeline to a specific cluster

Move a pipeline to a specific cluster to ensure the pipeline's builds run only on agents connected to that cluster.

Associating pipelines with cluster

A pipeline can only be associated with one cluster at a time. It is not possible to associate a pipeline with two or more clusters simultaneously.

A pipeline can be moved to a cluster by a cluster maintainer via the pipeline's General settings page, as well as the REST API's or GraphQL API's update a pipeline feature.

For these API requests, the cluster ID value submitted in the request is the target cluster the pipeline is being moved to.

Using the Buildkite interface

To move a pipeline to a specific cluster using the Buildkite interface:

Select Pipelines in the global navigation to access your organization's list of accessible pipelines.
Select the pipeline to be moved to a specific cluster.
Select Settings to open the pipeline's General settings page.
On this page, select Change Cluster in the Cluster section of this page.
Select the specific target cluster in the dialog and select Change.

The pipeline's General settings page indicates the current cluster the pipeline is associated with. The pipeline will also be visible and accessible from the cluster's Pipelines page.

Using the REST API

To move a pipeline to a specific cluster using the REST API, run the following curl command:

curl -H "Authorization: Bearer $TOKEN" \
  -X PATCH "https://api.buildkite.com/v2/organizations/{org.slug}/pipelines/{slug}" \
  -H "Content-Type: application/json" \
  -d '{ "cluster_id": "xxx" }'

where:

$TOKEN is an API access token scoped to the relevant Organization and REST API Scopes that your agent needs access to in Buildkite.

{org.slug} can be obtained:
- From the end of your Buildkite URL, after accessing Pipelines in the global navigation of your organization in Buildkite.
- By running the List organizations REST API query to obtain this value from slug in the response. For example:
```
curl -H "Authorization: Bearer $TOKEN" \
  - X GET "https://api.buildkite.com/v2/organizations"
```

{slug} can be obtained:
- From the end of your Buildkite URL, after accessing Pipelines in the global navigation of your organization in Buildkite, then accessing the specific pipeline to be moved to the cluster.
- By running the List pipelines REST API query to obtain this value from slug in the response from the specific pipeline. For example:
```
curl -H "Authorization: Bearer $TOKEN" \
  - X GET "https://api.buildkite.com/v2/organizations/{org.slug}/pipelines"
```

cluster_id can be obtained:
- From the Cluster Settings page of your target cluster. To do this:
  1. Select Agents (in the global navigation) > the specific cluster > Settings.
  2. Once on the Cluster Settings page, copy the id parameter value from the GraphQL API Integration section, which is the cluster_id value.
- By running the List clusters REST API query and obtain this value from the id in the response associated with the name of your target cluster (specified by the name value in the response). For example:
```
curl -H "Authorization: Bearer $TOKEN"
  - X GET "https://api.buildkite.com/v2/organizations/{org.slug}/clusters"
```

Using the GraphQL API

To move a pipeline to a specific cluster using the GraphQL API, run the following mutation:

mutation {
  pipelineUpdate(
    input: {
      id: "pipeline-id"
      clusterId: "cluster-id"
    }
  ) {
    pipeline {
      id
      uuid
      name
      description
      slug
      createdAt
      cluster {
        id
        uuid
        name
        description
      }
    }
  }
}

where:

id (required) is that of the pipeline to be moved, whose value can be obtained:
- From the pipeline's General settings page. To do this:
  1. Select Pipelines in the global navigation > the specific pipeline to be moved to the cluster > Settings.
  2. Copy the ID shown in the GraphQL API Integration section of this page, which is this id value.
- By running the getCurrentUsersOrgs GraphQL API query to obtain the organization slugs for the current user's accessible organizations, getOrgPipelines query to obtain the pipeline's id in the response. For example:
  
  Step 1. Run getCurrentUsersOrgs to obtain the organization slug values in the response for the current user's accessible organizations:
```
query getCurrentUsersOrgs {
  viewer {
    organization {
      edges {
        node {
          name
          slug
        }
      }
    }
  }
}
```
  Step 2. Run getOrgPipelines with the appropriate slug value above to obtain this organization's id in the response:
```
query getOrgPipelines {
  organization(slug: "organization-slug") {
    pipelines(first: 100) {
      edges {
        node {
          id
          uuid
          name
        }
      } 
    }
  }
}
```

clusterId (required) can be obtained:
- From the Cluster Settings page of your target cluster. To do this:
  1. Select Agents (in the global navigation) > the specific cluster > Settings.
  2. Once on the Cluster Settings page, copy the cluster parameter value from the GraphQL API Integration section, which is the cluster.id value.
- By running the List clusters GraphQL API query and obtain this value from the id in the response associated with the name of your target cluster (specified by the name value in the response). For example:
```
query getClusters {
  organization(slug: "organization-slug") {
    clusters(first: 10) {
      edges {
        node {
          id
          name
          uuid
          color
          description
        }
      }
    }
  }
}
```