CI/CD Integration

Run your workflow tests automatically as part of your CI/CD pipeline. The Canary CLI triggers remote test runs, streams results in real-time, and exits with the appropriate code for your pipeline.

01Prerequisites

• You have created Flows that you want to run in CI/CD.
• You have an API key with the permissions your automation needs.

New to the CLI? Start with Running tests with the CLI for a walkthrough.

021) Create an API Key

API keys authenticate your CI/CD pipeline with Canary. Unlike user sessions, they're designed for automated systems.

Canary now supports scoped API keys. For CI/CD, create a key with only the permissions your pipeline needs instead of using a full-access key.

1. Go to Settings → API Keys
2. Click Create Key
3. Give it a descriptive name (for example, github-actions-smoke or circleci-production)
4. Select the smallest permission scope that matches your workflow
5. Copy the key immediately — it will only be shown once

The key format is cnry_....

Recommended scopes for CI/CD

Use least-privilege access whenever possible.

Legacy keys and older full-access keys continue to work, but they grant broader access than most CI/CD pipelines need. Prefer a new scoped key for new pipelines, and replace older full-access keys over time where possible.

Security tips:

• Store the key as a secret in your CI/CD platform (never commit it to code)
• Create separate keys for different pipelines so you can revoke them independently
• Use separate keys for staging and production workflows
• Prefer scoped keys over full-access keys
• Rotate keys periodically

032) Install the Canary CLI

Install the CLI globally or as a dev dependency:

# Global install
npm install -g @canaryai/cli

# Or as a dev dependency
npm install -D @canaryai/cli

043) Run remote tests

Use canary test --remote to trigger your workflow tests:

# Run all workflows
canary test --remote --token cnry_your_api_key

# Filter by tag
canary test --remote --token cnry_your_api_key --tag smoke

# Filter by name pattern
canary test --remote --token cnry_your_api_key --name-pattern "checkout"

# Combine filters
canary test --remote --token cnry_your_api_key --tag smoke --name-pattern "auth"

# Verbose mode (stream all events)
canary test --remote --token cnry_your_api_key --verbose

Exit codes

Environment variables

Instead of passing --token on every command, set the CANARY_API_TOKEN environment variable:

export CANARY_API_TOKEN=cnry_your_api_key
canary test --remote --tag smoke

You can also set CANARY_API_URL if you're using a self-hosted Canary instance:

export CANARY_API_URL=https://api.your-canary-instance.com

Authentication guidance for CI/CD

Use a scoped API key stored in your CI/CD platform's secret manager, then expose it as CANARY_API_TOKEN only for the step that runs Canary.

For the most common setup, a key that can run workflows/tests is enough. If your automation also needs to review run history or fetch results as part of a follow-up step, use a key that also includes read access for workflow run results.

If you're migrating an older pipeline, note that legacy and full-access keys still authenticate successfully. The recommended change is to swap them for scoped keys so each pipeline only has the access it requires.

05CI/CD Examples

GitHub Actions

Use GitHub Actions when you want pull requests and branch pushes to trigger Canary runs directly from your repository workflow.

If you need automated release validation with GitHub check reporting, see Release QA. Release QA lets you link one or more GitHub repositories to a release, scope regression coverage by tags, and post regression-suite status back to matching pull requests.

name: Canary Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Canary CLI
        run: npm install -g @canaryai/cli

      - name: Run smoke tests
        env:
          CANARY_API_TOKEN: ${{ secrets.CANARY_API_TOKEN }}
        run: canary test --remote --tag smoke

GitLab CI

canary-tests:
  image: node:20
  stage: test
  script:
    - npm install -g @canaryai/cli
    - canary test --remote --tag smoke
  variables:
    CANARY_API_TOKEN: $CANARY_API_TOKEN

CircleCI

version: 2.1

jobs:
  canary-tests:
    docker:
      - image: cimg/node:20.0
    steps:
      - checkout
      - run:
          name: Install Canary CLI
          command: npm install -g @canaryai/cli
      - run:
          name: Run smoke tests
          command: canary test --remote --tag smoke
          environment:
            CANARY_API_TOKEN: $CANARY_API_TOKEN

workflows:
  test:
    jobs:
      - canary-tests

Jenkins

pipeline {
    agent any

    environment {
        CANARY_API_TOKEN = credentials('canary-api-token')
    }

    stages {
        stage('Canary Tests') {
            steps {
                sh 'npm install -g @canaryai/cli'
                sh 'canary test --remote --tag smoke'
            }
        }
    }
}

06Filtering workflows

Use tags and name patterns to run specific subsets of your workflows.

If you are deciding which flows belong in automated release validation, use the same tagging strategy here that you plan to use in Release QA. Consistent tags make it easier to reuse smoke and regression coverage across CLI-driven CI and release-level validation.

By tag

Tags let you categorize workflows (e.g., smoke, regression, checkout). Add tags when creating flows in the UI.

# Run only smoke tests
canary test --remote --tag smoke

# Run only checkout-related tests
canary test --remote --tag checkout

By name pattern

Filter workflows by name using case-insensitive matching:

# Run workflows with "login" in the name
canary test --remote --name-pattern login

# Run workflows with "payment" in the name
canary test --remote --name-pattern payment

Combining filters

When using both --tag and --name-pattern, workflows must match both criteria:

# Run smoke tests that also have "auth" in the name
canary test --remote --tag smoke --name-pattern auth

07Verbose mode

Use --verbose (or -v) to see all events as they stream:

canary test --remote --tag smoke --verbose

This shows:

• Suite start/completion events
• Individual workflow status changes
• Detailed timing information

Useful for debugging or when you want full visibility into test execution.

08Troubleshooting

"No API token found"

Make sure you've either:

• Set the CANARY_API_TOKEN environment variable, or
• Passed --token cnry_... on the command line

"Failed to start tests: 401"

Your API key may be:

• Invalid or mistyped
• Revoked (check Settings → API Keys)
• Expired
• Missing the required scope for the action your pipeline is trying to perform

Create a new API key if needed.

"No workflows found matching the filter criteria"

Check that:

• Your filter matches at least one workflow
• The workflows have the tag you're filtering by
• The name pattern matches workflow names (case-insensitive)

If you are preparing flows for release validation, also confirm that the tags you use in CI match the tags you plan to use in Release QA.

Tests pass locally but fail in CI

Common causes:

• Different environment (staging vs production)
• Timing issues (CI may be slower)
• Missing credentials or configuration

Check the workflow run details in the Canary UI for screenshots and traces.

09Next steps

• Create a smoke suite for your most critical flows
• Explore Release QA for automated release validation and GitHub status reporting
• Set up Slack notifications for test failures
• Learn about Runs to understand test execution in detail