Docs/Running your tests
#

Running your tests

Run the same suite three ways — on demand, on every change in CI/CD, and on a schedule — from the UI or the CLI.

Once you have flows worth trusting, the question becomes when to run them. Cofactor runs the same suite three ways — you pick the trigger that fits the moment.

  • On demand — kick off a run yourself while you build and debug.
  • On every change — wire a CI gate so every push and pull request runs your tests before merge.
  • On a schedule — run a suite on a cadence (a nightly smoke run is the classic) to catch drift that creeps in between changes.

Every strategy runs the same promoted flows and the same tags. Pick your journeys and tag them once (see Building your first suite), then reuse them in all three places below.

01Before you start

  • At least one promoted flow. Runs only ever include promoted flows — your work-in-progress drafts and paused flows stay out of the gate.
  • An API key for the CLI and CI/CD. (Interactive login also works for local use.)

Install the CLI

bash
# npm
npm install -g @runcofactor/cli

# or bun
bun add -g @runcofactor/cli
bash
cofactor --version

Authenticate

Create a scoped API key for automation:

  1. Go to Settings → API Keys (admin access required).
  2. Click Create Key and give it a descriptive name, e.g. github-actions-smoke.
  3. Pick the smallest scope that can run tests.
  4. Copy the key — it's shown once.

Store it in your CI platform's secret manager. Never commit it.

021 · Run on demand

Use this while you're building or debugging — kick off a run and watch it stream.

  1. Go to Suites → Test Runs and click Start Test Run.
  2. Choose a Scope:
    • All promoted workflows — run everything.
    • By tag — run every flow carrying a tag (e.g. smoke).
    • By suite — run a named, layered suite.
  3. Pick the environment to run against.
  4. Click Start. Results stream in live.

Tags and named suites are two ways to scope the same flows — a tag is the zero-maintenance quick path; a named suite is a saved, curated gate that can also run tests in order. See Building your first suite for when to use each.

032 · Run on every change

Gate your merges: run your tests automatically on every push and pull request so regressions never reach your main branch. Drive the CLI from your pipeline — a non-zero exit fails the step, so a failing test blocks the merge.

GitHub Actions

.github/workflows/cofactor.yml:

yaml
name: Cofactor Tests
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm install -g @runcofactor/cli
      - name: Run smoke tests
        env:
          COFACTOR_API_TOKEN: ${{ secrets.COFACTOR_API_TOKEN }}
        run: cofactor test-suite run --tag smoke

GitLab CI

yaml
cofactor-tests:
  image: node:20
  script:
    - npm install -g @runcofactor/cli
    - cofactor test-suite run --tag smoke
  variables:
    COFACTOR_API_TOKEN: $COFACTOR_API_TOKEN

CircleCI

yaml
version: 2.1
jobs:
  cofactor-tests:
    docker:
      - image: cimg/node:20.0
    steps:
      - run: npm install -g @runcofactor/cli
      - run:
          command: cofactor test-suite run --tag smoke
          environment:
            COFACTOR_API_TOKEN: $COFACTOR_API_TOKEN
workflows:
  test:
    jobs:
      - cofactor-tests

Jenkins

groovy
pipeline {
  agent any
  environment { COFACTOR_API_TOKEN = credentials('cofactor-api-token') }
  stages {
    stage('Cofactor Tests') {
      steps {
        sh 'npm install -g @runcofactor/cli'
        sh 'cofactor test-suite run --tag smoke'
      }
    }
  }
}

Run a fast --tag smoke subset on every change, and save the full suite for the nightly schedule below.

043 · Run on a schedule

A schedule catches what change-triggered runs miss: data drift, expiring credentials, third-party changes, and slow degradations. A nightly smoke run is the most common setup.

Schedule a run from Settings → Automations — no pipeline required:

  1. Click New automation and choose Describe your automation to set it up with AI, or configure it manually.
  2. Set the trigger to Schedule — run daily at a time (with timezone) or on any cron expression. Describe the cadence in plain language ("every weekday at 7am") and Cofactor writes the cron for you.
  3. Set the action to run your tests, scoped by tag or suite.
  4. Save. The automation runs your suite on that cadence and reports results like any other run.

05Read the results and triage

Open the run in the app to watch each flow pass or fail, then group the failures and dig into traces.

  • The suite run detail page shows pass/fail per flow, with screenshots and traces.
  • AI triage clusters related failures, so you fix one root cause instead of chasing every red row.

See Triaging a failure for the full workflow.

06Exit codes

Both run paths set process exit codes, so a pipeline step fails when a test fails.

Command012
cofactor test-suite runall passed (or nothing matched)one or more failed, or an error—
cofactor triage waittriage completedtimeout or errortriage reported a failure

07Next steps

← PREVIOUS
Building your first suite