Test suites
A test suite is a named, reusable collection of workflows you run together. Where a [test run](/reference/test-runs) is one execution, a test suite is the *template* that defines which workflows belong, what order they run in, and how failures are handled. Running a suite produces a test run.
01Layers
A suite organizes its workflows into layers:
- Workflows in the same layer run in parallel.
- Layers run in order — layer 0 finishes before layer 1 starts.
Use layers to express dependencies. Put setup-style flows (sign-up, seed an account) in an early layer, and the flows that depend on them in a later layer, so prerequisites are in place before the dependent workflows run.
02Staged members
A workflow can be a member of a suite but staged — included in the suite's definition without running. Staged members are excluded from execution until you assign them to a layer. Use staging to add a workflow to a suite while you're still stabilizing it, without affecting the suite's results.
03Failure handling
A suite runs with one of two failure modes:
| Mode | Behavior |
|---|---|
| Continue (default) | A failure in one layer doesn't stop later layers. You get full coverage every run. |
| Stop on layer failure | If any workflow in a layer fails, later layers don't run. Use this when a later layer is pointless once an earlier one breaks. |
04Running a suite
Run a suite from the app, or from the CLI:
canary test-suite run canary test-suite run --tag smoke
Each run creates a test run tied to the suite. The run summary identifies the suite and the person who started it, and includes pass and fail totals so you can assess the result at a glance. Re-running the same suite produces a new test run each time, so you can compare results across runs.
After starting a suite, keep the run open to monitor workflows that are still running, waiting, or rescheduled. Waiting work is paused until its event or condition is met; rescheduled work is queued to resume later. These states remain visible while the suite coordinates the run, rather than appearing as unexplained missing results.
After a suite finishes, open its run details to review the workflow results. Use the retry history to distinguish the original attempt from recovery attempts, and treat the latest valid attempt as the current result when an earlier retry is stale or duplicated.
05Reviewing suite results
Open the suite's test run and review the results for each workflow. The run summary shows the suite, who started the run, and the total number of passed and failed workflows. Hover over a person's name to see their complete name when the name is shortened in the summary.
Run details show the original execution, any retry attempts, and the retry history so you can follow recovery without losing the context of the first failure. When work is paused or scheduled to resume, review its status in the run instead of treating it as a pass or failure that has already been finalized.
06Run status
Review each workflow's status to understand whether the suite has completed or still needs to coordinate work:
| Status | Meaning |
|---|---|
| Running | The workflow is currently executing. |
| Waiting | The workflow is paused until an event or condition is met. It can resume automatically when the wait completes. |
| Rescheduled | The workflow is scheduled to resume later. |
| Passed | The workflow completed successfully. |
| Failed | The workflow completed with an error. |
Use the run summary for the completed pass and fail totals, then inspect Waiting or Rescheduled workflows before deciding that the suite has finished. When a workflow appears in more than one attempt, use the latest valid attempt for its current status and review earlier attempts to understand recovery.
07Retry failed workflows
Retry all failed workflows from the suite execution when you want to recover without starting the entire suite again. The retry runs the failed workflows and adds the new attempts to the run details and retry history.
- Open the suite execution with failed workflows.
- Choose Retry failed workflows.
- Wait for the retry to finish, then open the run details.
- Check the retry history and use the latest valid attempt to confirm each workflow's current result.
If a retry also fails, review the latest valid attempt and retry again after addressing the underlying failure. Successful workflows from the original execution do not need to run again.
08Related
- Test runs — what a suite produces when it runs
- Runs & results
- Flows — the workflows a suite is built from
- Triage & diagnostics