Runs

Canary supports four types of runs, each designed for different testing scenarios:

Run Type	Purpose	Prerequisites
Test Runs	Execute all published workflows as regression tests	At least one published workflow
Chaos Runs	Autonomous AI agents explore and test your app	A property configured
Ad-hoc Tests	Run one-off AI-driven tests with custom instructions on a property	The adhoc-tester flag enabled on the property
Workflow Runs	Execute a single workflow (documented below)	A workflow to run

Use Ad-hoc Tests when you want Canary to investigate a property based on custom instructions instead of running a saved workflow or a full regression suite. When the feature is enabled, Canary shows a dedicated Ad-hoc Tests area under Runs so these exploratory runs stay separate from Test Runs, Chaos Runs, and individual workflow runs.

01Workflow Runs

A workflow run is a single execution of one workflow, capturing pass/fail outcomes, screenshots, video recordings, assertion results, and detailed timing diagnostics.

Workflow run details now show richer assertion visibility at the step level. Use them to see which checks passed, raised warnings, or failed on each step, review clearer reasons for each outcome, and debug results without piecing together raw logs.

02Starting a run

1. Open a workflow in the Flow Designer
2. Click Run Flow in the header
3. If the workflow includes input parameters, review the prompt, enter the values you want to use for this run, and click Start Run
4. The sidebar switches to the Runs tab showing live execution

Use workflow inputs when you want to reuse the same workflow with different values at run time instead of editing the workflow before every execution.

Runs can also be triggered as part of a smoke suite or via the API.

03Run statuses

Status	Meaning
Queued	Waiting to start
Running	Execution in progress
Waiting	Paused at a Wait node and will resume automatically after the configured delay or condition
Success	All steps passed
Failed	One or more execution-breaking failures occurred
Canceled	Manually stopped
Completed	Finished and ready for post-run review, including AI-powered triage when available on test suite runs

Treat Waiting as a temporary in-progress state. After the run resumes or reaches its final outcome, Canary updates the run card and run details to show the current status instead of leaving the run on a waiting presentation.

For finished runs, expect playback and run details to reflect the final state:

• Review Waiting as a pause in execution, not a final result
• Review Completed when a run has finished and is ready for post-run analysis
• Review Failed when the run ended unsuccessfully, even if it waited earlier in the workflow
• Refresh the run only if live updates are disconnected or delayed, not because a finished run should still show a waiting card

04Live execution

While a run is in progress, you'll see:

• Browser stream: Live view of the current browser state
• Visible URL: The current page URL stays visible in the live browser so you can follow navigation progress as the run moves through your app
• Agent reasoning: What the AI is doing at each moment
• Canvas status: Running nodes pulse; completed nodes show green (success) or red (failure)
• Assertion results: Assertion outcomes appear inline as checks complete, with pass, warning, and failure counts shown directly on the relevant step
• Live timing: Relative timestamps update continuously so you can tell how recently each step started, changed state, or finished
• Status feedback: Running, waiting, and completed states refresh in place so you can follow progress without reloading the page
• Live run status: The run header and sidebar stay in sync with the current execution state so you can tell at a glance whether the run is still active, waiting, or finished
• Disconnected updates: If the live connection drops, Canary shows that updates are disconnected so you know the run may still be progressing in the background

Open the live run details if you want immediate feedback before the run finishes. As assertions execute, Canary updates each step inline so you can quickly tell whether that part of the workflow is clean, has warning-level issues, or has already failed.

Use the live timing labels to track progress during longer runs. Continuously updating relative timestamps help you tell whether a step is still actively moving, has just completed, or has been waiting for a while.

Use the visible URL in the live browser to confirm where Canary is currently navigating. It helps you distinguish route changes, redirects, and domain hops without waiting for the step list or screenshots to update.

If Canary shows a disconnected state during a live run, keep the run open and wait for updates to resume or refresh the page to reconnect. Use the visible run status and timestamps to confirm whether the run is still active or has already completed.

Warning-level assertions surface validation issues without necessarily stopping the rest of the run. Use them to spot degraded behavior in real time while the workflow continues through later steps.

05Run details

Click any run to see:

• Video: Full session recording with agent thoughts overlaid on the timeline, or a clear unavailable state that explains why playback is missing
• Screenshots: Per-step browser captures, including richer step-level screenshots for recorded workflows and generated workflows created from recordings
• Steps: Each node's status, duration, inline assertion counts, and any error messages
• Scenario label: The scenario name shown with the run so you can tell which variation executed
• Assertion results: A run-level and step-level view of passed, warning, and failed assertions with clearer outcome reasons
• Waterfall: A step-by-step timing view that shows where time was spent across the run
• Diagnostics: Snapshot evidence such as browser console errors and failed network requests surfaced alongside the step where they occurred, with exact wall-clock timestamps available on hover
• Debug view: A dedicated full-screen troubleshooting surface with tabs for overview, steps, navigation history, cache behavior, Playwright traces, and agent thoughts
• Cache badges: Indicators that show when Canary reused cached context or artifacts during execution
• Failure details: Expanded step-level debugging context for failed assertions, step errors, and related evidence
• Sentry trace link: A direct link to the run's trace for deeper investigation when advanced diagnostics are needed
• Docs Updates: Documentation pages created or changed during a Release QA run, including markdown diffs for updated pages

Use the Steps view to scan the run quickly. Each step surfaces an assertion summary directly in the run details, including pass, warning, and failure counts, so you can understand what happened without opening raw logs first.

When a workflow runs with scenarios, use the scenario label in the header or run summary to confirm which input variation produced the result. This is especially useful when the same workflow appears multiple times in run history with different scenarios.

For Release QA results and other release-related result surfaces, use the result icon or run link to jump straight into the full run details. Open the linked run when you need the full timeline, step list, playback, and diagnostics instead of staying on the summarized release result.

Open Debug view when you need a focused investigation workspace. It opens in a full-screen layout so you can review step context, navigation activity, cache behavior, traces, and agent thoughts without switching between smaller panels.

Use each Debug view tab for a different kind of investigation:

Tab	Use it to diagnose
Overview	The overall run outcome, failing areas, and the most important context to review first
Steps	Which workflow step ran, how long it took, and where assertions, errors, or retries happened
Navigation History	Where the browser moved during the run, including redirects, domain changes, and in-app route changes
Cache Behavior	Whether Canary reused cached context or artifacts and whether cache reuse may have affected the run
Playwright Traces	Lower-level browser interaction details when you need deeper investigation into what the browser did
Agent Thoughts	What Canary was trying to do at each point so you can compare intent with the observed result

Start in Overview, then move to Steps or Navigation History depending on whether you are debugging a failing action or an unexpected page transition.

Step details also surface diagnostics panels and cache-related badges in context. Use them to confirm whether Canary reused cached state, whether the browser logged errors, and whether failed network requests or other evidence explain the outcome on that step.

When a run fails or a page behaves unexpectedly, inspect the snapshot diagnostics on the affected step. Browser console errors and failed network requests give you extra evidence about JavaScript failures, missing assets, blocked API calls, and other issues that may not be obvious from the screenshot alone. Hover over diagnostic timestamps to see the exact wall-clock time so you can line up logs and requests with the screenshot, video, and what happened during the run.

Playback or recordings

Use playback to review what happened across the full run, including runs that pause at Wait nodes and then continue later. Canary keeps playback and step context aligned across those waits, so you can review long-running workflows as one continuous run instead of treating each resumed segment as separate investigation work.

When a run finishes after a wait, expect the player, run card, and run details to show the final state. A completed run should show Completed, Success, Failed, or Canceled as appropriate instead of remaining on a waiting card.

Review playback state with these expectations:

Playback state	What you should see	How to use it
Waiting	The run is paused at a Wait node and has not reached its final outcome yet	Monitor the countdown or resume condition, then return later for the final result
Completed	The run has finished and is ready for review	Open playback, Steps, and Waterfall to inspect the full outcome
Failed	The run ended with an execution-breaking failure	Start with the failed step, then compare playback, screenshots, and diagnostics

If a run resumes after a wait, use the same run record to continue your review. Workflow state and notification outputs now carry forward more reliably, so longer or multi-segment workflows remain easier to trace from the earlier steps through the resumed portion of the run.

If a run does not have a playable recording, Canary now shows an explanation instead of leaving the video area empty. Use that state to understand whether playback was never expected, could not be created, or may be recoverable on a later run.

When playback is available, keep an eye on the visible page URL in the player. Use it to follow navigation progress during review and confirm where the browser was at each point in the recording.

In multi-tab workflows, recordings follow the active tab more reliably. Use the recording together with the visible URL and step list to confirm that the captured playback matches the tab Canary was actively driving.

Common missing-recording states include:

State	What it means	What to do next
API-only run	The run executed without a browser session, so there is no browser recording to play	Review step results, assertions, logs, and other artifacts instead of expecting playback
No browser interaction	The workflow ran, but no browser actions occurred that would produce a useful recording	Use screenshots, step timing, and assertion results to review the run
Failed before recording started	The run stopped very early, before Canary could begin recording	Fix the early failure and run the workflow again if you need playback
Recording upload problem	The run completed browser activity, but the recording could not be uploaded successfully	Re-run the workflow to generate a new recording and use screenshots and step details for the current run

Treat the missing-video message as guidance, not a generic error. It tells you whether the absence of playback is expected for that run type or whether a fresh run is the best recovery path.

If playback is unavailable, continue debugging from the rest of the run details. Open Steps, Assertion results, Debug view, screenshots, diagnostics, and the Waterfall view to understand what happened even without a recording.

Understanding run details

Start with the assertion summary at the run level, then drill into the step-level breakdown.

• Use the run-level summary to see the total number of assertions that Passed, Warned, or Failed across the whole run
• Use the step list to find where those results occurred
• Check the scenario label first when the workflow has multiple scenarios so you know which variation you are reviewing
• Open the run from a Release QA result or other summarized result surface when you need the full run context behind that result
• Open Debug view to inspect the selected step with timing, cache details, navigation history, thought summaries, and trace access in one place
• Open a step to review the individual assertions for that step and read the condition text and outcome reason in full
• Review diagnostics panels for browser console errors and failed network requests on the same step
• Hover over diagnostic timestamps when you need the exact wall-clock time for a console message or network event
• Check cache badges and cache details to understand whether Canary reused cached context during that part of the run
• Use Navigation History to confirm whether redirects, cross-domain moves, or in-app route changes led to the observed result
• Use the visible page URL in playback to confirm exactly where the browser was when a step ran or a failure occurred
• Compare assertion results with synchronized screenshots, video, agent thoughts, and timing data to confirm what happened
• Open the trace link when you need deeper investigation beyond the run detail page

Assertions in run results

Assertion reporting appears in two places in run details:

View	What you see	How to use it
Run summary	Total counts for passed, warned, and failed assertions	Quickly assess overall run quality before opening individual steps
Step details	Per-step assertion results with clearer condition text and richer reasons	Identify the exact checks that passed, raised warnings, or failed

Inspect step-level assertion results to understand severity at a glance:

Result state	What it means	How to use it
Pass	The assertion succeeded	Confirm the expected behavior happened on that step
Warn	The assertion did not meet the expected condition, but the run continued and remains separate from execution-breaking failures	Review behavior that needs attention without treating it as the primary failure reason
Fail	The assertion did not meet the expected condition and marked the step or run as failed	Start debugging from this assertion and compare it with step artifacts

Use the assertion text to understand what Canary evaluated. Clearer condition text and outcome details help you confirm what was expected, what Canary observed, and why the assertion passed, warned, or failed.

Use the Waterfall view to inspect timing in order from the start of the run to the end. Each row represents a workflow step, and expanded timing details can show additional turn-level timing when available.

The waterfall helps you quickly answer questions like:

• Which step took the longest
• Whether time was spent in one slow step or spread across many steps
• Which agent turns within a step added most of the delay
• Whether a run spent time actively working or waiting between actions

Timing details are available from the run details view. The level of detail can vary by run and step, so some runs show only step duration while others also show richer per-turn timing.

For Release QA runs, open the Docs Updates tab to review documentation pages included in the release. Use this tab to confirm which pages were added, inspect markdown diffs for updated pages, and verify doc changes before you finish your release review.

06Investigating failures

When a run fails:

1. Open Debug view first to see the failing step, step timing, navigation history, cache behavior, thought summaries, and trace access together
2. If the failed run is a completed test suite run and AI-powered triage is available, wait for diagnostics to finish and review the triage summary first
3. Check whether Canary grouped related failures into clusters, then open the most urgent cluster to start with the issue most likely blocking the suite
4. Review the suggested next actions in the triage panel to decide which failing workflow, assertion, or app issue to inspect first
5. Check the Assertion results and Steps list to see whether the issue came from a failed assertion, a warning that points to a degraded state, or another step error
6. Confirm the scenario label on the failing run so you know which workflow variation failed
7. Compare the same workflow across run history to see whether the failure repeats on one scenario or appears only on a new scenario
8. If you started from a Release QA result or another summarized result, open the linked run details before you investigate further
9. Review the inline assertion counts on the affected step to see whether the step includes passed, warning, or failed checks
10. Open the affected step and read the assertion text and outcome reason for each result to identify the exact check that passed, warned, or failed
11. Review the step diagnostics panels for browser console errors, failed network requests, and other supporting evidence that happened at the same time as the failure
12. Check Navigation History to see whether redirects, route changes, or unexpected domain transitions happened before or during the failing step
13. Hover over console and network diagnostic timestamps to get the exact wall-clock time when you need to align events with screenshots, video, or external logs
14. Check cache badges and cache details on the step so you can tell whether cached context may have influenced the run
15. Review Playwright Traces when you need deeper browser-level evidence beyond the step summary
16. Review Agent Thoughts to understand what Canary was trying to do at the point of failure
17. Click View Details to open the failure panel with:

• A wider layout for inspecting more context at once
• Expandable sections for assertions, diagnostics, and related evidence
• A failure screenshot with zoom so you can inspect the browser state closely
• Separate actual-versus-expected details where available
• Additional context for debugging

1. Use synchronized playback to compare the failed step with the video timeline, screenshots, and agent thoughts
2. If the visible page URL is shown in playback, use it to confirm the exact route or page where the failure happened
3. If the failure happened after a wait, confirm the run is shown as Failed or Completed rather than assuming it is still paused
4. If you need deeper diagnostics, open the Sentry trace link from run details
5. Use the canvas to confirm which node failed

Use Debug view and the step panel together. Start with Overview for the high-level failure context, then use Steps, Navigation History, Playwright Traces, and Agent Thoughts to confirm what happened and why.

On completed test suite runs, use AI-powered triage to reduce the amount of manual sorting you need to do. Canary can wait for diagnostics, cluster related failures, mark the most urgent cluster, and surface likely next actions so you can move from suite-level failure to root-cause investigation faster.

Use the scenario name in run history to spot patterns faster. If the same scenario fails across multiple runs, treat it as a repeated failure. If only one newly added scenario fails, investigate what is unique about that variation first.

Use the snapshot diagnostics to separate frontend and backend issues faster. Browser console errors often point to broken scripts or rendering problems, while failed network requests can show missing resources, authorization failures, or upstream errors. Hover over the timestamp on either diagnostic to match the event to the exact moment shown in other run artifacts.

Use the Waterfall view alongside the failure details to see whether the run slowed down before the failure or failed immediately at a specific step.

If the run includes warnings but does not fully fail, review those steps the same way. Warning-level assertions help you catch validation issues that need follow-up without blocking the rest of the workflow.

If the run has no recording, do not treat that by itself as the root cause. First read the playback explanation to see whether the run was API-only, had no browser interaction, failed before recording started, or hit an upload problem. Then continue with screenshots, step details, assertions, diagnostics, and timing to find the actual failure.

Interpreting warnings and failures

Warnings and failures now appear as distinct assertion states in run details. Treat them differently when you review results.

State	Effect on the run	What to do next
Warn	Flags an assertion that did not meet the expected condition, but does not mark the run as failed on its own	Review the step, confirm whether the behavior is acceptable, and decide whether follow-up is needed
Fail	Marks the assertion as failed and can cause the step or run to fail	Debug the failing step first and compare the assertion with screenshots, video, and timing

Use warnings to catch degraded behavior early. A warning can point to a UI issue, missing content, or another unexpected state that did not stop execution but still needs attention.

Use failures to identify the primary reason a run stopped or ended unsuccessfully. Read the full condition text and outcome reason on the failed assertion, then review related warnings on the same step for additional context.

07Wait nodes

Workflows with Wait nodes pause mid-execution:

1. Run executes until it hits the Wait node
2. Status shows Waiting with a countdown timer
3. Execution resumes automatically after the delay
4. The run continues in the same run record, and Canary preserves workflow state, step context, and notification outputs more reliably across the wait
5. When the run finishes, the final status changes to Success, Failed, Canceled, or Completed as appropriate

Wait time appears in the run timing details, so you can distinguish intentional delays from unexpectedly slow execution.

For long-running or multi-segment workflows, keep reviewing the same run after it resumes. Canary carries the run forward more reliably after waits, so you can follow the earlier steps, resumed activity, and final outputs in one place.

08Artifacts

Each run stores:

Artifact	Description
Screenshots	Browser capture at each step, with richer step-level screenshots preserved for recorded workflows and generated workflows created from recordings
Video	Full session recording when available, or an explanation of why playback is unavailable. In multi-tab workflows, recordings more reliably follow the active tab.
Diagnostics	Snapshot evidence including browser console errors and failed network requests collected for investigated steps, with exact wall-clock timestamps available on hover

Use diagnostics artifacts when screenshots or video do not explain the failure on their own. They help you confirm whether the browser logged errors or whether important requests failed during the step.

Use richer step-level screenshots when you review a recorded workflow or a generated workflow from a recording. They give you more visual context for each step so you can verify what Canary saw without relying only on the final video.

If the Video panel is unavailable, use the explanation shown in run details to decide your next step. API-only runs and runs with no browser interaction do not need recovery, while early failures and upload problems are best handled by fixing the underlying issue and running the workflow again.

09Interpreting results

Use the run details views together to understand both outcome and performance:

• Check Steps to confirm which nodes passed, warned, failed, or waited
• Check Assertion results to see the outcome of each check directly in the run details
• Interpret assertion statuses as follows:
  • Pass: The assertion succeeded
  • Warn: The assertion did not meet the expected condition, but remains separate from execution-breaking failures
  • Fail: The assertion did not meet the expected condition and marked the run or step as failed
• Check the run-level assertion summary first to understand the overall mix of passed, warned, and failed checks
• Use warning counts to identify steps that need review even when the overall run succeeds
• If the workflow uses scenarios, check the scenario label before comparing this run with another run of the same workflow
• Use scenario labels in run history to tell whether a failure is repeated on the same variation or isolated to a different one
• If the run is a completed test suite run and AI-powered triage is available, review the triage summary after diagnostics finish
• Use failure clusters to see which failures likely share the same cause
• Start with the cluster marked as most urgent when you need to prioritize what to investigate first
• Review suggested next actions to move from suite-level signals into the right failing workflow or step
• Open summarized release results into the linked run details when you need the full inspection workflow
• Open step details to review the per-step assertion breakdown and read the full condition text and outcome reason for each result
• Open failure details when you need the expanded debugging view with zoomable screenshots, expandable sections, and actual-versus-expected context
• Review diagnostics panels for console errors, failed network requests, and other supporting evidence on the same step
• Check cache badges when you want to understand whether Canary reused cached state during execution
• Use synchronized playback context to compare screenshots, video, and agent thoughts with the selected step
• Use the visible page URL in playback to track where Canary was in the application during the run
• Check Waterfall to compare duration across steps in sequence
• Expand timing details to inspect per-turn timing when available
• If the run needs deeper investigation, open the Sentry trace link from run details
• For Release QA runs, check Docs Updates to review which documentation pages were created or changed

Treat warnings and failures differently during review. Start with failures when a run is unsuccessful, then review warnings for additional issues that need follow-up. If a run succeeds with warnings, use those warning states to prioritize manual review instead of treating the run as broken.

On completed test suite runs, let AI-powered triage help you interpret the suite before you drill into individual failures. Grouped failures, urgency signals, and suggested next actions help you understand whether the suite is failing because of one broad issue or several separate problems.

A long step does not always indicate a problem. Review whether the step includes intentional waiting, repeated agent turns, or external app delays before you treat it as a workflow issue.

When you review a Release QA run, use Docs Updates alongside the other run details. Open changed pages to inspect markdown diffs and confirm that release-related documentation updates are complete and accurate.

10Troubleshooting slow runs

If a run takes longer than expected:

1. Open the run and review the Waterfall view
2. Identify the slowest step or group of slow steps
3. Expand the step to review per-turn timing when available
4. Compare the timing with synchronized screenshots, video, agent thoughts, assertion results, and any error details
5. Check whether the delay comes from a Wait node, repeated retries, cache recovery, or slow page/app behavior
6. Review the live and completed relative timestamps to confirm whether the slowdown is ongoing or isolated to one part of the run

Focus on whether the slowdown is isolated or consistent:

• If one step dominates the timeline, inspect that step first
• If many steps are slightly slow, look for broader app or environment issues
• If the run includes waiting, confirm the delay is expected
• If the run already finished after a wait, review the final run status instead of assuming the run is still paused
• If cache badges appear around a slow or retried step, review that context before deciding the workflow itself is broken
• If per-turn timing is not shown, use step duration, screenshots, video, and assertions to narrow down where time was spent

Some runs may not show the same level of timing detail for every step. When richer diagnostics are unavailable, use the available step durations and artifacts to continue debugging.

11Managing runs

From the Runs area, open the run type you want to review. When the adhoc-tester flag is enabled for your property, the left navigation includes a dedicated Ad-hoc Tests section under Runs.

Use this section to keep one-off AI-driven tests separate from other run types:

• Open Ad-hoc Tests to review exploratory tests started from custom instructions
• Open Test Runs to review regression runs across published workflows
• Open Chaos Runs to review autonomous exploratory testing sessions
• Open workflow runs from the Flow Designer when you need the result of a single workflow execution

The Runs panel also gives you more context while runs are active or recently completed. Use the list to monitor live status, see when a run stopped sending live updates, and scan summary indicators before opening the full run details.

In the run list, you may also see:

• Pass rate: A summary signal for how many checks or workflows are currently passing in the run set
• Quarantined items: Indicators that separate quarantined workflows or results from the main pass/fail signal
• Replay activity: Status cues when Canary is performing certain background replay work related to the run
• Scenario names: Clearer labels on workflow runs so you can distinguish the variation that executed

Use these indicators to prioritize what to open first. For example, open runs with lower pass rate, new quarantined items, active replay activity, or an unexpected scenario name when you need to understand changes in run quality or investigate unusual behavior.

For completed test suite runs, the run detail view can also surface AI-powered triage after diagnostics are ready. Use it to understand clustered failures, identify the most urgent issue group, and choose the next investigation step faster.

For Release QA and other release-related result views, use the result icon or linked run entry to move from the summarized result into the underlying run. Open related runs this way when you need to inspect a specific test outcome, compare it with the full run timeline, or continue debugging in Run details.

When multiple runs are related to the same release result, use those links to move between the summarized result and the individual runs it references. This helps you compare the release-level signal with the exact workflow run, test run, or related run details behind it.

For details about starting, monitoring, and canceling ad-hoc testing sessions, see Ad-hoc Tests.

From the run detail page, you can cancel a Release QA run that is no longer needed.

1. Open the Release QA run
2. Click Cancel Run
3. Confirm the cancellation

After you cancel the run, its status changes to Canceled. Use this when a run is stuck, no longer relevant, or was started by mistake.

12Related

• Ad-hoc Tests
• Workflow inputs
• Test Runs
• Chaos Runs
• Smoke Suites
• CI/CD integration
• Release QA
• Run timeline and history
• Sentry trace reference