Triaging a failure
Turn a red run into a verdict: real bug, flake, agent confusion, or environment, and route each one to the right fix.
A red run is not a verdict. It's a question: what actually broke — the app, the flow, or the setup?
Answer that question fast and everything downstream gets easy: real bugs get filed with evidence attached, flaky flows get fixed instead of rerun, and setup problems stop masquerading as regressions. Cofactor does the first pass for you.
Don't want to read this? Copy this prompt and give it to your agent:
Or read on to do it yourself.
01Cofactor triages first
Every failed run gets an automatic investigation. A diagnostic agent replays the evidence — screenshots, the agent's own reasoning, console and network activity, the run's history — and writes up what it thinks happened before you've even opened the run.
Find it on the issue linked to the failure: open the failed run, click through to its issue, and open the Diagnostics tab.
The diagnosis leads with a one-sentence thesis and a category, backed by failure points, typed evidence, and — when the agent finds one — a smoking gun: the single artifact that explains the failure. It closes with a recommendation for what to do next.
The category tells you where the fix lives:
| Category | What it means | Where the fix lives |
|---|---|---|
software_bug | A clear defect — error page, 5xx, broken behavior | The app. Work the issue. |
app_changed | The app behaves differently than the flow expects | Decide. Intentional change → update the flow. Not intentional → it's a regression. |
agent_confusion | The agent misread the UI or took a path no user would | The flow's intent. Sharpen step descriptions and goals. |
test_underspecified | The flow doesn't say enough about what success looks like | The flow. Add assertions, tighten the scenario. |
data_collision | Test data clashed — duplicate records, shared state | The flow's data. Give each run its own data; use macros. |
dependency_issue | Something the test depends on let it down — an external service, or data that was missing or wrong | The dependency. Fix the data or service, or remove it from the test path. |
platform_error | Cofactor infrastructure hiccup | Us. Rerun; report it if it persists. |
If a diagnosis didn't complete, hit Re-run Analysis on the Diagnostics tab.
02What an issue is
When Cofactor verifies a failure — positive evidence, not just "something went wrong" — it becomes an issue: the durable record of a problem, separate from any single run.
Issues deduplicate themselves. When the same problem shows up again — next run, next suite, next day — the new occurrence attaches to the existing issue instead of creating noise. The occurrence count and last-seen date tell you whether a problem is live, recurring, or stale.
Each issue carries everything triage needs in one place:
- a video replay of the failing moment, plus repro steps
- the diagnosis (the Diagnostics tab above)
- console and network evidence captured at failure time
- every occurrence, with links back to the runs that hit it
- status —
open,closed, ornot_a_bug— severity, comments, and links out to external tickets
The same record is scriptable:
cofactor issues list --status open --severity high cofactor issues get <issueId> cofactor issues close <issueId> --not-a-bug --reason "Expected copy change, not a defect"
not_a_bug is a first-class outcome. Closing a false positive with a reason teaches the record — the why stays attached if the pattern ever resurfaces.
03Triage a whole suite
Seven failures rarely mean seven problems. When a suite execution comes back red, suite-level triage waits for each failure's diagnosis, then clusters the failures by root cause — so you fix causes, not symptoms.
Open the suite execution and switch to the Triage tab. Clusters are ranked, the most pressing one is front and center, and each failure carries a rationale for why it belongs to its cluster. New failures — ones that weren't failing before — are flagged, because a new red is a stronger signal than a known one.
The same view from the terminal:
# Run a suite — the execution id (and a triage hint) print at the end cofactor test-suite run --suite "smoke" # Block until triage finishes, then print the clustered summary cofactor triage wait latest # The full picture: ranked clusters, rationale per failure, suggested next steps cofactor triage get latest # Just the regressions — newly-failing flows only cofactor triage failures latest --new-only
latest resolves to your most recent suite execution; pass an explicit id (from cofactor test-suite list) to look at older ones. Each cluster comes with a suggested next step, and --cluster <id> drills into one cluster when you're ready to work it.
04Go deeper with the CLI
For a stubborn root cause, cofactor workflow debug opens up everything a run recorded. Start wide and narrow in:
# The diagnosis dashboard: failed steps, slow steps, cache misses cofactor workflow debug overview <runId> # Where did the time go? cofactor workflow debug steps <runId> --slow # What was the agent thinking at the moment it went wrong? cofactor workflow debug thoughts <runId> --grep "delete" # Why did cached steps fall back to the agent? cofactor workflow debug cache <runId> --misses-only # Open the full Playwright trace in the viewer cofactor workflow debug playwright <runId> --open
The run id is on the run page, in the suite detail (cofactor test-suite get <executionId>), and in triage output.
And when you'd rather watch it happen: cofactor debug <workflowId> replays the flow in a headed browser on your machine — watch each step live, stop early with --to-step N, then take the wheel with cofactor session take-control to poke at the page yourself.
Let your coding agent do the digging
Root-causing a run is exactly the kind of work a coding agent is great at — and Cofactor ships a skill that teaches it the whole playbook. Install debug-run once:
cofactor skill install debug-run # auto-detects Claude Code, Cursor, Copilot…
Then hand your agent a failure in plain language — "why did run 257e24c6 fail?" — or just paste the run's URL. The skill takes it from there:
- Materializes the run into a local workspace (
cofactor workflow debug materialize <runId>) — the same evidence Cofactor's own diagnostic agent reads: the run summary, every step, the agent's thoughts, network and console logs, DOM snapshots, and the last passing run for comparison. - Treats the reported failure as a hypothesis, not a verdict — it verifies against the evidence and compares with the last green run before blaming anything.
- Reports back with a category, the evidence that earned it, and a recommendation.
The payoff: it's your coding agent, sitting in your repo. When the verdict is software_bug, it can go fix the bug in the same session; when it's agent_confusion or test_underspecified, it can tighten the flow. Triage and fix, one conversation.
Keep the skill current with cofactor skill sync. For setup and the rest of the agent toolkit, see Driving Cofactor from your coding agent.
05Don't blame the app without evidence
The cardinal rule of triage: a regression claim needs positive evidence, not absence of green.
- A rerun that passes points to flake or timing, not a bug. Stabilize the flow — see Designing a good test.
- A missing precondition — no seed data, expired credential, wrong environment — is a setup problem. Fix the setup, rerun, and don't file anything.
- A green-to-red flip is only a strong app-change signal when the flow was stably passing before. An oscillating history points at flake or agent confusion first.
- A path no user would take means the flow's intent needs tightening, not the app.
When the evidence does hold up — the diagnosis says software_bug, the replay shows it, a rerun reproduces it — the issue is already filed, with the receipts attached. That's the point of triage: by the time a human says "this is real," the proof is one link away.
06Related
- Issues — the full issue model
- Runs — what every run records
- Designing a good test — stabilize flaky flows for good
- Driving Cofactor from your coding agent — agent-driven root-causing