Ad-hoc testing

Sometimes you don't want a test suite. You want an answer: does this thing work right now?

That's an ad-hoc test. You describe what to check in plain language — "sign up as a new user and make sure the welcome email settings save" — and Cofactor's agent opens a real browser, does it, and reports back with a verdict, verified issues, and a full recording. No flow to author, no selectors, no setup ceremony.

And because questions come up everywhere, you can ask from anywhere: the app, a GitHub pull request, or a Slack thread. Results come back to wherever you asked.

01In the app

The home base. Go to Runs → Ad-hoc Tests and click New Ad-hoc Test.

Three decisions, two of which are usually already made for you:

1. Application — which app to test.
2. Environment — if the application has a default sandbox, it's preselected ("Targeting staging-sandbox"). Keep it. (More on why below.)
3. Instructions — what to test and what "working" looks like.

Optionally add a start URL to drop the agent on a specific page, or a credential if you're targeting the live app and need a signed-in user. Click Start Test and watch the live browser session as the agent works.

02From a GitHub pull request

The fastest way to answer "does this PR actually work?" — without leaving review.

Mention @cofactor (Cofactor's GitHub app) in a comment on any PR in a connected repository:

@cofactor make sure the new pricing banner shows for free-tier users

Or just @cofactor on its own — the agent defaults to testing the behavior changed by the diff and described in the PR.

Here's the good part: Cofactor provisions a sandbox running your PR's branch. The agent isn't testing production or staging — it's testing this change, on a fresh environment, before it merges.

Cofactor posts a status comment immediately and edits it in place as the run progresses, ending with:

• a verdict — ✅ passed, ❌ issues found, or ⚠️ couldn't complete
• the test plan with per-item results
• verified issues, each with a replay link
• a recording link that never expires — safe to share in review threads

Reviewers get evidence in the conversation they're already in. Nobody pastes screenshots into Slack at 5pm.

03From Slack

Mention the Cofactor assistant in any channel it's installed in and ask it to test something:

@Cofactor can you check that checkout works with a discount code? See PROJ-142

The assistant figures out which application you mean, starts the run (using the default sandbox when one is configured), and replies in the thread when it's done — verdict, issues, and replay videos uploaded right into the conversation.

Tickets ride along

Mention a ticket key — Jira or Linear — anywhere in your request and Cofactor links it to the run automatically:

• Jira (PROJ-142): when the run finishes, Cofactor writes the results back to the ticket — verdict, verified issues, per-issue replay links, and video attachments. The ticket becomes the record; teammates who live in Jira never have to leave it.
• Linear (ENG-308): the assistant reads the ticket so the run has full context about what it's verifying, and links it to the run. Results land in the Slack thread and the run page.

This works from the app too — mention a ticket key in your instructions and it gets picked up the same way.

04Prefer a sandbox

Wherever you start from, the best place for an ad-hoc test to run is a sandbox: a fresh, disposable copy of your application provisioned just for this run.

Set a default once — Application Settings → Ad Hoc Testing → Default environment — and every ad-hoc test after that is zero-config: the start dialog preselects it, Slack uses it automatically, and PR tests already build their own sandbox from the branch.

Use Live deliberately, when the question is specifically about production — and give the agent a scoped credential if it needs to be signed in.

05What you get back

Every ad-hoc test — however you started it — lands in Runs → Ad-hoc Tests with the full picture:

• A verdict — passed, failed, blocked, or canceled, with a written summary
• Plan, Report, and Issues tabs — what the agent set out to cover, how each check went, and verified findings with evidence
• A recording, synced to the timeline — scrub to the exact moment something broke

Quick triage that works: read the summary, open the issues, watch the replay of the moment it failed. Two minutes, evidence in hand.

06Writing instructions that work

The agent is good, but it can't read minds. The difference between a great run and a wandering one is usually one sentence of specificity.

• Name the goal, not the clicks. "Verify a new user can complete checkout with a saved card" beats "click the cart icon, then…"
• Say what success looks like. "The order should appear in order history" gives the agent something to assert, not just attempt.
• One question per run. Testing checkout and password reset and the admin panel? That's three runs. Each gets a cleaner verdict.
• Start close to the action. A start URL deep-linked to the right page saves the agent a scavenger hunt.

If a run goes sideways, cancel it, sharpen the instructions, and go again — ad-hoc tests are cheap by design. And when you find yourself running the same check every week, that's your cue to promote it to a flow.

07Related

• Ad-hoc Tests reference — every field, state, and outcome in detail
• Sandboxes — templates, seeding, and login provisioning
• Slack · Jira · Linear — connecting the integrations used here
• Credentials & login — for live-environment runs