Docs/Bring your existing e2e suite
#

Bring your existing e2e suite

Run your own Playwright, Selenium or Cypress repo on Cofactor: one suite, one area per test folder, each on its own sandbox.

Already have a Playwright, Selenium or Cypress suite? Cofactor runs it for you: one suite for the repo, one area per test folder, each area on its own disposable sandbox, all at once. Setup is a handful of CLI commands — and it is exactly the kind of work an agent should do for you.

01What you get

  • Every area runs in parallel. Each area leases its own sandbox and runs at the same time as the others, so a suite that takes an hour end-to-end finishes in about as long as its slowest folder. Areas beyond your organization's concurrency limit wait their turn rather than failing.
  • Video and traces, captured for you. Each attempt records a video, and Playwright traces are collected and attached to the run. --capture-policy retry records only when a test is retried (cheap); always records every time. Nothing to wire up in your own config — download them with cofactor external-run artifacts download <run-id>, or watch the video inline on the run page.
  • AI diagnostics on failures. When a test fails, an agent investigates it: it steps through the recorded video frame by frame and reads the relevant code in your repo, then writes up what it thinks broke and why. You get a diagnosis attached to the failure instead of a stack trace and a guess.
  • Grouped results you can act on. The run page rolls the areas up, groups the tests under them, and lets you drill into one area or re-run just that area — no re-running 50 folders to retry one.

Don't want to read this? Copy this prompt and give it to your agent:

Onboard my existing test suite as a Cofactor external suite

Or read on to do it yourself.

02How it fits together

  • One template, one suite, many areas. A sandbox template built from your repo is the substrate. A run fans out one fresh instance per area (your code is cloned at provision), runs that folder inside the framework's container, and terminates the instance. Nothing is shared between areas, and they all run at once.
  • Areas are your folders. The suite's --shard-dirs list is the folder list; each entry shows up on the run page as an area with its own state, counts and duration, and the tests are grouped under it. Click an area to drill into it; Re-run area starts a fresh run of just that folder, linked back to the original.
  • Secrets stay in credentials. The suite's env vars are declared as a mapping; values come from a credential ($var.USER.X), an environment ($var.ENV.X) or a literal. The mapping is also the allow-list: only mapped variables reach the container.

03Before you start: connect GitHub

This guide covers the GitHub-connected path: Cofactor clones your suite from GitHub at the start of every run, so the repo must be linked to an application.

  1. In Settings → GitHub, install the Cofactor GitHub App on the repository that holds the suite.
  2. Link it: cofactor application repo link --application "<name>" --repo owner/name, or check with cofactor application list (linked repos are printed under each application).

04Step 1 — Build the sandbox template from the suite repo

bash
cofactor sandbox create --property "<application>" --name "<repo> tests"
cofactor sandbox list                     # note the template id (T) and the authoring instance (I)

You (or your agent) can read the suite through the authoring instance — useful when whoever is doing the setup doesn't have the repo checked out:

bash
cofactor sandbox run-command I "cat /workspace/package.json"
cofactor sandbox run-command I "ls /workspace/<tests-dir>"          # → the area list
cofactor sandbox run-command I "grep -rhoE 'process\.env\.[A-Z_]+' /workspace --exclude-dir=node_modules | sort -u"

Add a compose file with a single service on the framework's official image at the version your lockfile pins. The service only needs to exist and answer / and /health — the runner action starts your tests inside it. For Playwright 1.61.1:

yaml
services:
  playwright:
    image: mcr.microsoft.com/playwright:v1.61.1-noble
    working_dir: /workspace
    volumes:
      - .:/workspace
    command:
      - node
      - -e
      - >-
        require('node:http').createServer((req, res) => {
          const healthy = req.url === '/health' || req.url === '/';
          res.writeHead(healthy ? 200 : 404, { 'content-type': 'application/json' });
          res.end(JSON.stringify({ status: healthy ? 'ok' : 'not-found' }));
        }).listen(3000, '0.0.0.0')
    labels:
      sandbox.route: "/"
      sandbox.port: "3000"
    restart: unless-stopped
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
bash
cofactor sandbox artifact create T docker-compose ./compose.yml
cofactor sandbox build I
cofactor sandbox checklist complete T environment-analyzed --status success --summary "Test repo; env vars come from the suite's mapping"
cofactor sandbox checklist complete T docker-compose-created --status success --summary "Single playwright service + health stub"
for i in setup-script-created unit-tests-runnable dockerfile-created integration-tests-runnable seed-script-created adhoc-test-ready; do cofactor sandbox checklist skip T $i; done
cofactor sandbox verify T

The health stub must answer / as well as /health — verify probes the root.

05Step 2 — Add the runner action

The action is the adapter between Cofactor and your suite: Cofactor hands it the command to run and the environment to inject, it runs them inside the container, and it leaves results where Cofactor picks them up.

bash
cofactor sandbox guide external-test-runner      # the contract: reserved params + output config

That section is the authority. In short, every launch injects INPUT_RUNNER_COMMAND (the command to run), INPUT_RUNNER_LAUNCH_KEY (unique per attempt — name any container you start after it), INPUT_RUNNER_ENV_JSON (env vars to expose to the test process) and INPUT_RUNNER_TIMEOUT_MS. Don't declare those in your --input-schema; they're injected regardless. Your output config declares what Cofactor uploads when the script exits.

bash
cofactor sandbox action create T playwright-run-suite \
  --label "Playwright Run Suite" \
  --script ./playwright-run-suite.sh \
  --input-schema ./empty-object.json \
  --output-config ./playwright-output.json

Ask us for Cofactor's reference Playwright adapter if you'd rather start from a working one.

06Step 3 — Put secrets in a credential

bash
cofactor environment list --application "<application>"        # the environment name you'll target
cofactor credential list --application "<application>"         # or create one with the suite's secret as a variable

For example, a suite that logs in as a fixture user and reads a config key needs TEST_USER_PASSWORD (a secret, so it lives on a credential as a variable) and TEST_ENV (not a secret, so it is a literal in the mapping).

07Step 4 — One suite, one area per folder

bash
cofactor test-suite create --name "web-e2e/all" --kind external --provider playwright \
  --repo acme/web-e2e --ref main --environment Staging --credential e2e-user

cofactor test-suite update "web-e2e/all" \
  --sandbox-template T --sandbox-action playwright-run-suite \
  --env-map 'TEST_USER_PASSWORD=$var.USER.TEST_USER_PASSWORD' --env-map TEST_ENV=staging \
  --capture-policy retry --retry-granularity test --max-retries 2

The command that runs inside the container, and the area list, are set by Cofactor (we decide what executes in our network). Tell us the install command, the run command and the folders.

Playwright:

bash
cofactor test-suite update "web-e2e/all" \
  --install-cmd "npm ci && npx playwright install chromium" \
  --run-cmd "npx playwright test <spec> <cofactor-playwright-args>" \
  --shard-timeout-ms 3600000 \
  --shard-dirs tests/checkout,tests/search,tests/account

Selenium (Node + Mocha + mochawesome):

bash
cofactor test-suite update "web-e2e/all" \
  --install-cmd "npm ci --ignore-scripts && prepare-chromedriver" \
  --run-cmd "npx mocha --no-config --timeout 120000 --unhandled-rejections=strict --reporter mochawesome --reporter-options reportDir=\$OUT,reportFilename=mochawesome,json=true,html=false,overwrite=true <spec>" \
  --shard-timeout-ms 3600000 \
  --shard-dirs test/checkout,test/search,test/account

The placeholders are the important part:

  • <spec> — every provider. Cofactor substitutes the area's directory here, which is how one command runs 50 different folders.
  • <cofactor-playwright-args> — Playwright only. Cofactor injects its reporter and the capture flags here, so keep it even though it looks like a no-op.
  • $OUT — where your reporter must write its machine-readable report. Cofactor reads results from there; a suite that writes an HTML report and no JSON will run green with zero tests.

If your suite generates its specs (a BDD codegen step, for example), run the generator first in the same command and point --shard-dirs at the generated directories.

Three things to get right:

  • --shard-dirs entries are directories, comma-separated, relative to the repo root. Each becomes an area (labelled by its last segment) that runs on its own sandbox. For a generated tree (playwright-bdd), list the generated directories, not the source ones. The cap is 64.
  • --env-map replaces the whole mapping. Pass every variable each time you change one.
  • Only the flags you pass are changed. update --name never touches the runner.

cofactor test-suite suites lists what you have.

08Step 5 — Run it, then run one area

bash
cofactor test-suite run --suite "web-e2e/all"              # streams progress; exit code reflects the verdict
cofactor external-run get <run-id>                         # areas block: label · state · counts · re-runs
cofactor external-run failures <run-id>                    # failed/flaky scenarios with evidence
cofactor external-run rerun <run-id> --area checkout        # a fresh run of just that area, linked to this one

An area that finishes with zero tests is a configuration problem (wrong folder, missing secret, install failure), not a passing empty area — cofactor external-run debug overview <run-id> shows the action transcript.

Each area leases its own sandbox, so they run in parallel up to your organization's concurrency limit; areas beyond it wait their turn rather than failing. A mid-run item count of 0 is normal for Playwright — only failures stream, and passes land when the area finishes.

Schedule the suite with cofactor schedule suite create and route alerts with cofactor alert.

← PREVIOUS
Driving Cofactor from your coding agent