QA Testing

Previewops can run an automated QA analysis on any pull request with a single comment. You get a ready-to-read report posted directly in the PR — no CI configuration required.

Quick start

Run on demand

/qa-previewops

Previewops queues the analysis and posts a confirmation comment immediately. The full report appears in the PR within a few minutes. If AI analysis is enabled and a preview environment is already live for this PR, Previewops also runs any HTTP smoke tests generated for endpoints found in the selected PR changes.

A PR only ever carries one QA comment. Each run removes the previous one — including the "QA analysis queued" confirmation — and posts a fresh one at the bottom of the thread, so the current report is always the newest comment rather than buried in the history.

Run against a specific URL

/qa-previewops --url=https://pr-42-preview.example.com

Use --url to target a preview that was deployed outside of Previewops, or to override the auto-detected URL.

What's in the report

The report is always posted as a PR comment and is organised into sections:

📖 Summary

Available on all plans — no AI key required.

A collapsed walkthrough of the diff, aimed at whoever opens the PR cold. It stays collapsed so the findings below it remain visible without scrolling, and the headline sits in the disclosure label so the most useful sentence is readable while collapsed:

📖 Summary — 42 source files, +6,930 −93, mostly in ATS board indexing

Opened, it builds up in order of usefulness rather than dumping a file list:

The exhaustive per-file inventory and the excluded-file list stay behind their own nested disclosures, so they remain available without dominating the section.

With an AI key, the PR summary and review order are written by the model, which orders the changed files by consequence — the file that sets the blast radius first (a shared helper or data model before its callers), each with a sentence on why it matters at that point.

Without an AI key — or when AI analysis is unavailable — every factual section above still renders, derived from the diff alone. The review order falls back to file-role convention (models, then persistence, then core logic, then providers, then entry points) and is labelled as such, because it reflects naming convention rather than real dependency analysis. On a repository whose filenames carry little meaning (index.ts, utils.ts, helpers.ts), the name-derived sections are suppressed rather than guessed at, and only the counts and tables are shown.

If a PR changes no source files at all — documentation, lockfiles or config only — the section says so in one line instead of walking through nothing.

To get the most from Author notes, use headings in your PR description:

## Testing
Ran the integration suite against staging.

## Rollback
Revert the release tag; no migration to undo.

📁 Test coverage

Available on all plans — no AI key required.

Previewops inspects up to 300 changed files and maps each source file to its expected test counterpart. The result is a table showing which files are covered (a test file was found), uncovered (no test file could be matched), and — for covered files — the full path of the matched test file, so you can check the pairing rather than take it on trust.

Supported languages:

A fuzzy fallback handles cases where the test file name doesn't exactly mirror the source file name — for example, provider subdirectories (aws-ecs/index.ts matched to ecs.test.ts) or cross-root test layouts (Java src/main/ ↔ src/test/, Ruby app/ ↔ spec/).

Where a single test file covers a whole directory of small modules, a test named after the directory counts too — src/parsers/csv.py is covered by tests/test_parsers.py. The test's filename has to be exactly what the language's convention would produce for that directory name, and a test named after the file itself always takes precedence. Directory names too nondescript to name a test after (src/, lib/, utils/, app/, and similar) are not considered, because a test_utils.py is more often a shared helper than a test of utils/.

In a monorepo, matching is scoped to the application a file belongs to. If apps/web/ and apps/api/ each have their own test directory, a test under apps/api/tests/ is never counted as covering a file in apps/web/ — and an application with no test directory of its own is reported uncovered rather than being credited to a sibling's tests. Repository-wide test directories (a top-level tests/ or spec/) still apply everywhere, and layouts that deliberately keep tests elsewhere — Maven's src/test/java, Cypress's cypress/e2e — are unaffected.

Files that are not mappable are silently excluded from the coverage table. This includes: configuration files (*.config.js, tsconfig*.json, .eslintrc.*, etc.), migrations, Markdown, JSON, and files under build-output or static-asset directories (public/, dist/, build/, out/, .next/).

📝 Code review

Available on Pro, Enterprise, and Custom plans, and on Free and Premium BYOC with the Custom LLM add-on activated.

An AI model reviews selected and truncated PR changes and produces:

• A 1–3 sentence summary of what changed.
• A table of flagged issues, each rated high / medium / low severity, tagged with a category, and — where the model can identify one confidently — the file and line the finding is in.

The review covers any language in the diff.

Treat severity as the model's opinion, not a verdict. It is the least consistent part of an AI review: on one test pull request the same genuine vulnerability was rated high, medium and low by different models, and twice differently by the same model. We tried anchoring it with an explicit rubric and measured the result — it did not improve consistency, and on one model it systematically lowered ratings, so it was removed rather than shipped. This is why a high-severity finding no longer fails the Check Run. Read the finding, not the dot.

Categories are one of security, correctness, data-loss, performance, maintainability, test-coverage or dependency. A count by category appears above the issues table. A finding the model didn't classify is left out of that count rather than guessed at.

🧪 HTTP smoke tests

Available on Pro, Enterprise, and Custom plans, and on Free and Premium BYOC with the Custom LLM add-on activated. Requires a live preview URL.

The AI generates 3–6 HTTP test scenarios based on the PR diff, then executes them against the preview. Each scenario specifies a method, path, and expected status code. Results are shown in a pass/fail table.

If the AI determines the diff contains no HTTP endpoints (e.g. the PR only touches test files, configuration, or documentation), the HTTP test section is omitted automatically.

If no preview URL is available (neither via --url nor an active preview for this PR), the HTTP test section is omitted.

⚠️ Regression risks

Available on Pro, Enterprise, and Custom plans, and on Free and Premium BYOC with the Custom LLM add-on activated.

The AI identifies areas of the codebase that may regress based on what the PR touches — for example, auth changes raising risk for all authenticated routes. Each risk is rated high / medium / low with a brief explanation.

Plan comparison

¹ Free and Premium BYOC AI features require the free Custom LLM add-on. Activate it in the dashboard and store your LLM_PROVIDER, LLM_API_KEY, and LLM_MODEL credentials. Without it, the command returns test coverage only; AI review, generated HTTP smoke tests, and regression-risk analysis are skipped, no AI QA credit is consumed, and an activation prompt appears in the report. Free BYOC includes 50 AI-powered QA runs per month. See add-ons.md. You may also set LLM_MAX_PATCH_CHARS (integer, 100–10000) to control the per-file patch limit sent to the model.

Example report

> 🕵️ QA Report for PR #42

📋 **2/3** files covered · **1** issue · **3/3** HTTP tests passed · **2** risks

<details>
<summary>📖 Summary — 3 source files, +214 −31, mostly in src/utils</summary>

#### PR summary

Adds a rate-limiting layer to the auth utility and refactors the deploy
orchestrator for concurrent provider calls.

#### Suggested reading order

1. **`src/utils/rate-limit.ts`**

   Start here — the new limiter's semantics decide how every authenticated
   route behaves below.

2. **`src/utils/auth.ts`**

   Wires the limiter into the auth path. Check the failure mode when the
   limiter rejects.

3. **`src/deployer/index.ts`**

   Independent of the above; adds the concurrent provider path.

#### Change size

| Area | Files | Change | Largest |
|---|---|---|---|
| src/utils | 2 | +166 −12 | `rate-limit.ts` (+140), `auth.ts` (+26 −12) |
| src/deployer | 1 | +48 −19 | `index.ts` (+48 −19) |

#### Review status

AI analysis was applied to this PR. Findings appear in the sections below.

---

<details>
<summary>Complete file inventory (3 source files)</summary>
…
</details>

<details>
<summary>Excluded from source walkthrough (1 file — 1 docs/config)</summary>
…
</details>

</details>

### 📁 Test Coverage — 2/3 files covered

| File | Status | Test file |
|---|---|---|
| `src/utils/auth.ts` | ✅ covered | `src/__tests__/utils/auth.test.ts` |
| `src/utils/rate-limit.ts` | ✅ covered | `src/__tests__/utils/rate-limit.test.ts` |
| `src/deployer/index.ts` | ❌ uncovered |  |

---

### 📝 Code Review — 1 issue

The changes introduce a new rate-limiting layer in the auth utility and refactor
the deploy orchestrator to support concurrent provider calls.

| Severity | Issue |
|---|---|
| 🟡 medium | `rateLimiter` is not reset between test runs — consider clearing state in teardown. |

---

### 🧪 HTTP Test Scenarios — 3/3 passed

| Scenario | Status | Details |
|---|---|---|
| GET /api/health | ✅ pass | HTTP 200 |
| POST /api/deploy | ✅ pass | HTTP 202 |
| GET /api/unknown-route | ✅ pass | HTTP 404 (expected 404) |

---

### ⚠️ Regression Risks — 2 areas

| Area | Risk | Reason |
|---|---|---|
| Auth middleware | 🟡 medium | Rate-limiting logic now wraps all authenticated routes. |
| Deploy orchestrator | 🔴 high | Concurrent provider path is new and lacks tests. |

---

<details>
<summary>🔧 Test Setup</summary>

- **Preview:** [https://pr-42-preview.fly.dev](https://pr-42-preview.fly.dev)
- **Model:** gpt-5.6
- **Diff sent to model:** 3 of 3 changed files

</details>

How much of the diff is reviewed

The AI review works to a token budget, so on a large pull request it does not read every changed file. Files are prioritised — application source over tests, larger changes over smaller ones, and generated output (lockfiles, dist/, vendor/, minified bundles) excluded entirely — and the highest-priority slice that fits the budget is what the model sees. That is roughly 26 files on a large PR.

The report always tells you the real scope, in 🔧 Test Setup:

- **Diff sent to model:** 26 of 76 changed files _(diff truncated)_

To review everything, either pass the flag for a single run:

/qa-previewops --full-diff

or turn it on for the repository in .previewops.yaml:

qa:
  fullDiff: true

Setting qa.fullDiff: false explicitly is an absolute deny — --full-diff will not override it. Full-diff review multiplies the work behind a run, so the repository owner keeps the final say, the same way qa.generateDrafts: false works.

The reviewable files are then split across up to 3 passes and the findings merged, with duplicate findings collapsed and the higher severity kept. The footer says how many passes ran:

- **Diff sent to model:** 74 of 76 changed files across 3 passes _(diff truncated)_

Worth knowing before you turn it on:

• It still costs one QA run. Billing is per run, not per pass.
• It is slower — three sequential model calls rather than one.
• Very large PRs can still be truncated. Three passes is a hard ceiling; beyond that the footer keeps saying diff truncated, honestly.
• If a later pass fails (a rate limit, say), the findings from the passes that succeeded are still reported, and the file count reflects what was actually reviewed rather than what was planned.
• Regression risks come from the first pass, and are not merged across passes. They are a judgement about the pull request as a whole rather than about one slice of it, so combining three passes' worth produced the same risk listed more than once. Code issues are merged from every pass — they are anchored to a file, so they can be reconciled reliably.

Coverage analysis is unaffected by this setting — it reads the repository's test tree directly and is not subject to the prompt budget.

Options

Playwright draft generation

Available through QA Hub on managed Pro, Enterprise, and Custom plans. QA Hub is not available on Free or Premium BYOC.

When you pass --generate-drafts, Previewops uses the LLM to write a Playwright .spec.ts script for each of the repository's active QA scenarios, then saves the drafts in the QA Hub under Generated Tests with status draft.

Requirements

• QA Hub access on a managed Pro, Enterprise, or Custom plan.
• A live preview URL (auto-detected or supplied via --url), unless --draft-no-preview is used.
• At least one active QA scenario defined for the repository in the dashboard.
• An LLM configured (same requirement as AI code review).

How it works

1. Scenarios are sorted P0-first. Up to 5 are drafted per run (override with --draft-max).
2. An idempotency check runs for each scenario — if a draft already exists for this scenario at the current commit SHA, it is skipped to avoid duplicates.
3. The LLM generates a Playwright script following stability best practices: data-testid locators, waitForLoadState() instead of waitForTimeout(), and a 30-second cold-start timeout on the first goto() call for preview environments.
4. Each generated script is saved as status: draft in the QA Hub. No script is executed automatically.
5. The QA report footer shows how many drafts were generated with a link to the QA Hub.

Selecting which scenarios to draft

By default, Previewops drafts up to 5 scenarios scoped to the current repository (plus global/unscoped scenarios), sorted P0-first. Use the selector flags to narrow or change this:

# Only draft P0 smoke scenarios, up to 10
/qa-previewops --generate-drafts --draft-priorities=P0 --draft-categories=smoke --draft-max=10

# Draft a specific pair of scenarios by ID
/qa-previewops --generate-drafts --draft-scenario-ids=uuid1,uuid2

# Include scenarios from all repositories in the organisation
/qa-previewops --generate-drafts --draft-scope=all --draft-max=8

All selector flags are optional and additive — you can combine any of them. The default behaviour (5 drafts, repository scope, all categories and priorities) is preserved when no selector flags are provided.

No-preview mode

By default, draft generation requires a live preview URL. When no preview is available, you can still generate structural drafts:

/qa-previewops --generate-drafts --draft-no-preview --draft-priorities=P0

Generated scripts use relative paths such as page.goto('/login'); the Playwright runner supplies the base URL. Before running a no-preview draft, select or provide a live target environment. The Generated Tests card in the QA Hub shows a warning badge on scripts generated without a preview.

Note: No-preview mode is useful for seeding your test library before a preview environment has been deployed. Review the generated script and configure its target environment before running it.

Reviewing and approving drafts

Open the QA Hub in the dashboard (QA → Generated Tests). Each draft can be:

• Approved — marks the generated test as approved and supersedes any previously approved version for the same scenario. Approval does not itself queue a run.
• Rejected — marks the generated test as rejected; running /qa-previewops --generate-drafts on a new commit can generate a fresh draft.
• Edited — opens the script in a code editor. Changing an approved script returns it to awaiting-approval so the updated version can be reviewed again.

Controlling draft generation via .previewops.yaml

The qa.generateDrafts field lets you set a default without requiring the CLI flag each time:

qa:
  # 'auto' (default): uses the org-level dashboard setting.
  # true: always generate drafts when the billing gate is met.
  # false: never generate drafts — overrides the CLI flag and org setting.
  generateDrafts: auto

The full precedence order (highest wins):

1. qa.generateDrafts: false in .previewops.yaml → always off (cannot be overridden by the CLI flag)
2. --generate-drafts CLI flag → on (subject to billing gate)
3. qa.generateDrafts: true in .previewops.yaml → on (subject to billing gate)
4. Org-level dashboard setting → on/off (subject to billing gate)
5. Default → off

Who can trigger the command

Any collaborator with Write, Maintain, or Admin permission on the repository. Comments from users with read-only access are silently ignored.

Frequently asked questions

Does QA testing require a deployed preview?

No. The test coverage mapping and AI code review run on the PR diff — no live URL needed. HTTP smoke tests require a preview URL, but if one isn't available that section is simply omitted.

What happens if I run /qa-previewops again without pushing a new commit?

Previewops removes the previous QA report comment and posts a fresh report at the bottom of the PR conversation. This keeps the latest result close to the command that triggered it without leaving duplicate QA reports.

Does QA post a commit status / Check Run?

Yes — after posting the report, Previewops creates a GitHub Check Run named previewops / QA on the PR head commit:

A high-severity finding used to conclude failure and turn the check red. It no longer does. Severity is the least stable part of an AI review — on one test PR the same genuine vulnerability was rated high, medium and low by different models, and twice differently by the same model — and a red check is a blocking signal that shouldn't rest on a label that moves between runs. neutral still refuses to look like a pass, so a finding is never hidden; it just doesn't block. Previewops does not currently fail a check on its own AI findings at all.

I have multiple active previews for this PR. Which one does QA test against?

By default, Previewops uses the most recently deployed active preview for the PR. To target a specific one, use --provider=<key> (e.g. --provider=render) or --env=<name> to match by environment name. You can also combine both flags. If no preview matching your filter is found, the AI analysis and coverage report still run — only the HTTP smoke test section is skipped.

Which AI model is used?

The model name is shown in the 🔧 Test Setup collapsible at the bottom of every AI-powered report. For Pro, Enterprise, and Custom plans, the model is determined by the operator's Previewops configuration. For Free and Premium BYOC with the Custom LLM add-on, it reflects the model configured with your OpenAI, Anthropic, or Gemini credentials. Contact your team's Previewops administrator or see plans.md to learn more.

Why is my file shown as uncovered when I know a test exists?

The mapping uses the test files that exist at the PR head. If your test file was added in the same PR and is already committed, it should be detected. If the test file uses an unusual naming convention or lives in an unexpected location, it may fall through the fuzzy fallback. Primary mapping conventions are listed in the Test coverage table above.

The matcher deliberately errs towards reporting uncovered. It requires a test name to match your source file as a whole word, not as a fragment — cli.py is not treated as covered by test_clients.py just because "cli" appears inside "clients". A coverage figure that is quietly inflated by pairings that don't exist is worse than one that is a little conservative, so if you see a file you believe is covered, check whether the test's name actually contains the file's name or its directory's name.

What happens if the AI is temporarily unavailable?

Previewops retries the AI call automatically, and where several LLM credentials are stored it tries each in turn before giving up. If all attempts fail, the report still includes test coverage and the structural diff summary, but AI review, regression-risk analysis, and AI-generated HTTP smoke tests are unavailable for that run.

A run that produced no AI analysis does not consume an AI QA credit — it delivers the same output as --coverage-only, which is free, so the credit is returned and no metered overage is billed. The Code Review section is headed unavailable rather than "no issues", and the Check Run concludes neutral, so a failed analysis is never mistaken for a clean pass.

If a credential fails authentication or has exhausted its quota, Previewops marks it inactive and skips it on later runs until you re-save or re-test it in the dashboard — so a dead key stops costing you a run every push.

Is the diff sent to a third-party AI service?

Yes. Depending on the operation, Previewops sends the configured OpenAI, Anthropic, or Gemini provider the PR title, up to 500 characters of the PR body, selected and truncated file patches, and relevant QA scenario details. Playwright draft generation can also send scenario preconditions, test steps, and a limited amount of repository source or diff context.

Previewops does not intentionally include repository access tokens or configured credential values in these prompts. However, secrets or restricted customer data written directly into PR titles, descriptions, diffs, source files, or QA scenarios may be included. Do not run AI features on content that violates your organisation's data-handling policy, and never commit credentials or secrets to a repository.

File patches are limited to 3,000 characters each by default; operators can set LLM_MAX_PATCH_CHARS from 100 to 10,000. Additional aggregate and file-count limits also apply, so the provider does not necessarily receive every changed file or the complete diff. Self-hosted operators control which provider is used. Contact your Previewops administrator for details.

Related

• qa-hub.md — Dashboard-based Playwright test library, suite management, and runs against preview URLs (Pro / Enterprise / Custom)
• commands.md — Full command reference including /qa-previewops options
• plans.md — Plan comparison and AI feature access
• performance-testing.md — Run k6 load tests against previews with /perf-previewops