QA Testing

Previewops can run an automated QA analysis on any pull request with a single comment. You get a ready-to-read report posted directly in the PR — no CI configuration required.

Quick start

Run on demand

/qa-previewops

Previewops queues the analysis and posts a confirmation comment immediately. The full report appears in the PR within a few minutes. If a preview environment is already live for this PR, HTTP smoke tests run against it automatically.

Run against a specific URL

/qa-previewops --url=https://pr-42-preview.example.com

Use --url to target a preview that was deployed outside of Previewops, or to override the auto-detected URL.

What's in the report

The report is always posted as a PR comment and is organised into sections:

📁 Test coverage

Available on all plans — no AI key required.

Previewops inspects up to 300 changed files and maps each source file to its expected test counterpart. The result is a table showing which files are covered (a test file was found), uncovered (no test file could be matched), and — for covered files — the name of the matched test file.

Supported languages:

A fuzzy fallback handles cases where the test file name doesn't exactly mirror the source file name — for example, provider subdirectories (aws-ecs/index.ts matched to ecs.test.ts) or cross-root test layouts (Java src/main/ ↔ src/test/, Ruby app/ ↔ spec/).

Files that are not mappable are silently excluded from the coverage table. This includes: configuration files (*.config.js, tsconfig*.json, .eslintrc.*, etc.), migrations, Markdown, JSON, and files under build-output or static-asset directories (public/, dist/, build/, out/, .next/).

📝 Code review

Available on Pro, Enterprise, and Custom plans, and on Premium BYOC with the Custom LLM add-on activated.

An AI model reviews the full PR diff and produces:

• A 1–3 sentence summary of what changed.
• A table of flagged issues, each rated high / medium / low severity.

The review covers any language in the diff.

🧪 HTTP smoke tests

Available on Pro, Enterprise, and Custom plans, and on Premium BYOC with the Custom LLM add-on activated. Requires a live preview URL.

The AI generates 3–6 HTTP test scenarios based on the PR diff, then executes them against the preview. Each scenario specifies a method, path, and expected status code. Results are shown in a pass/fail table.

If the AI determines the diff contains no HTTP endpoints (e.g. the PR only touches test files, configuration, or documentation), the HTTP test section is omitted automatically.

If no preview URL is available (neither via --url nor an active preview for this PR), the HTTP test section is omitted.

⚠️ Regression risks

Available on Pro, Enterprise, and Custom plans, and on Premium BYOC with the Custom LLM add-on activated.

The AI identifies areas of the codebase that may regress based on what the PR touches — for example, auth changes raising risk for all authenticated routes. Each risk is rated high / medium / low with a brief explanation.

Plan comparison

¹ Premium BYOC AI features require the Custom LLM add-on. Activate it in the dashboard and store your LLM_PROVIDER, LLM_API_KEY, and LLM_MODEL credentials. Without it, the command runs and returns test coverage, but AI analysis is skipped and an add-on activation prompt is shown in the report. See add-ons.md. You may also set LLM_MAX_PATCH_CHARS (integer, 100–10000) to control how many characters of each file's diff are sent to the model — useful for large files with many route definitions.

Free plan users will see an upgrade prompt in the report where the AI sections would appear.

Example report

> 🕵️ QA Report for PR #42

📋 **2/3** files covered · **1** issue · **3/3** HTTP tests passed · **2** risks

### 📁 Test Coverage — 2/3 files covered

| File | Status | Test file |
|---|---|---|
| `src/utils/auth.ts` | ✅ covered | `auth.test.ts` |
| `src/utils/rate-limit.ts` | ✅ covered | `rate-limit.test.ts` |
| `src/deployer/index.ts` | ❌ uncovered |  |

**2/3** source files have corresponding test files.

---

### 📝 Code Review — 1 issue

The changes introduce a new rate-limiting layer in the auth utility and refactor
the deploy orchestrator to support concurrent provider calls.

| Severity | Issue |
|---|---|
| 🟡 medium | `rateLimiter` is not reset between test runs — consider clearing state in teardown. |

---

### 🧪 HTTP Test Scenarios — 3/3 passed

| Scenario | Status | Details |
|---|---|---|
| GET /api/health | ✅ pass | HTTP 200 |
| POST /api/deploy | ✅ pass | HTTP 202 |
| GET /api/unknown-route | ✅ pass | HTTP 404 (expected 404) |

---

### ⚠️ Regression Risks — 2 areas

| Area | Risk | Reason |
|---|---|---|
| Auth middleware | 🟡 medium | Rate-limiting logic now wraps all authenticated routes. |
| Deploy orchestrator | 🔴 high | Concurrent provider path is new and lacks tests. |

---

<details>
<summary>🔧 Test Setup</summary>

- **Preview:** [https://pr-42-preview.fly.dev](https://pr-42-preview.fly.dev)
- **Model:** gpt-4o
- **Files analyzed:** 3

</details>

Options

Playwright draft generation

Available on Premium BYOC, Pro, Enterprise, and Custom plans with the QA Hub add-on activated.

When you pass --generate-drafts, Previewops uses the LLM to write a Playwright .spec.ts script for each of the repo's active QA scenarios, then saves the drafts in the QA Hub under Generated Tests with status draft.

Requirements

• QA Hub add-on enabled for your organisation.
• A live preview URL (auto-detected or supplied via --url).
• At least one active QA scenario defined for the repo in the dashboard.
• An LLM configured (same requirement as AI code review).

How it works

1. Scenarios are sorted P0-first. Up to 5 are drafted per run (override with --draft-max).
2. An idempotency check runs for each scenario — if a draft already exists for this scenario at the current commit SHA, it is skipped to avoid duplicates.
3. The LLM generates a Playwright script following stability best practices: data-testid locators, waitForLoadState() instead of waitForTimeout(), and a 30-second cold-start timeout on the first goto() call for preview environments.
4. Each generated script is saved as status: draft in the QA Hub. No script is executed automatically.
5. The QA report footer shows how many drafts were generated with a link to the QA Hub.

Selecting which scenarios to draft

By default, Previewops drafts up to 5 scenarios scoped to the current repo (plus global/unscoped scenarios), sorted P0-first. Use the selector flags to narrow or change this:

# Only draft P0 smoke scenarios, up to 10
/qa-previewops --generate-drafts --draft-priorities=P0 --draft-categories=smoke --draft-max=10

# Draft a specific pair of scenarios by ID
/qa-previewops --generate-drafts --draft-scenario-ids=uuid1,uuid2

# Include scenarios from all repos in the org (not just this repo)
/qa-previewops --generate-drafts --draft-scope=all --draft-max=8

All selector flags are optional and additive — you can combine any of them. The default behaviour (5 drafts, repo scope, all categories and priorities) is preserved when no selector flags are provided.

No-preview mode

By default, draft generation requires a live preview URL because the LLM uses the URL to generate accurate goto() calls and assertions. When no preview is available, you can still generate structural drafts:

/qa-previewops --generate-drafts --draft-no-preview --draft-priorities=P0

Scripts generated in no-preview mode contain https://PREVIEW_URL_PLACEHOLDER as the target URL. Replace this value before running the test. The Generated Tests card in the QA Hub shows a warning badge on scripts generated without a preview.

Note: No-preview mode is useful for seeding your test library before a preview environment has been deployed, but scripts should be reviewed and the placeholder replaced before they are approved for automated runs.

Reviewing and approving drafts

Open the QA Hub in the dashboard (QA → Generated Tests). Each draft can be:

• Approved — moves to awaiting-approval and is queued for a test run against the preview URL.
• Rejected — removed from the queue; re-running /qa-previewops --generate-drafts on a new commit generates a fresh draft.
• Edited — the script is shown in a code editor before approval.

Controlling draft generation via .previewops.yaml

The qa.generateDrafts field lets you set a default without requiring the CLI flag each time:

qa:
  # 'auto' (default): uses the org-level dashboard setting.
  # true: always generate drafts when the billing gate is met.
  # false: never generate drafts — overrides the CLI flag and org setting.
  generateDrafts: auto

The full precedence order (highest wins):

1. qa.generateDrafts: false in .previewops.yaml → always off (cannot be overridden by the CLI flag)
2. --generate-drafts CLI flag → on (subject to billing gate)
3. qa.generateDrafts: true in .previewops.yaml → on (subject to billing gate)
4. Org-level dashboard setting → on/off (subject to billing gate)
5. Default → off

Who can trigger the command

Any collaborator with Write, Maintain, or Admin permission on the repository. Comments from users with read-only access are silently ignored.

Frequently asked questions

Does QA testing require a deployed preview?

No. The test coverage mapping and AI code review run on the PR diff — no live URL needed. HTTP smoke tests require a preview URL, but if one isn't available that section is simply omitted.

What happens if I run /qa-previewops again without pushing a new commit?

Previewops detects the existing QA comment for that commit and updates it in place rather than posting a new one. You won't get duplicate comments.

Does QA post a commit status / Check Run?

Yes — after posting the report, Previewops creates a GitHub Check Run named previewops / QA on the PR head commit. The check shows failure if any high-severity issue was found, and success otherwise. I have multiple active previews for this PR. Which one does QA test against?

By default, Previewops uses the most recently deployed active preview for the PR. To target a specific one, use --provider=<key> (e.g. --provider=render) or --env=<name> to match by environment name. You can also combine both flags. If no preview matching your filter is found, the AI analysis and coverage report still run — only the HTTP smoke test section is skipped. Which AI model is used?

The model name is shown in the 🔧 Test Setup collapsible at the bottom of every report. For Pro, Enterprise, and Custom plan users, the model is determined by the operator's Previewops configuration. For Premium BYOC users with the Custom LLM add-on, it reflects the exact model you selected when saving your credentials — for example gpt-4-turbo or claude-sonnet-4-5. Contact your team's Previewops administrator or see plans.md to learn more.

Why is my file shown as uncovered when I know a test exists?

The mapping uses the test files that exist at the PR head. If your test file was added in the same PR and is already committed, it should be detected. If the test file uses an unusual naming convention or lives in an unexpected location, it may fall through the fuzzy fallback. Primary mapping conventions are listed in the Test coverage table above.

What happens if the AI is temporarily unavailable?

Previewops retries the AI call automatically before falling back. If all attempts fail, the code review section shows a short error message and the rest of the report (coverage, HTTP tests if applicable) is still posted.

Is the diff sent to a third-party AI service?

Yes — when AI features are enabled, the PR diff is sent to the configured LLM provider (OpenAI or Anthropic). File patches are truncated to 2 000 characters each by default (operators can raise this up to 10 000 via LLM_MAX_PATCH_CHARS), and no repository access tokens or credentials are included in the payload. Self-hosted operators control which provider is used. Contact your Previewops administrator for details.

Related

• qa-hub.md — Dashboard-based Playwright test library, suite management, and runs against preview URLs (Pro / Enterprise / Custom)
• commands.md — Full command reference including /qa-previewops options
• plans.md — Plan comparison and AI feature access
• performance-testing.md — Run k6 load tests against previews with /perf-previewops