Stably - AI-native Testing Platform

There are two ways to run your Stably-powered Playwright tests:

Method 1: stably test (Recommended)
Method 2: npx playwright test

The simplest approach — the Stably reporter is automatically configured.

Setup Requirements

Set environment variables

export STABLY_API_KEY=your_api_key_here
export STABLY_PROJECT_ID=your_project_id_here

Run tests

stably test

Options

Stably is fully Playwright compatible. All Playwright CLI options are supported:

# Run all tests
stably test

# Run with Playwright options
stably test --headed --project=chromium

# Run specific test file
stably test tests/login.spec.ts

# More Playwright options
stably test --workers=4 --retries=2 --grep="login"

# Override the reported suite name
stably test --suiteName="nightly smoke tests"

`--suiteName`

Sets the suite name reported to the Stably dashboard and Slack notifications.

stably test --suiteName="daily staging run"
stably test --suiteName="nightly smoke tests" --project=chromium

When --suiteName is not provided, the default name is automatically derived from the Playwright project names that are running (sorted alphabetically, joined with , ). For example, running --project=chromium --project=firefox produces a default suite name of chromium, firefox.You can also set the suite name via the STABLY_SUITE_NAME environment variable, or directly in your Playwright config with stablyReporter({ suiteName: "..." }).

GitHub Actions Workflow

# .github/workflows/stably.yml
name: Stably Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Install browsers
        run: npx stably install

      - name: Run Stably tests
        env:
          STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
          STABLY_PROJECT_ID: ${{ secrets.STABLY_PROJECT_ID }}
        run: npx stably test

Use this if you need more control over your Playwright configuration.

Setup Requirements

Set environment variables

export STABLY_API_KEY=your_api_key_here
export STABLY_PROJECT_ID=your_project_id_here

Configure the reporter

Add the Stably reporter to your playwright.config.ts:

import { stablyReporter } from "@stablyai/playwright-test";

export default defineConfig({
  reporter: [
    ["list"],
    stablyReporter({
      apiKey: process.env.STABLY_API_KEY,
      projectId: process.env.STABLY_PROJECT_ID,
    }),
  ],
  use: {
    trace: "on", // Required for full context
  },
});

Run tests

npx playwright test

See the Stably Test Reporter documentation for complete setup instructions.

GitHub Actions Workflow

# .github/workflows/playwright.yml
name: Playwright Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Install browsers
        run: npx stably install

      - name: Run Playwright tests
        env:
          STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
          STABLY_PROJECT_ID: ${{ secrets.STABLY_PROJECT_ID }}
        run: npx playwright test

Running Specific Test Groups

If you’ve organized your tests using Playwright projects, you can run specific groups:

# Run smoke tests only
npx stably test --project=smoke

# Run multiple test groups
npx stably test --project=smoke --project=release

# Combine with grep for fine-grained control
npx stably test --project=critical --grep @p0

Environment Variables

There are several ways to pass environment variables to your tests:

# Inline variables
BASE_URL=http://localhost:3000 npx stably test

# Load from a named environment on Stably
npx stably --env Staging test

# Load from a local .env file
npx stably test --env-file .env.staging

# Combine sources
npx stably --env Production test --env-file .env

When using multiple sources, variables are resolved in this order (highest priority wins):

Stably internals (STABLY_API_KEY, STABLY_PROJECT_ID)
process.env (system/shell environment)
--env-file (local .env files)
--env (remote environment from Stably)

See Environments for managing named environments on the dashboard — you can get started quickly by uploading a .env file via Bulk Import.

Advanced Options

Debug Mode

Run tests in debug mode with Playwright Inspector:

npx stably test --debug

Parallel Execution

Control the number of parallel tests:

# Run with 4 parallel tests
npx stably test --workers=4

# Run serially (one test at a time)
npx stably test --workers=1

Retries

Configure test retries:

# Retry failed tests twice
npx stably test --retries=2

Headed Mode & UI Mode

Run with a visible browser or Playwright’s interactive UI:

# See the browser as tests run
npx stably test --headed

# Open Playwright's interactive UI
npx stably test --ui

Next Steps

Auto-fix Failures

Automatically diagnose and repair failing tests

Defining Test Groups

Organize tests using Playwright projects

Test Reporter

Stream results and traces to the Stably dashboard

CI/CD Integration

Integrate tests into your deployment pipeline

Getting started

Stably Web Editor

Stably CLI

Stably Cloud Runner

Stably SDK

Stably Test Reporter

Configuration

Troubleshooting

Migration Guides

Resources

Run Tests (stably run)

Setup Requirements

Options

`--suiteName`

Setup Requirements

Running Specific Test Groups

Environment Variables

Advanced Options

Next Steps

Auto-fix Failures

Defining Test Groups

Test Reporter

CI/CD Integration

​Setup Requirements

​Options

​--suiteName

​Setup Requirements

​Running Specific Test Groups

​Environment Variables

​Advanced Options

​Next Steps

Auto-fix Failures

Defining Test Groups

Test Reporter

CI/CD Integration

Setup Requirements

Options

`--suiteName`

Setup Requirements

Running Specific Test Groups

Environment Variables

Advanced Options

Next Steps