Stably - AI-native Testing Platform

There are two ways to run your Stably-powered Playwright tests:

Method 1: stably test (Recommended)
Method 2: npx playwright test

The simplest approach — the Stably reporter is automatically configured.

Setup Requirements

Set environment variables

export STABLY_API_KEY=your_api_key_here
export STABLY_PROJECT_ID=your_project_id_here

Run tests

stably test

Options

Stably is fully Playwright compatible. All Playwright CLI options are supported:

# Run all tests
stably test

# Run with Playwright options
stably test --headed --project=chromium

# Run specific test file
stably test tests/login.spec.ts

# More Playwright options
stably test --workers=4 --retries=2 --grep="login"

GitHub Actions Workflow

# .github/workflows/stably.yml
name: Stably Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Install browsers
        run: npx stably install

      - name: Run Stably tests
        env:
          STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
          STABLY_PROJECT_ID: ${{ secrets.STABLY_PROJECT_ID }}
        run: npx stably test

Use this if you need more control over your Playwright configuration.

Setup Requirements

Set environment variables

export STABLY_API_KEY=your_api_key_here
export STABLY_PROJECT_ID=your_project_id_here

Configure the reporter

Add the Stably reporter to your playwright.config.ts:

import { stablyReporter } from "@stablyai/playwright-test";

export default defineConfig({
  reporter: [
    ["list"],
    stablyReporter({
      apiKey: process.env.STABLY_API_KEY,
      projectId: process.env.STABLY_PROJECT_ID,
    }),
  ],
  use: {
    trace: "on", // Required for full context
  },
});

Run tests

npx playwright test

See the Stably Test Reporter documentation for complete setup instructions.

GitHub Actions Workflow

# .github/workflows/playwright.yml
name: Playwright Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Install browsers
        run: npx stably install

      - name: Run Playwright tests
        env:
          STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
          STABLY_PROJECT_ID: ${{ secrets.STABLY_PROJECT_ID }}
        run: npx playwright test

Running Specific Test Groups

If you’ve organized your tests using Playwright projects, you can run specific groups:

# Run smoke tests only
npx stably test --project=smoke

# Run multiple test groups
npx stably test --project=smoke --project=release

# Combine with grep for fine-grained control
npx stably test --project=critical --grep @p0

Environment Variables

There are several ways to pass environment variables to your tests:

# Inline variables
BASE_URL=http://localhost:3000 npx stably test

# Load from a named environment on Stably
npx stably test --env Staging

# Load from a local .env file
npx stably test --env-file .env.staging

# Combine sources
npx stably test --env-file .env --env Production

When using multiple sources, variables are resolved in this order (highest priority wins):

Stably internals (STABLY_API_KEY, STABLY_PROJECT_ID)
--env (remote environment from Stably)
--env-file (local .env files)
process.env (system/shell environment)

See Environments for managing named environments on the dashboard.

Advanced Options

Debug Mode

Run tests in debug mode with Playwright Inspector:

npx stably test --debug

Parallel Execution

Control the number of parallel workers:

# Run with 4 workers
npx stably test --workers=4

# Run serially (one test at a time)
npx stably test --workers=1

Retries

Configure test retries:

# Retry failed tests twice
npx stably test --retries=2

Headed Mode & UI Mode

Run with a visible browser or Playwright’s interactive UI:

# See the browser as tests run
npx stably test --headed

# Open Playwright's interactive UI
npx stably test --ui

Next Steps

Auto-fix Failures

Automatically diagnose and repair failing tests

Defining Test Groups

Organize tests using Playwright projects

Test Reporter

Stream results and traces to the Stably dashboard

CI/CD Integration

Integrate tests into your deployment pipeline

Getting started

Stably Web Editor

Stably CLI

Stably Cloud Runner

Stably SDK

Stably Test Reporter

Stably Classic

Configuration

Troubleshooting

Run Tests

Setup Requirements

Options

Setup Requirements

Running Specific Test Groups

Environment Variables

Advanced Options

Next Steps

Auto-fix Failures

Defining Test Groups

Test Reporter

CI/CD Integration

Getting started

Stably Web Editor

Stably CLI

Stably Cloud Runner

Stably SDK

Stably Test Reporter

Stably Classic

Configuration

Troubleshooting

​Setup Requirements

​Options

​Setup Requirements

​Running Specific Test Groups

​Environment Variables

​Advanced Options

​Next Steps

Auto-fix Failures

Defining Test Groups

Test Reporter

CI/CD Integration

Setup Requirements

Options

Setup Requirements

Running Specific Test Groups

Environment Variables

Advanced Options

Next Steps