Stably AI - AI-Powered Testing Platform Documentation

Overview

The Stably CLI is a command-line tool for developers who prefer working in the terminal. It provides essential commands for authentication, resource creation, test execution, and the powerful stably fix command for automatic test maintenance.

Interactive Agent

Launch the interactive agent with stably:

stably

This opens a conversational interface where you can work with the AI agent to:

Create tests — Describe what you want to test and the agent generates Playwright tests
Fix failing tests — Paste error output or describe issues and get fixes applied
Explore your test suite — Ask questions about coverage, flaky tests, or test structure
Get guidance — Learn best practices or troubleshoot problems interactively

Example Session

$ stably

🤖 Stably Agent
   Type your request or question. Press Ctrl+C to exit.

> Create a test for the checkout flow on our e-commerce site

Analyzing your application...
I'll create a test that covers:
  • Adding items to cart
  • Proceeding to checkout
  • Completing payment

✓ Created tests/checkout.spec.ts

> The login test is failing with a timeout error

Looking at the failure context...
The selector '.login-btn' no longer exists. I found a matching
element with '[data-testid="sign-in"]'.

Apply fix? (y/n): y
✓ Updated tests/auth.spec.ts

> What's our test coverage for the dashboard?

You have 12 tests covering the dashboard:
  • 4 tests for user settings
  • 3 tests for analytics widgets
  • 5 tests for navigation

Missing coverage: notification preferences, export functionality

The interactive agent is ideal when you want a flexible, back-and-forth workflow rather than running individual commands.

Running Tests

There are two ways to run your Playwright tests with Stably integration:

Method 1: stably test (Recommended)
Method 2: npx playwright test

The simplest approach — the Stably reporter is automatically configured.

Setup Requirements

Set environment variables

export STABLY_API_KEY=your_api_key_here
export STABLY_PROJECT_ID=your_project_id_here

Run tests

stably test

Options

# Run all tests
stably test

# Run with Playwright options
stably test --headed --project=chromium

# Run specific test file
stably test tests/login.spec.ts

GitHub Actions Workflow

# .github/workflows/stably.yml
name: Stably Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Install browsers
        run: npx playwright install --with-deps

      - name: Run Stably tests
        env:
          STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
          STABLY_PROJECT_ID: ${{ secrets.STABLY_PROJECT_ID }}
        run: npx stably test

Use this if you need more control over your Playwright configuration.

Setup Requirements

Set environment variables

export STABLY_API_KEY=your_api_key_here
export STABLY_PROJECT_ID=your_project_id_here

Configure the reporter

Add the Stably reporter to your playwright.config.ts:

import { stablyReporter } from "@stablyai/playwright-test";

export default defineConfig({
  reporter: [
    ["list"],
    stablyReporter({
      apiKey: process.env.STABLY_API_KEY,
      projectId: process.env.STABLY_PROJECT_ID,
    }),
  ],
  use: {
    trace: "on", // Required for full context
  },
});

Run tests

npx playwright test

See the Stably Test Reporter documentation for complete setup instructions.

GitHub Actions Workflow

# .github/workflows/playwright.yml
name: Playwright Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Install browsers
        run: npx playwright install --with-deps

      - name: Run Playwright tests
        env:
          STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
          STABLY_PROJECT_ID: ${{ secrets.STABLY_PROJECT_ID }}
        run: npx playwright test

The `stably fix` Command

The most powerful CLI feature — diagnoses test failures and applies AI fixes:

# In CI: automatically uses the last failed run from the current CI job
stably fix

# Locally: provide the run ID from a failed test execution
stably fix <runId>

CI vs Local Usage

Environment	Run ID Required?	Notes
CI	No	Automatically detects the run ID from the CI environment
Local	Yes	Copy the run ID from the test output or Stably dashboard

(Upcoming) Local runs will also support automatic run ID detection, so you won’t need to provide the run ID manually.

How It Works

Run tests

Execute your test suite with stably test (reporter captures run data)

Identify failures

For failures, use stably fix [runId] with the run ID from the output

AI analysis

The AI analyzes the failure context (screenshots, logs, DOM, traces)

Apply fixes

It generates and applies fixes to your test code

Verify

Rerun the test to verify the fix worked

Example Output

stably test
# Running tests...
# ✓ 45 tests passed
# ✗ 3 tests failed
# Run ID: run_abc123

stably fix run_abc123
# Analyzing failures...
#
# Test: checkout.spec.ts > complete purchase
# Issue: Selector '.checkout-button' not found
# Fix: Updated to '[data-testid="checkout-btn"]'
# Status: ✓ Fixed and verified
#
# Test: login.spec.ts > invalid credentials
# Issue: Error message assertion failed
# Fix: Updated expected text to match new design
# Status: ✓ Fixed and verified
#
# Summary: 2 auto-fixed

Command Reference

Core Commands

Command	Description
`stably`	Start interactive agent session
`stably init`	Initialize Playwright and Stably SDK (handles login automatically)
`stably login`	Login via browser authentication (optional, init handles this)
`stably logout`	Logout and clear credentials
`stably whoami`	Show authentication status
`stably create [prompt]`	Create tests with AI
`stably test`	Run Playwright tests
`stably fix [runId]`	Fix failing tests with AI (runId optional in CI)
`stably install`	Install browser dependencies

Additional Options

# Get version
stably --version

# Get help
stably --help

# Help for specific command
stably help [command]

Troubleshooting

Authentication Issues

# Clear and re-authenticate
stably logout
stably login

# Or use API key directly
export STABLY_API_KEY=stably_...

# Check current status
stably whoami

Installation Issues

If you encounter browser-related errors:

# Install browser dependencies
stably install

# Or use Playwright directly
npx playwright install

Getting started

Stably Web Editor

Stably CLI

Stably Cloud Runner

Stably SDK

Stably Test Reporter

Stably Classic

Troubleshooting

CLI commands

Overview

Interactive Agent

Running Tests

Setup Requirements

Options

Setup Requirements

The `stably fix` Command

CI vs Local Usage

How It Works

Command Reference

Core Commands

Additional Options

Troubleshooting

Next Steps

GitHub Actions

Test Reporter

Getting started

Stably Web Editor

Stably CLI

Stably Cloud Runner

Stably SDK

Stably Test Reporter

Stably Classic

Troubleshooting

​Overview

​Interactive Agent

​Running Tests

​Setup Requirements

​Options

​Setup Requirements

​The stably fix Command

​CI vs Local Usage

​How It Works

​Command Reference

​Core Commands

​Additional Options

​Troubleshooting

​Next Steps

GitHub Actions

Test Reporter

Overview

Interactive Agent

Running Tests

Setup Requirements

Options

Setup Requirements

The `stably fix` Command

CI vs Local Usage

How It Works

Command Reference

Core Commands

Additional Options

Troubleshooting

Next Steps