Stably - AI-native Testing Platform

Overview

The Stably CLI is a command-line tool for developers who prefer working in the terminal. It provides essential commands for authentication, coverage planning (stably plan), test generation (stably create), test execution (stably test), and automated maintenance with stably fix while keeping your tests and fixes in your local repository.

Browser-backed CLI agent commands can use a Stably-hosted browser with --browser=cloud or STABLY_CLOUD_BROWSER=1. See Cloud Browsers for Stably CLI.

Interactive Agent

Launch the interactive agent with stably:

stably
stably --browser=cloud

This opens a conversational interface where you can work with the AI agent to:

Create tests — Describe what you want to test and the agent generates Playwright tests
Fix failing tests — Paste error output or describe issues and get fixes applied
Explore your test suite — Ask questions about coverage, flaky tests, or test structure
Get guidance — Learn best practices or troubleshoot problems interactively

Example Session

$ stably

🤖 Stably Agent
   Type your request or question. Press Ctrl+C to exit.

> Create a test for the checkout flow on our e-commerce site

Analyzing your application...
I'll create a test that covers:
  • Adding items to cart
  • Proceeding to checkout
  • Completing payment

✓ Created tests/checkout.spec.ts

> The login test is failing with a timeout error

Looking at the failure context...
The selector '.login-btn' no longer exists. I found a matching
element with '[data-testid="sign-in"]'.

Apply fix? (y/n): y
✓ Updated tests/auth.spec.ts

> What's our test coverage for the dashboard?

You have 12 tests covering the dashboard:
  • 4 tests for user settings
  • 3 tests for analytics widgets
  • 5 tests for navigation

Missing coverage: notification preferences, export functionality

The interactive agent is ideal when you want a flexible, back-and-forth workflow rather than running individual commands.

Cloud Browser Mode

The interactive agent can run in a Stably-hosted browser:

stably --browser=cloud

This is useful in remote dev environments, coding-agent sessions, or containers where local browser setup is undesirable.

Create Tests on Autopilot

stably create is a headless, one-shot command designed for automation pipelines, background agents, and batch processing. It generates tests and exits — making it ideal for CI/CD workflows, shell scripts, and integration with AI coding agents.

stably create "login with valid and invalid credentials"

The prompt is optional. If no prompt is provided, Stably automatically analyzes:

Current PR — If running in a CI environment with PR context
Git diffs — Changes against origin/HEAD when running locally

This makes it easy to auto-generate tests for your recent changes without describing them manually.

For interactive, back-and-forth test creation, use the Interactive Agent instead. stably create is optimized for unattended execution.

Use Cases

Scenario	Example
CI/CD pipelines	Auto-generate tests for new features in PR workflows
Background agents	Let AI coding assistants create tests autonomously
Batch processing	Script bulk test generation across multiple features
Scheduled jobs	Generate tests for new API endpoints on a cron schedule

Output Location

# Auto-generate tests from PR/git diffs (no prompt)
stably create
# → Analyzes changes and creates relevant tests

# With explicit prompt
stably create "login with valid credentials"
# → Creates tests/login.spec.ts

# Specify output directory
stably create "checkout flow" --output ./e2e/
# → Creates e2e/checkout-flow.spec.ts

Output Directory Resolution

If --output is not specified, Stably automatically detects the output directory:

playwright.config.ts — Uses testDir if defined
Auto-detect — First existing: tests/ → e2e/ → __tests__/ → test/
Fallback — Current working directory

The command prints created file paths, making it easy to parse in CI:

# Capture output paths
stably create "login" | grep "^- " | cut -c3-

Example Output

$ stably create "checkout flow for guest users"

Analyzing application...
Generating tests for: checkout flow for guest users

Created files:
- /absolute/path/to/tests/checkout-guest-add-to-cart.spec.ts
- /absolute/path/to/tests/checkout-guest-payment.spec.ts

Integration Patterns

GitHub Actions: Generate Tests for PR Changes

# .github/workflows/auto-tests.yml
name: Auto-generate Tests

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  generate-tests:
    # Skip PRs created by stably-bot to prevent infinite loops
    if: github.event.pull_request.user.login != 'stably-bot'
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.head_ref }}

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Install browsers
        run: npx stably install

      - name: Generate tests
        env:
          STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
          STABLY_PROJECT_ID: ${{ secrets.STABLY_PROJECT_ID }}
          # App credentials for login during test generation
          TEST_USERNAME: ${{ secrets.TEST_USERNAME }}
          TEST_PASSWORD: ${{ secrets.TEST_PASSWORD }}
        run: npx stably create  # Automatically analyzes PR changes

      - name: Check for new tests
        id: check
        run: |
          # Checks for changes in common test directories
          if [[ -n $(git status --porcelain tests/ e2e/ __tests__ 2>/dev/null) ]]; then
            echo "has_changes=true" >> $GITHUB_OUTPUT
          fi

      - name: Create PR with generated tests
        if: steps.check.outputs.has_changes == 'true'
        continue-on-error: true
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          BRANCH="auto-tests/${{ github.head_ref }}-${{ github.run_number }}"
          git config user.name "stably-bot"
          git config user.email "bot@stably.ai"
          git checkout -b "$BRANCH"
          git add tests/ e2e/ __tests__/
          git commit -m "test: auto-generate tests for PR #${{ github.event.pull_request.number }}"
          git push -u origin "$BRANCH"
          gh pr create \
            --title "Generated tests for #${{ github.event.pull_request.number }}" \
            --body "Auto-generated tests for the changes in #${{ github.event.pull_request.number }}" \
            --base "${{ github.head_ref }}"

GitHub Actions: Generate Tests from Staging Deployment

# .github/workflows/staging-tests.yml
name: Generate Tests from Staging

on:
  deployment_status:
    # Triggers when any deployment status changes.
    # Example: if you have a "staging" deployment environment in GitHub,
    # this fires automatically when that deployment succeeds.
    # See: https://docs.github.com/en/actions/deployment/about-deployments
  workflow_dispatch:
    # Allow manual triggers for ad-hoc test generation

jobs:
  generate-staging-tests:
    runs-on: ubuntu-latest
    # Only run on successful staging deployments (skip production, preview, etc.)
    # Skip PRs created by stably-bot to prevent infinite loops
    if: >
      (github.event_name == 'workflow_dispatch' ||
       (github.event.deployment_status.state == 'success' &&
        github.event.deployment.environment == 'staging')) &&
      github.actor != 'stably-bot'
    permissions:
      contents: write
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
        with:
          # Check out the exact commit that was deployed
          ref: ${{ github.event.deployment.sha || github.sha }}

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Install browsers
        run: npx stably install

      - name: Generate tests for staging
        env:
          STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
          STABLY_PROJECT_ID: ${{ secrets.STABLY_PROJECT_ID }}
          # App credentials for login during test generation
          TEST_USERNAME: ${{ secrets.TEST_USERNAME }}
          TEST_PASSWORD: ${{ secrets.TEST_PASSWORD }}
        run: |
          npx stably create "Go to ${{ vars.STAGING_URL }} and create tests for any new features between this and the last staging deployment. Plan it out first."

      - name: Check for new tests
        id: check
        run: |
          # Checks for changes in common test directories
          if [[ -n $(git status --porcelain tests/ e2e/ __tests__ 2>/dev/null) ]]; then
            echo "has_changes=true" >> $GITHUB_OUTPUT
          fi

      - name: Create PR with generated tests
        if: steps.check.outputs.has_changes == 'true'
        continue-on-error: true
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          BRANCH="staging-tests/$(date +%Y%m%d-%H%M%S)"
          git config user.name "stably-bot"
          git config user.email "bot@stably.ai"
          git checkout -b "$BRANCH"
          git add tests/ e2e/ __tests__/
          git commit -m "test: auto-generate tests from staging deployment"
          git push -u origin "$BRANCH"
          gh pr create \
            --title "Generated tests from staging deployment" \
            --body "Auto-generated tests based on new features detected on staging." \
            --base main

Background Agent Integration

# Called by AI coding agents (Cursor, Copilot, etc.)
# The agent can invoke this command to generate tests autonomously

stably create "PaymentService class with edge cases"

# Chain with test execution
stably create "user login and logout flow" && stably test

Avoid infinite PR loops. If a PR created by npx stably create triggers the same workflow, it can create an endless cycle of auto-generated PRs. Always add a precondition to skip the workflow when the PR author is stably-bot:

jobs:
  generate-tests:
    if: github.event.pull_request.user.login != 'stably-bot'

Plan Test Coverage

stably plan analyzes your repository, identifies likely coverage gaps, and generates user-reviewable test.fixme() plan files. Use it when you want a concrete test plan before generating or writing real tests.

# Plan coverage across the repository
stably plan

# Plan around a specific area or workflow
stably plan "focus on checkout, login, and billing edge cases"

Unlike stably verify, stably plan does not open a browser. Unlike stably create, it does not try to finish real tests in one pass. It stays focused on repo analysis and produces plan files you can review, refine, and turn into actual tests later. See the full Test Planning (stably plan) guide for examples and workflow guidance.

Running Tests

stably test runs your Playwright tests with the Stably reporter automatically configured — no manual setup needed.

# Run all tests
stably test

# Run with any Playwright option
stably test --headed --project=chromium
stably test tests/login.spec.ts
stably test --workers=4 --retries=2 --grep="login"

# Override the reported suite name
stably test --suiteName="nightly smoke tests"

Use --suiteName to set a custom name for the suite as it appears on the Stably dashboard and in Slack notifications. When omitted, the suite name defaults to the full list of Playwright project names being run (sorted alphabetically, joined with , ). See the full Run Tests guide for environment variables, CI workflows, debug mode, and more.

Fix Tests on Autopilot

stably fix automatically diagnoses test failures and applies AI-generated fixes — ideal for self-healing CI pipelines, background agents, and automated maintenance.

stably fix               # Auto-detects the last test run
stably fix <runId>       # With explicit run ID
stably test || stably fix  # Chain: test then fix on failure

See the full Fix Tests (stably fix) guide for run ID detection, CI integration, diagnosis categories, monitoring, and configuration.

Verify App Behavior

stably verify checks whether your application works correctly by describing expected behavior in plain English. An AI agent launches a real browser, interacts with your app, and reports a structured PASS / FAIL / INCONCLUSIVE verdict — no test files generated.

# Verify a feature works
stably verify "the login form accepts email and password and redirects to /dashboard"

# With a specific starting URL
stably verify "the pricing page shows 3 tiers" --url http://localhost:3000/pricing

# Set a budget cap (default: $5)
stably verify "checkout flow completes successfully" --max-budget 10

Exit codes: 0 = PASS, 1 = FAIL, 2 = INCONCLUSIVE — making it composable in scripts and CI pipelines.

See the full Verify with AI Agents guide for detailed output examples, agent iteration workflows, and the stably-verify skill prompt.

Run History (stably runs)

stably runs lets you browse and inspect test run history from the terminal — filter by branch, status, source, and more.

stably runs list --branch main --status failed
stably runs view <runId>

See the full Run History (stably runs) guide for all filtering options, pagination, JSON output, and example workflows.

Test Health Analytics (stably analytics)

stably analytics surfaces your most problematic tests — ranked by flaky rate or failure rate — so you can prioritize fixes where they matter most.

# Show the most flaky tests over the last 7 days
stably analytics flaky

# Show the most failing tests on the main branch over the last 30 days
stably analytics failures --branch main --days 30

Subcommands

Command	Description
`stably analytics flaky`	Show most flaky tests ranked by flaky rate
`stably analytics failures`	Show most failing tests ranked by failure rate

Options

Both subcommands share the same options:

Option	Description
`--days <n>`	Look-back window in days (1–90, default: 7)
`-b, --branch <name>`	Filter by branch name
`-n, --limit <n>`	Max rows returned (1–100, default: 10)
`--json`	Output as JSON

Examples

# Top 5 flakiest tests in the last 14 days
stably analytics flaky --days 14 --limit 5

# Failures on the develop branch, output as JSON for scripting
stably analytics failures --branch develop --json

# Quick health check on main
stably analytics flaky --branch main
stably analytics failures --branch main

Agent Configuration

Configure CLI agent behavior using stably.yaml in your repository root and STABLY-CREATE.md for test generation rules.

Feature	Configuration	Scope
`stably fix` custom rules	`agent.fix` in `stably.yaml`	Fix agent behavior
`stably create` custom rules	`STABLY-CREATE.md` in project root	Test generation behavior
Scheduled runs	`schedules` in `stably.yaml`	Cloud runner schedules

See Fix Tests (stably fix) — Configuration for the full agent.fix reference.

Command Reference

A complete reference of all available Stably CLI commands.

Commands

Setup

Command	Description
`stably init`	Initialize Playwright and Stably SDK in your project
`stably install`	Install browser dependencies
`stably login`	Authenticate via browser-based OAuth

Core Workflow

Command	Description
`stably`	Start interactive agent session
`stably plan [prompt]`	Discover coverage gaps and generate `test.fixme()` plan files
`stably create [prompt]`	Generate tests directly from prompt, PR context, or git diffs
`stably test [options] [--suiteName <name>]`	Run Playwright tests with Stably reporter
`stably verify <prompt>`	Verify app behavior against a natural language description
`stably fix [runId]`	Auto-fix failing tests (auto-detects run ID from last test run or CI)
`stably runs list`	List recent test runs with filtering
`stably runs view <runId>`	View detailed results for a specific run
`stably analytics flaky`	Show most flaky tests ranked by flaky rate
`stably analytics failures`	Show most failing tests ranked by failure rate

Maintenance & Utility

Command	Description
`stably upgrade`	Upgrade Stably CLI to the latest version
`stably logout`	Clear stored credentials
`stably whoami`	Display current authentication status
`stably help [command]`	Show help for a specific command

Global Options

These options are available for all commands:

Option	Description
`--help`, `-h`	Display help information
`--version`	Display CLI version number
`--verbose`, `-v`	Enable verbose output with debug information
`--no-telemetry`	Disable anonymous telemetry for this session
`--browser <type>`	Browser mode for browser-backed agent commands: `local` or `cloud`
`--env <name>`	Load variables from a named environment stored in Stably
`--env-file <path>`	Load variables from a local `.env` file (can be specified multiple times)
`-C`, `--cwd <path>`	Change working directory before running the command

Environment Variables

Configure Stably CLI behavior using environment variables:

Variable	Description	Required
`STABLY_API_KEY`	API key for authentication (from Settings → API Keys)	Yes
`STABLY_PROJECT_ID`	Project identifier (from app.stably.ai)	Yes
`STABLY_BASE_URL`	Custom API endpoint (for enterprise deployments)	No
`STABLY_CLOUD_BROWSER`	Set to `1` to default browser-backed agent commands to cloud mode	No
`STABLY_LOG_LEVEL`	Console log level: `error`, `warn`, `info`, `debug` (default: `warn`)	No
`STABLY_DISABLE_TELEMETRY`	Set to `1` to disable anonymous telemetry	No
`DO_NOT_TRACK`	Standard opt-out for telemetry (set to `1`)	No

To disable telemetry, set any one of: STABLY_DISABLE_TELEMETRY=1, DO_NOT_TRACK=1, or use the --no-telemetry flag.

Setting environment variables:

Unix/macOS
Windows (PowerShell)
CI/CD

export STABLY_API_KEY=stably_xxxxxxxxxxxx
export STABLY_PROJECT_ID=proj_xxxxxxxxxxxx

Add to ~/.bashrc, ~/.zshrc, or ~/.profile for persistence.

$env:STABLY_API_KEY = "stably_xxxxxxxxxxxx"
$env:STABLY_PROJECT_ID = "proj_xxxxxxxxxxxx"

For persistence, use System Properties → Environment Variables.

# GitHub Actions
env:
  STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
  STABLY_PROJECT_ID: ${{ secrets.STABLY_PROJECT_ID }}

Store secrets in your CI platform’s secret management system.

Test Environment Variables

Beyond Stably configuration, you can pass your own variables to tests using --env and --env-file:

# Load from a named environment on Stably
stably --env Staging test

# Load from a local .env file
stably test --env-file .env.staging

# Combine both (local overrides remote)
stably --env Production test --env-file .env

Variable precedence (highest priority wins):

Stably internals (STABLY_API_KEY, STABLY_PROJECT_ID)
process.env — system/shell environment
--env-file — local .env file(s)
--env — remote environment from Stably

See Environments for managing named environments on the Stably dashboard.

Exit Codes

Stably CLI uses standard exit codes for scripting and CI/CD integration:

Code	Description
`0`	Success — command completed successfully
`1`	Failure — command failed (test failures, errors, etc.)
`2`	Invalid usage — incorrect arguments or missing required options

Example usage in scripts:

# Run tests and handle exit codes
stably test
if [ $? -eq 0 ]; then
  echo "All tests passed"
elif [ $? -eq 1 ]; then
  echo "Tests failed, attempting auto-fix..."
  stably fix
fi

# One-liner: run tests, fix on failure, re-run
stably test || (stably fix && stably test)

Debug Logging

The Stably CLI automatically writes detailed debug logs to help troubleshoot issues. Logs are organized by date with descriptive session names for easy discovery.

Log Location

Logs are stored in your system’s temp directory:

/tmp/stably-logs/
  2024-01-15/
    10-30-45-login.log
    10-31-02-init.log
    10-32-15-write-a-test-for-the-login-page.log
    10-45-12-fix-github-myorg_myrepo-123-1.log

Naming convention: HH-MM-SS-{session-name}.log

Named commands use the command name (e.g., login, init, test)
stably create uses the prompt text (sanitized, max 100 chars)
stably fix uses fix-{runId}
Interactive chat uses the first message

Logs in /tmp are automatically cleaned up by your operating system on reboot or via system cleanup policies.

Verbose Mode

Use --verbose (or -v) to see debug output in your terminal and display the log file path:

stably --verbose create "login test"

Output:

Debug log: /tmp/stably-logs/2024-01-15/10-30-45-login-test.log

debug Checking authentication
debug Authentication resolved { authType: 'oauth' }
debug Fetching system prompt { mode: 'single' }
...

Log Levels

You can also set the log level via environment variable:

STABLY_LOG_LEVEL=debug stably create "login test"

Level	Description
`error`	Unexpected errors and crashes
`warn`	Configuration issues, auth failures (default console output)
`info`	Normal operations (session start/end)
`debug`	Detailed debugging (API calls, state changes)

When errors occur, the CLI automatically displays the log file path (no --verbose required):

Debug log written to:
  /tmp/stably-logs/2024-01-15/10-30-45-login-test.log
Share this file with support for assistance.

Attach this file when contacting support for faster resolution.

To see the log file path for successful runs, use --verbose. The path will be shown at startup and when you press Ctrl+C.

Troubleshooting

Authentication Issues

# Clear and re-authenticate
stably logout
stably login

# Or use API key directly
export STABLY_API_KEY=stably_...

# Check current status
stably whoami

Installation Issues

If you encounter browser-related errors:

# Install browser dependencies
stably install

# Or use Playwright directly
npx stably install

​Overview

​Interactive Agent

​Cloud Browser Mode

​Create Tests on Autopilot

​Use Cases

​Output Location

​Integration Patterns

​Plan Test Coverage

​Running Tests

​Fix Tests on Autopilot

​Verify App Behavior

​Run History (stably runs)

​Test Health Analytics (stably analytics)

​Subcommands

​Options

​Examples

​Agent Configuration

​Command Reference

​Commands

​Setup

​Core Workflow

​Maintenance & Utility

​Global Options

​Environment Variables

​Test Environment Variables

​Exit Codes

​Debug Logging

​Log Location

​Verbose Mode

​Log Levels

​Sharing Logs with Support

​Troubleshooting

​Next Steps

GitHub Actions

Test Reporter

Overview

Interactive Agent

Cloud Browser Mode

Create Tests on Autopilot

Use Cases

Output Location

Integration Patterns

Plan Test Coverage

Running Tests

Fix Tests on Autopilot

Verify App Behavior

Run History (stably runs)

Test Health Analytics (stably analytics)

Subcommands

Options

Examples

Agent Configuration

Command Reference

Commands

Setup

Core Workflow

Maintenance & Utility

Global Options

Environment Variables

Test Environment Variables

Exit Codes

Debug Logging

Log Location

Verbose Mode

Log Levels

Sharing Logs with Support

Troubleshooting

Next Steps