Overview
The Stably CLI is a command-line tool for developers who prefer working in the terminal. It provides essential commands for authentication, resource creation, test execution, and the powerful stably fix command for automatic test maintenance.
Interactive Agent
Launch the interactive agent with stably:
This opens a conversational interface where you can work with the AI agent to:
Create tests — Describe what you want to test and the agent generates Playwright tests
Fix failing tests — Paste error output or describe issues and get fixes applied
Explore your test suite — Ask questions about coverage, flaky tests, or test structure
Get guidance — Learn best practices or troubleshoot problems interactively
$ stably
🤖 Stably Agent
Type your request or question. Press Ctrl+C to exit.
> Create a test for the checkout flow on our e-commerce site
Analyzing your application...
I 'll create a test that covers:
• Adding items to cart
• Proceeding to checkout
• Completing payment
✓ Created tests/checkout.spec.ts
> The login test is failing with a timeout error
Looking at the failure context...
The selector ' .login-btn ' no longer exists. I found a matching
element with ' [data-testid = "sign-in" ] '.
Apply fix? (y/n): y
✓ Updated tests/auth.spec.ts
> What' s our test coverage for the dashboard?
You have 12 tests covering the dashboard:
• 4 tests for user settings
• 3 tests for analytics widgets
• 5 tests for navigation
Missing coverage: notification preferences, export functionality
The interactive agent is ideal when you want a flexible, back-and-forth workflow rather than running individual commands.
Create Tests on Autopilot
stably create is a headless, one-shot command designed for automation pipelines, background agents, and batch processing. It generates tests and exits — making it ideal for CI/CD workflows, shell scripts, and integration with AI coding agents.
stably create "login with valid and invalid credentials"
The prompt is optional. If no prompt is provided, Stably automatically analyzes:
Current PR — If running in a CI environment with PR context
Git diffs — Changes against origin/HEAD when running locally
This makes it easy to auto-generate tests for your recent changes without describing them manually.
For interactive, back-and-forth test creation, use the Interactive Agent instead. stably create is optimized for unattended execution.
Use Cases
Scenario Example CI/CD pipelines Auto-generate tests for new features in PR workflows Background agents Let AI coding assistants create tests autonomously Batch processing Script bulk test generation across multiple features Scheduled jobs Generate tests for new API endpoints on a cron schedule
Output Location
# Auto-generate tests from PR/git diffs (no prompt)
stably create
# → Analyzes changes and creates relevant tests
# With explicit prompt
stably create "login with valid credentials"
# → Creates tests/login.spec.ts
# Specify output directory
stably create "checkout flow" --output ./e2e/
# → Creates e2e/checkout-flow.spec.ts
Output Directory Resolution
If --output is not specified, Stably automatically detects the output directory:
playwright.config.ts — Uses testDir if defined
Auto-detect — First existing: tests/ → e2e/ → __tests__/ → test/
Fallback — Current working directory
The command prints created file paths, making it easy to parse in CI: # Capture output paths
stably create "login" | grep "^- " | cut -c3-
$ stably create "checkout flow for guest users"
Analyzing application...
Generating tests for: checkout flow for guest users
Created files:
- /absolute/path/to/tests/checkout-guest-add-to-cart.spec.ts
- /absolute/path/to/tests/checkout-guest-payment.spec.ts
Integration Patterns
GitHub Actions: Generate Tests as PR
# .github/workflows/auto-tests.yml
name : Auto-generate Tests
on :
pull_request :
types : [ opened , synchronize ]
jobs :
generate-tests :
runs-on : ubuntu-latest
permissions :
contents : write
pull-requests : write
steps :
- uses : actions/checkout@v4
with :
ref : ${{ github.head_ref }}
- name : Setup Node
uses : actions/setup-node@v4
with :
node-version : '20'
- name : Install dependencies
run : npm ci
- name : Generate tests
env :
STABLY_API_KEY : ${{ secrets.STABLY_API_KEY }}
STABLY_PROJECT_ID : ${{ secrets.STABLY_PROJECT_ID }}
run : npx stably create # Automatically analyzes PR changes
- name : Check for new tests
id : check
run : |
if [[ -n $(git status --porcelain tests/) ]]; then
echo "has_changes=true" >> $GITHUB_OUTPUT
fi
- name : Create PR with generated tests
if : steps.check.outputs.has_changes == 'true'
env :
GH_TOKEN : ${{ secrets.GITHUB_TOKEN }}
run : |
BRANCH="auto-tests/${{ github.head_ref }}"
git config user.name "stably-bot"
git config user.email "[email protected] "
git checkout -b "$BRANCH"
git add tests/
git commit -m "test: auto-generate tests for PR #${{ github.event.pull_request.number }}"
git push -u origin "$BRANCH"
gh pr create \
--title "Generated tests for #${{ github.event.pull_request.number }}" \
--body "Auto-generated tests for the changes in #${{ github.event.pull_request.number }}" \
--base "${{ github.head_ref }}"
Background Agent Integration
# Called by AI coding agents (Cursor, Copilot, etc.)
# The agent can invoke this command to generate tests autonomously
stably create "PaymentService class with edge cases"
# Chain with test execution
stably create "user login and logout flow" && stably test
Running Tests
There are two ways to run your Stably-powered Playwright tests:
The simplest approach — the Stably reporter is automatically configured . Setup Requirements
Set environment variables
export STABLY_API_KEY = your_api_key_here
export STABLY_PROJECT_ID = your_project_id_here
Options Stably is fully Playwright compatible. All Playwright CLI options are supported: # Run all tests
stably test
# Run with Playwright options
stably test --headed --project=chromium
# Run specific test file
stably test tests/login.spec.ts
# More Playwright options
stably test --workers=4 --retries=2 --grep= "login"
# .github/workflows/stably.yml
name : Stably Tests
on :
push :
branches : [ main ]
pull_request :
jobs :
test :
runs-on : ubuntu-latest
steps :
- uses : actions/checkout@v4
- name : Setup Node
uses : actions/setup-node@v4
with :
node-version : '20'
- name : Install dependencies
run : npm ci
- name : Install browsers
run : npx stably install
- name : Run Stably tests
env :
STABLY_API_KEY : ${{ secrets.STABLY_API_KEY }}
STABLY_PROJECT_ID : ${{ secrets.STABLY_PROJECT_ID }}
run : npx stably test
Use this if you need more control over your Playwright configuration. Setup Requirements
Set environment variables
export STABLY_API_KEY = your_api_key_here
export STABLY_PROJECT_ID = your_project_id_here
Configure the reporter
Add the Stably reporter to your playwright.config.ts: import { stablyReporter } from "@stablyai/playwright-test" ;
export default defineConfig ({
reporter: [
[ "list" ],
stablyReporter ({
apiKey: process . env . STABLY_API_KEY ,
projectId: process . env . STABLY_PROJECT_ID ,
}),
] ,
use: {
trace: "on" , // Required for full context
} ,
}) ;
See the Stably Test Reporter documentation for complete setup instructions.
# .github/workflows/playwright.yml
name : Playwright Tests
on :
push :
branches : [ main ]
pull_request :
jobs :
test :
runs-on : ubuntu-latest
steps :
- uses : actions/checkout@v4
- name : Setup Node
uses : actions/setup-node@v4
with :
node-version : '20'
- name : Install dependencies
run : npm ci
- name : Install browsers
run : npx stably install
- name : Run Playwright tests
env :
STABLY_API_KEY : ${{ secrets.STABLY_API_KEY }}
STABLY_PROJECT_ID : ${{ secrets.STABLY_PROJECT_ID }}
run : npx playwright test
Fix Tests on Autopilot
stably fix is a headless command that automatically diagnoses test failures and applies AI-generated fixes. Designed for unattended execution, it’s ideal for self-healing CI pipelines, background maintenance agents, and automated test repair workflows.
# In CI: automatically uses the last failed run from the current CI job
stably fix
# With explicit run ID
stably fix < runI d >
For interactive debugging, use the Interactive Agent instead. stably fix is optimized for automated, hands-off repair.
Use Cases
Scenario Example Self-healing CI Auto-fix flaky tests before they block deployments Background agents Let AI assistants maintain tests autonomously Nightly maintenance Scheduled jobs that repair broken tests overnight PR workflows Fix test failures and commit patches automatically
CI vs Local Usage
Environment Run ID Required? Notes CI No Automatically detects the run ID from the CI environment Local Yes Copy the run ID from the test output or Stably dashboard
(Upcoming) Local runs will also support automatic run ID detection, so you won’t need to provide the run ID manually.
How It Works
The command analyzes failure context (screenshots, logs, DOM, traces) and applies fixes in a single execution:
Run tests
Execute your test suite with stably test (reporter captures run data)
AI analysis
The AI analyzes the failure context automatically
Apply fixes
Fixes are generated and applied to your test code
Exit
Command completes — chain with verification or commit steps
$ stably fix run_abc123
Analyzing 3 failures...
checkout.spec.ts > complete purchase
Issue: Selector '.checkout-button' not found
Fix: Updated to '[data-testid="checkout-btn"]'
✓ Fixed
login.spec.ts > invalid credentials
Issue: Error message assertion failed
Fix: Updated expected text to match new design
✓ Fixed
Summary: 2 auto-fixed, 1 requires manual review
Integration Patterns
GitHub Actions: Self-Healing Pipeline
# .github/workflows/self-healing-tests.yml
name : Self-Healing Tests
on :
push :
branches : [ main ]
pull_request :
jobs :
test-and-fix :
runs-on : ubuntu-latest
steps :
- uses : actions/checkout@v4
- name : Setup Node
uses : actions/setup-node@v4
with :
node-version : '20'
- name : Install dependencies
run : npm ci && npx stably install
- name : Run tests
id : test
continue-on-error : true
env :
STABLY_API_KEY : ${{ secrets.STABLY_API_KEY }}
STABLY_PROJECT_ID : ${{ secrets.STABLY_PROJECT_ID }}
run : npx stably test
- name : Auto-fix failures
if : steps.test.outcome == 'failure'
env :
STABLY_API_KEY : ${{ secrets.STABLY_API_KEY }}
STABLY_PROJECT_ID : ${{ secrets.STABLY_PROJECT_ID }}
run : npx stably fix
- name : Commit fixes
if : steps.test.outcome == 'failure'
run : |
git config user.name "stably-bot"
git config user.email "[email protected] "
git add tests/
git commit -m "fix: auto-repair failing tests" || exit 0
git push
Background Agent Integration
# Called by AI coding agents for autonomous test maintenance
# Chain test execution with automatic repair
stably test || stably fix
# Full pipeline: test → fix → verify
stably test || ( stably fix && stably test )
Command Reference
A complete reference of all available Stably CLI commands.
Commands
Setup
Command Description stably initInitialize Playwright and Stably SDK in your project stably installInstall browser dependencies stably loginAuthenticate via browser-based OAuth
Core Workflow
Command Description stablyStart interactive agent session stably create [prompt]Generate tests from prompt, PR context, or git diffs stably test [options]Run Playwright tests with Stably reporter stably fix [runId]Auto-fix failing tests using AI analysis
Maintenance & Utility
Command Description stably upgradeUpgrade Stably CLI to the latest version stably logoutClear stored credentials stably whoamiDisplay current authentication status stably help [command]Show help for a specific command
Global Options
These options are available for all commands:
Option Description --help, -hDisplay help information --versionDisplay CLI version number --verbose, -vEnable verbose output with debug information --no-telemetryDisable anonymous telemetry for this session
Environment Variables
Configure Stably CLI behavior using environment variables:
Variable Description Required STABLY_API_KEYAPI key for authentication (from Settings → API Keys) Yes STABLY_PROJECT_IDProject identifier (from app.stably.ai) Yes STABLY_BASE_URLCustom API endpoint (for enterprise deployments) No STABLY_LOG_LEVELConsole log level: error, warn, info, debug (default: warn) No STABLY_DISABLE_TELEMETRYSet to 1 to disable anonymous telemetry No DO_NOT_TRACKStandard opt-out for telemetry (set to 1) No
To disable telemetry, set any one of: STABLY_DISABLE_TELEMETRY=1, DO_NOT_TRACK=1, or use the --no-telemetry flag.
Setting environment variables:
Unix/macOS
Windows (PowerShell)
CI/CD
export STABLY_API_KEY = stably_xxxxxxxxxxxx
export STABLY_PROJECT_ID = proj_xxxxxxxxxxxx
Add to ~/.bashrc, ~/.zshrc, or ~/.profile for persistence. $ env: STABLY_API_KEY = "stably_xxxxxxxxxxxx"
$ env: STABLY_PROJECT_ID = "proj_xxxxxxxxxxxx"
For persistence, use System Properties → Environment Variables. # GitHub Actions
env :
STABLY_API_KEY : ${{ secrets.STABLY_API_KEY }}
STABLY_PROJECT_ID : ${{ secrets.STABLY_PROJECT_ID }}
Store secrets in your CI platform’s secret management system.
Exit Codes
Stably CLI uses standard exit codes for scripting and CI/CD integration:
Code Description 0Success — command completed successfully 1Failure — command failed (test failures, errors, etc.) 2Invalid usage — incorrect arguments or missing required options
Example usage in scripts:
# Run tests and handle exit codes
stably test
if [ $? -eq 0 ]; then
echo "All tests passed"
elif [ $? -eq 1 ]; then
echo "Tests failed, attempting auto-fix..."
stably fix
fi
# One-liner: run tests, fix on failure, re-run
stably test || ( stably fix && stably test )
Debug Logging
The Stably CLI automatically writes detailed debug logs to help troubleshoot issues. Logs are organized by date with descriptive session names for easy discovery.
Log Location
Logs are stored in your system’s temp directory:
/tmp/stably-logs/
2024-01-15/
10-30-45-login.log
10-31-02-init.log
10-32-15-write-a-test-for-the-login-page.log
10-45-12-fix-github-myorg_myrepo-123-1.log
Naming convention: HH-MM-SS-{session-name}.log
Named commands use the command name (e.g., login, init, test)
stably create uses the prompt text (sanitized, max 100 chars)
stably fix uses fix-{runId}
Interactive chat uses the first message
Logs in /tmp are automatically cleaned up by your operating system on reboot or via system cleanup policies.
Verbose Mode
Use --verbose (or -v) to see debug output in your terminal and display the log file path:
stably --verbose create "login test"
Output:
Debug log: /tmp/stably-logs/2024-01-15/10-30-45-login-test.log
debug Checking authentication
debug Authentication resolved { authType: 'oauth' }
debug Fetching system prompt { mode: 'single' }
...
Log Levels
You can also set the log level via environment variable:
STABLY_LOG_LEVEL = debug stably create "login test"
Level Description errorUnexpected errors and crashes warnConfiguration issues, auth failures (default console output) infoNormal operations (session start/end) debugDetailed debugging (API calls, state changes)
Sharing Logs with Support
When errors occur, the CLI automatically displays the log file path (no --verbose required):
Debug log written to:
/tmp/stably-logs/2024-01-15/10-30-45-login-test.log
Share this file with support for assistance.
Attach this file when contacting support for faster resolution.
To see the log file path for successful runs , use --verbose. The path will be shown at startup and when you press Ctrl+C.
Troubleshooting
# Clear and re-authenticate
stably logout
stably login
# Or use API key directly
export STABLY_API_KEY = stably_ ...
# Check current status
stably whoami
If you encounter browser-related errors: # Install browser dependencies
stably install
# Or use Playwright directly
npx playwright install
Next Steps