Stably - AI-native Testing Platform

Overview

The Stably GitHub Action runs your Playwright tests on Stably’s cloud infrastructure directly from your GitHub Actions workflows. It supports two modes: Agents (v2) for project-based runs, and Classic (v1) for test-suite-based runs. The action auto-detects which version to use based on your inputs.

Quick Start

# .github/workflows/stably.yml
name: Stably Tests

on:
  push:
    branches: [main]
  pull_request:

permissions:
  pull-requests: write
  contents: write

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Run Stably Tests
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: your-project-id

Inputs (v2 — Agents)

Input	Required	Default	Description
`api-key`	Yes	—	Your Stably API key
`project-id`	Yes	—	Your Stably project ID
`playwright-project-name`	No	—	Playwright project name(s) to run (maps to `--project` CLI flag). If omitted, all projects run.
`environment-name`	No	—	Environment to use (e.g. “Staging”, “Production”). Falls back to the project’s default.
`env-overrides`	No	—	YAML string or JSON object of environment variable overrides
`github-comment`	No	`true`	Post test results as a PR/commit comment
`github-token`	No	`${{ github.token }}`	Token for posting comments. Override with a PAT if needed.
`async`	No	`false`	Launch tests without waiting for results. Comments won’t work in async mode.

Inputs (v1 — Classic)

Input	Required	Default	Description
`api-key`	Yes	—	Your Stably API key
`test-suite-id`	Yes	—	The test suite ID to execute
`environment`	No	`PRODUCTION`	Environment to inherit variables from
`variable-overrides`	No	`{}`	JSON object of variable overrides
`note`	No	—	Optional note to help identify the run
`github-comment`	No	`true`	Post test results as a PR/commit comment
`github-token`	No	`${{ github.token }}`	Token for posting comments
`async`	No	`false`	Launch tests without waiting for results

Outputs

Output	Description
`success`	Boolean indicating if the run was successful
`runId`	The resulting run ID (v2 only)
`testSuiteRunId`	The resulting test suite run ID (v1 only)

Examples

Run Specific Playwright Projects

- name: Run Smoke Tests
  uses: stablyai/stably-runner-action@v4
  with:
    api-key: ${{ secrets.STABLY_API_KEY }}
    project-id: your-project-id
    playwright-project-name: smoke-tests

Override Environment Variables

- name: Run Tests Against Staging
  uses: stablyai/stably-runner-action@v4
  with:
    api-key: ${{ secrets.STABLY_API_KEY }}
    project-id: your-project-id
    environment-name: Staging
    env-overrides: |
      BASE_URL: https://staging.example.com
      API_KEY: abc123

Use a Dynamic Preview URL

If your deployment URL is generated by a previous step (e.g. Vercel, Netlify):

steps:
  - name: Deploy Preview
    id: deploy
    run: |
      PREVIEW_URL="https://my-app-${{ github.sha }}.vercel.app"
      echo "preview_url=$PREVIEW_URL" >> $GITHUB_OUTPUT

  - name: Run Stably Tests
    uses: stablyai/stably-runner-action@v4
    with:
      api-key: ${{ secrets.STABLY_API_KEY }}
      project-id: your-project-id
      env-overrides: |
        BASE_URL: ${{ steps.deploy.outputs.preview_url }}

Classic (v1) Example

- name: Run Tests (v1)
  uses: stablyai/stably-runner-action@v4
  with:
    api-key: ${{ secrets.STABLY_API_KEY }}
    test-suite-id: YOUR_TEST_SUITE_ID
    variable-overrides: |
      {
        "APP_URL": "https://example.com"
      }

Check Outputs

- name: Run Tests
  id: stably
  uses: stablyai/stably-runner-action@v4
  with:
    api-key: ${{ secrets.STABLY_API_KEY }}
    project-id: your-project-id

- name: Check Results
  run: echo "Success: ${{ steps.stably.outputs.success }}"

Permissions

If using the github-comment feature (enabled by default), add these permissions at the top of your workflow:

permissions:
  pull-requests: write
  contents: write

Alternatively, enable read/write permissions for all workflows at Settings > Actions > General > Workflow permissions.

For organizations, you must first allow these permissions at the organization level before they can be set per-repository.

Using the CLI Instead

You can also run Stably tests directly via the CLI in GitHub Actions:

steps:
  - uses: actions/checkout@v4

  - uses: actions/setup-node@v4
    with:
      node-version: '20'

  - run: npm ci

  - name: Run Stably Tests
    env:
      STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
      STABLY_PROJECT_ID: ${{ secrets.STABLY_PROJECT_ID }}
    run: npx stably test

Getting started

Stably Web Editor

Stably CLI

Stably Cloud Runner

Stably SDK

Stably Test Reporter

Configuration

Troubleshooting

Migration Guides

Resources

Stably GitHub Action

Overview

Quick Start

Inputs (v2 — Agents)

Inputs (v1 — Classic)

Outputs

Examples

Run Specific Playwright Projects

Override Environment Variables

Use a Dynamic Preview URL

Classic (v1) Example

Check Outputs

Permissions

Using the CLI Instead

Next Steps

GitLab CI

Alerting

​Overview

​Quick Start

​Inputs (v2 — Agents)

​Inputs (v1 — Classic)

​Outputs

​Examples

​Run Specific Playwright Projects

​Override Environment Variables

​Use a Dynamic Preview URL

​Classic (v1) Example

​Check Outputs

​Permissions

​Using the CLI Instead

​Next Steps

GitLab CI

Alerting

Overview

Quick Start

Inputs (v2 — Agents)

Inputs (v1 — Classic)

Outputs

Examples

Run Specific Playwright Projects

Override Environment Variables

Use a Dynamic Preview URL

Classic (v1) Example

Check Outputs

Permissions

Using the CLI Instead

Next Steps