Skip to main content

Overview

The Stably GitHub Action integrates test execution into your CI/CD pipeline. Run tests on every push, pull request, or on a schedule.

Quick Start

# .github/workflows/stably.yml
name: Stably Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run Stably Tests
        uses: stablyai/stably-runner-action@v4
        with:
          group: smoke
          stably-api-key: ${{ secrets.STABLY_API_KEY }}

Configuration

Inputs

InputRequiredDescriptionDefault
stably-api-keyYesYour Stably API key-
groupYesTest group to run-
shardsNoNumber of parallel workers1
timeoutNoMaximum run duration30m
fail-on-test-failureNoFail workflow if tests failtrue

Example with All Options

- name: Run Stably Tests
  uses: stablyai/stably-runner-action@v4
  with:
    group: regression
    stably-api-key: ${{ secrets.STABLY_API_KEY }}
    shards: 5
    timeout: 60m
    fail-on-test-failure: true

Common Workflows

Run on Pull Requests

name: PR Tests

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run Smoke Tests
        uses: stablyai/stably-runner-action@v4
        with:
          group: smoke
          stably-api-key: ${{ secrets.STABLY_API_KEY }}

Scheduled Regression Tests

name: Nightly Regression

on:
  schedule:
    - cron: '0 2 * * *'  # Daily at 2 AM

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run Regression Tests
        uses: stablyai/stably-runner-action@v4
        with:
          group: regression
          stably-api-key: ${{ secrets.STABLY_API_KEY }}
          shards: 10

Run Tests Against Staging

name: Staging Tests

on:
  workflow_dispatch:
    inputs:
      environment:
        description: 'Environment to test'
        required: true
        default: 'staging'

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run Tests
        uses: stablyai/stably-runner-action@v4
        with:
          group: smoke
          stably-api-key: ${{ secrets.STABLY_API_KEY }}
        env:
          BASE_URL: https://${{ inputs.environment }}.example.com

Multiple Test Groups

name: Full Test Suite

on:
  push:
    branches: [main]

jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Smoke Tests
        uses: stablyai/stably-runner-action@v4
        with:
          group: smoke
          stably-api-key: ${{ secrets.STABLY_API_KEY }}

  regression:
    runs-on: ubuntu-latest
    needs: smoke
    steps:
      - uses: actions/checkout@v4
      - name: Regression Tests
        uses: stablyai/stably-runner-action@v4
        with:
          group: regression
          stably-api-key: ${{ secrets.STABLY_API_KEY }}
          shards: 5

Using the CLI Instead

You can also use the Stably CLI directly in GitHub Actions:
name: Stably Tests

on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run tests on Stably Cloud
        env:
          STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
        run: npx stably cloud test smoke
      
      # Or run locally in your CI
      - name: Run tests locally
        run: npx stably test smoke

Outputs

The action provides outputs you can use in subsequent steps:
- name: Run Tests
  id: stably
  uses: stablyai/stably-runner-action@v4
  with:
    group: smoke
    stably-api-key: ${{ secrets.STABLY_API_KEY }}

- name: Check Results
  run: |
    echo "Run ID: ${{ steps.stably.outputs.run-id }}"
    echo "Passed: ${{ steps.stably.outputs.passed }}"
    echo "Failed: ${{ steps.stably.outputs.failed }}"
    echo "Dashboard: ${{ steps.stably.outputs.dashboard-url }}"

Available Outputs

OutputDescription
run-idUnique identifier for the test run
passedNumber of passed tests
failedNumber of failed tests
skippedNumber of skipped tests
dashboard-urlLink to results in Stably dashboard

Secrets Setup

  1. Go to your repository Settings → Secrets and variables → Actions
  2. Click “New repository secret”
  3. Name: STABLY_API_KEY
  4. Value: Your API key from app.stably.ai/settings

Troubleshooting

Tests Timeout

Increase the timeout value:
- uses: stablyai/stably-runner-action@v4
  with:
    group: regression
    stably-api-key: ${{ secrets.STABLY_API_KEY }}
    timeout: 60m

Environment Variables Not Set

Pass environment variables explicitly:
- uses: stablyai/stably-runner-action@v4
  with:
    group: smoke
    stably-api-key: ${{ secrets.STABLY_API_KEY }}
  env:
    BASE_URL: ${{ vars.BASE_URL }}
    API_KEY: ${{ secrets.APP_API_KEY }}

Next Steps