GitHub Actions Integration - Automated Testing

Stably provides a dedicated GitHub Action that runs your Playwright tests on Stably Cloud as part of your GitHub workflows. The action automatically detects whether to use v2 (project-based) or v1 (test-suite-based) mode based on the inputs you provide.

Overview

The Stably GitHub Action allows you to:

Run all or specific Playwright projects on Stably Cloud
Override environment variables per-run
Receive test results as PR comments
Block deployments on test failures

Setup Instructions

Create API Key Secret

Get your API key from the settings page on the Web UI.Then, go to your GitHub repository, click on “Settings” > “Secrets and variables” > “Actions”. Create a new Repository Secret named STABLY_API_KEY and paste your API key into the secret value.

Get Project ID

Find your project ID in your Stably dashboard settings or project URL.

Create Workflow File

Create the file .github/workflows/stably.yaml in your GitHub repository with your test configuration.

Basic Configuration

Run All Tests

.github/workflows/stably.yaml

name: Run Stably Tests

on:
  pull_request:
  push:
    branches:
      - main

permissions:
  pull-requests: write
  contents: write

jobs:
  test:
    name: Run Tests
    runs-on: ubuntu-latest

    steps:
      - name: Run Stably Tests
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: your-project-id

Run Specific Playwright Projects

Use the playwright-project-name input to run specific Playwright projects:

.github/workflows/stably.yaml

name: Run Smoke Tests

on:
  pull_request:

permissions:
  pull-requests: write
  contents: write

jobs:
  smoke-tests:
    name: Smoke Tests
    runs-on: ubuntu-latest

    steps:
      - name: Run Smoke Tests
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: your-project-id
          playwright-project-name: smoke

Select an Environment

Use environment-name to load variables from a specific environment configured in Stably:

.github/workflows/stably.yaml

name: Run Staging Tests

on:
  push:
    branches:
      - main

permissions:
  pull-requests: write
  contents: write

jobs:
  staging-tests:
    name: Staging Tests
    runs-on: ubuntu-latest

    steps:
      - name: Run Tests on Staging
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: your-project-id
          environment-name: Staging

Pull Request Comments

PR comments are enabled by default (github-comment: true). To use them, add the required permissions:

.github/workflows/stably.yaml

name: PR Testing

on:
  pull_request:

permissions:
  pull-requests: write
  contents: write

jobs:
  test:
    name: Run PR Tests
    runs-on: ubuntu-latest

    steps:
      - name: Run Tests with PR Comments
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: your-project-id

Environment Variable Overrides

Override environment variables for a specific run using env-overrides:

- name: Run Tests
  uses: stablyai/stably-runner-action@v4
  with:
    api-key: ${{ secrets.STABLY_API_KEY }}
    project-id: your-project-id
    env-overrides: |
      BASE_URL: https://staging.example.com
      API_KEY: abc123

Using Dynamic URLs from Previous Steps

If your preview URL is generated dynamically (e.g. from Vercel or Netlify), capture it and pass it via env-overrides:

.github/workflows/stably.yaml

name: Deploy and Test

on:
  pull_request:

permissions:
  pull-requests: write
  contents: write

jobs:
  deploy-and-test:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy Preview
        id: deploy
        run: |
          # Your deployment step — capture the URL
          PREVIEW_URL="https://my-app-${{ github.sha }}.vercel.app"
          echo "preview_url=$PREVIEW_URL" >> $GITHUB_OUTPUT

      - name: Run Stably Tests
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: your-project-id
          env-overrides: |
            BASE_URL: ${{ steps.deploy.outputs.preview_url }}

Action Inputs Reference

Input	Required	Default	Description
`api-key`	Yes	—	Your Stably API key
`project-id`	Yes (v2)	—	Your Stably project ID. Use this OR `test-suite-id`, not both.
`playwright-project-name`	No	—	Playwright project name(s) to run. Maps to the `--project` CLI flag.
`environment-name`	No	—	Environment to use (e.g. “Staging”). Falls back to the project’s default.
`env-overrides`	No	—	YAML string or JSON object of environment variable overrides.
`test-suite-id`	Yes (v1)	—	Test suite ID to execute (v1 only). Use this OR `project-id`.
`github-comment`	No	`true`	Post test results as a PR/commit comment.
`github-token`	No	`${{ github.token }}`	Token for posting comments.
`async`	No	`false`	Launch tests without waiting. Comments won’t work in async mode.

Action Outputs

Output	Description
`success`	Boolean indicating if the run was successful
`runId`	The run ID (v2 only)
`testSuiteRunId`	The test suite run ID (v1 only)

Complete Example

.github/workflows/stably.yaml

name: Stably Test Suite

on:
  pull_request:
  push:
    branches:
      - main

permissions:
  pull-requests: write
  contents: write

jobs:
  smoke-tests:
    name: Smoke Tests
    runs-on: ubuntu-latest
    steps:
      - name: Run Smoke Tests
        id: smoke
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: ${{ secrets.STABLY_PROJECT_ID }}
          playwright-project-name: smoke

      - name: Print Result
        run: echo "Success: ${{ steps.smoke.outputs.success }}"

  regression-tests:
    name: Regression Tests
    runs-on: ubuntu-latest
    needs: smoke-tests
    if: github.event_name == 'push'
    steps:
      - name: Run Full Regression
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: ${{ secrets.STABLY_PROJECT_ID }}
          playwright-project-name: regression
          environment-name: Production

Test Organization

For optimal GitHub Actions integration, organize your tests using Playwright projects:

playwright.config.ts

import { defineConfig } from '@playwright/test';

export default defineConfig({
  projects: [
    {
      name: 'smoke',
      testMatch: '/smoke/**'
    },
    {
      name: 'regression',
      testMatch: '/regression/**'
    }
  ]
});

Then reference these projects in your workflows:

- uses: stablyai/stably-runner-action@v4
  with:
    api-key: ${{ secrets.STABLY_API_KEY }}
    project-id: ${{ secrets.STABLY_PROJECT_ID }}
    playwright-project-name: smoke

Troubleshooting

Permission Denied for PR Comments

Make sure you’ve set the required permissions in your workflow:

permissions:
  pull-requests: write
  contents: write

For organizations, you must first allow these permissions at the organization level.

Project Not Found

Verify the project names in your playwright-project-name input match exactly what’s defined in your playwright.config.ts.

Next Steps

Define Test Groups

Organize tests using Playwright projects

Run on Cloud

Learn more about cloud execution

Schedule Tests

Configure automatic test schedules

Configure Notifications

Set up test result notifications

Getting started

Stably Web Editor

Stably CLI

Stably Cloud Runner

Stably SDK

Stably Test Reporter

Configuration

Troubleshooting

GitHub Actions

Overview

Setup Instructions

Basic Configuration

Run All Tests

Run Specific Playwright Projects

Select an Environment

Pull Request Comments

Environment Variable Overrides

Using Dynamic URLs from Previous Steps

Action Inputs Reference

Action Outputs

Complete Example

Test Organization

Troubleshooting

Next Steps

Define Test Groups

Run on Cloud

Schedule Tests

Configure Notifications

Getting started

Stably Web Editor

Stably CLI

Stably Cloud Runner

Stably SDK

Stably Test Reporter

Configuration

Troubleshooting

​Overview

​Setup Instructions

​Basic Configuration

​Run All Tests

​Run Specific Playwright Projects

​Select an Environment

​Pull Request Comments

​Environment Variable Overrides

​Using Dynamic URLs from Previous Steps

​Action Inputs Reference

​Action Outputs

​Complete Example

​Test Organization

​Troubleshooting

​Next Steps

Define Test Groups

Run on Cloud

Schedule Tests

Configure Notifications

Overview

Setup Instructions

Basic Configuration

Run All Tests

Run Specific Playwright Projects

Select an Environment

Pull Request Comments

Environment Variable Overrides

Using Dynamic URLs from Previous Steps

Action Inputs Reference

Action Outputs

Complete Example

Test Organization

Troubleshooting

Next Steps