Skip to main content
Stably provides a dedicated GitHub Action that runs your Playwright tests in the cloud as part of your GitHub workflows. The action supports all Playwright project configurations and test selection options.

​
Overview

The Stably GitHub Action allows you to:
  • Run specific Playwright projects or all projects
  • Filter tests using grep patterns
  • Control parallelism and retries
  • Receive test results as PR comments
  • Block deployments on test failures

​
Setup Instructions

1

Deploy Your Tests

First, ensure your tests are deployed to Stably Cloud:
npx stably deploy
2

Create API Key Secret

Get your API key from the settings page on the Web UI.Then, go to your GitHub repository, click on “Settings” > “Secrets and variables” > “Actions”. Create a new Repository Secret named STABLY_API_KEY and paste your API key into the secret value.
3

Get Project ID

Find your project ID in your Stably dashboard settings or project URL.
4

Create Workflow File

Create the file .github/workflows/stably.yaml in your GitHub repository with your test configuration.

​
Basic Configuration

​
Run All Tests

.github/workflows/stably.yaml
name: Run Stably Tests

on:
  pull_request:
  push:
    branches:
      - main

jobs:
  test:
    name: Run Tests
    runs-on: ubuntu-latest

    steps:
      - name: Run Stably Tests
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: your-project-id

​
Run Specific Projects

Use the projects parameter to run specific Playwright projects:
.github/workflows/stably.yaml
name: Run Smoke Tests

on:
  pull_request:

jobs:
  smoke-tests:
    name: Smoke Tests
    runs-on: ubuntu-latest

    steps:
      - name: Run Smoke Tests
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: your-project-id
          projects: "smoke,critical"

​
Run Tests with Grep Pattern

Combine projects with grep patterns for fine-grained control:
.github/workflows/stably.yaml
name: Run P0 Tests

on:
  push:
    branches:
      - main

jobs:
  critical-tests:
    name: P0 Critical Tests
    runs-on: ubuntu-latest

    steps:
      - name: Run P0 Tests
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: your-project-id
          projects: "smoke,release"
          grep: "@p0"

​
Pull Request Comments

Enable PR comments to get test results posted directly on your pull requests:
.github/workflows/stably.yaml
name: PR Testing

on:
  pull_request:

# Required for posting PR comments
permissions:
  pull-requests: write
  contents: read

jobs:
  test:
    name: Run PR Tests
    runs-on: ubuntu-latest

    steps:
      - name: Run Tests with PR Comments
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: your-project-id
          projects: "smoke,regression"
          github-comment: true

​
Advanced Configuration Options

The GitHub Action supports various options to customize test execution:

​
Core Options

  • api-key (required): Your Stably API key
  • project-id (required): Your Stably project ID
  • projects: Comma-separated list of Playwright projects to run (e.g., “smoke,critical”)
  • grep: Pattern to filter tests (e.g., “@p0” to run only P0 tests)

​
Execution Options

  • workers: Number of parallel workers (default: auto, max: 100)
  • retries: Number of retries for failed tests (default: 0)
  • timeout: Test timeout in milliseconds (default: 30000)

​
Reporting Options

  • github-comment: Post test results as PR comments (default: false)
  • note: Add contextual information to test runs (e.g., “PR #123”)

​
Environment Options

  • environment: Environment name for variable overrides (e.g., “STAGING”, “PRODUCTION”)
  • env-vars: JSON object with environment variable overrides

​
Complete Example

Here’s a comprehensive example showing multiple test jobs:
.github/workflows/stably.yaml
name: Stably Test Suite

on:
  pull_request:
  push:
    branches:
      - main

permissions:
  pull-requests: write
  contents: read

jobs:
  smoke-tests:
    name: Smoke Tests
    runs-on: ubuntu-latest
    steps:
      - name: Run Smoke Tests
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: ${{ secrets.STABLY_PROJECT_ID }}
          projects: "smoke"
          github-comment: true
          note: "Smoke tests for PR #${{ github.event.pull_request.number }}"

  regression-tests:
    name: Regression Tests
    runs-on: ubuntu-latest
    needs: smoke-tests
    if: github.event_name == 'push'
    steps:
      - name: Run Full Regression
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: ${{ secrets.STABLY_PROJECT_ID }}
          projects: "regression,e2e"
          workers: 50
          retries: 2
          note: "Full regression for commit ${{ github.sha }}"

  critical-path:
    name: Critical Path Tests
    runs-on: ubuntu-latest
    steps:
      - name: Run Critical Tests
        uses: stablyai/stably-runner-action@v4
        with:
          api-key: ${{ secrets.STABLY_API_KEY }}
          project-id: ${{ secrets.STABLY_PROJECT_ID }}
          grep: "@p0"
          github-comment: true

​
Test Organization

For optimal GitHub Actions integration, organize your tests using Playwright projects:
playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  projects: [
    {
      name: 'smoke',
      testMatch: '/smoke/**'
    },
    {
      name: 'regression',
      testMatch: '/regression/**'
    },
    {
      name: 'critical',
      testMatch: '/critical/**'
    }
  ]
});
Then reference these projects in your workflows: 
- uses: stablyai/stably-runner-action@v4
  with:
    api-key: ${{ secrets.STABLY_API_KEY }}
    project-id: ${{ secrets.STABLY_PROJECT_ID }}
    projects: "smoke,regression"

​
Troubleshooting

Ensure you’ve deployed your tests to Stably Cloud:
npx stably deploy
Verify the project names in your workflow match exactly what’s defined in playwright.config.ts.
Make sure you’ve set the required permissions in your workflow:
permissions:
  pull-requests: write
  contents: read

​
Next Steps

Define Test Groups

Organize tests using Playwright projects

Run on Cloud

Learn more about cloud execution

Schedule Tests

Configure automatic test schedules

Configure Notifications

Set up test result notifications