Stably - AI-native Testing Platform

Stably provides two ways to run your Playwright tests on GitLab CI/CD:

Stably Cloud: Run tests on Stably’s infrastructure via the REST API with high parallelism and AI-powered features
Self-Hosted Runners: Run tests on your own GitLab runners using the Stably SDK

Both approaches support your existing playwright.config.ts configuration and Playwright projects.

Running on Stably Cloud (Recommended)

Run your tests on Stably’s cloud infrastructure for maximum scalability and AI-powered features. Since GitLab doesn’t have a native Stably action, you trigger cloud runs via the REST API.

Prerequisites

Playwright tests published to Stably Cloud from the Web UI
Stably API key from settings page
GitLab repository with CI/CD enabled

Setup

Configure GitLab Variables

Add your credentials as GitLab CI/CD variables:

Go to Settings > CI/CD > Variables
Add STABLY_API_KEY with your API key (mark as protected and masked)
Add STABLY_PROJECT_ID with your project ID (mark as masked)

Create Pipeline Configuration

Create or update .gitlab-ci.yml:

.gitlab-ci.yml

stages:
  - test

stably-tests:
  stage: test
  image: alpine:latest
  before_script:
    - apk add --no-cache curl jq
  script:
    - |
      # Trigger test run
      RESPONSE=$(curl -s -X POST \
        "https://api.stably.ai/v1/projects/${STABLY_PROJECT_ID}/runs" \
        -H "accept: application/json" \
        -H "Authorization: Bearer ${STABLY_API_KEY}" \
        -H "Content-Type: application/json" \
        -d '{}')

      RUN_ID=$(echo $RESPONSE | jq -r '.id')
      echo "Started test run: $RUN_ID"

      # Poll for completion
      while true; do
        STATUS_RESPONSE=$(curl -s -X GET \
          "https://api.stably.ai/v1/projects/${STABLY_PROJECT_ID}/runs/${RUN_ID}" \
          -H "Authorization: Bearer ${STABLY_API_KEY}" \
          -H "accept: application/json")

        STATUS=$(echo $STATUS_RESPONSE | jq -r '.status')

        if [ "$STATUS" = "FINISHED" ] || [ "$STATUS" = "COMPLETED" ]; then
          echo "Test run completed"

          PASSED=$(echo $STATUS_RESPONSE | jq -r '.passed // 0')
          FAILED=$(echo $STATUS_RESPONSE | jq -r '.failed // 0')
          echo "Passed: $PASSED, Failed: $FAILED"

          if [ "$FAILED" -gt 0 ]; then
            echo "❌ Some tests failed"
            exit 1
          else
            echo "✅ All tests passed"
            exit 0
          fi
        fi

        echo "Status: $STATUS — waiting..."
        sleep 30
      done
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID

Running Specific Projects

Organize your tests using Playwright projects and run them selectively by passing projects and grep in the API request body:

.gitlab-ci.yml

stages:
  - test

smoke-tests:
  stage: test
  image: alpine:latest
  before_script:
    - apk add --no-cache curl jq
  script:
    - |
      RESPONSE=$(curl -s -X POST \
        "https://api.stably.ai/v1/projects/${STABLY_PROJECT_ID}/runs" \
        -H "accept: application/json" \
        -H "Authorization: Bearer ${STABLY_API_KEY}" \
        -H "Content-Type: application/json" \
        -d '{"projects": ["smoke"]}')

      RUN_ID=$(echo $RESPONSE | jq -r '.id')
      echo "Started smoke test run: $RUN_ID"
      # Poll for completion (see basic example above)
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID

regression-tests:
  stage: test
  image: alpine:latest
  before_script:
    - apk add --no-cache curl jq
  script:
    - |
      RESPONSE=$(curl -s -X POST \
        "https://api.stably.ai/v1/projects/${STABLY_PROJECT_ID}/runs" \
        -H "accept: application/json" \
        -H "Authorization: Bearer ${STABLY_API_KEY}" \
        -H "Content-Type: application/json" \
        -d '{"projects": ["regression", "e2e"]}')

      RUN_ID=$(echo $RESPONSE | jq -r '.id')
      echo "Started regression test run: $RUN_ID"
      # Poll for completion (see basic example above)
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
  only:
    - main

critical-tests:
  stage: test
  image: alpine:latest
  before_script:
    - apk add --no-cache curl jq
  script:
    - |
      RESPONSE=$(curl -s -X POST \
        "https://api.stably.ai/v1/projects/${STABLY_PROJECT_ID}/runs" \
        -H "accept: application/json" \
        -H "Authorization: Bearer ${STABLY_API_KEY}" \
        -H "Content-Type: application/json" \
        -d '{"grep": "@p0"}')

      RUN_ID=$(echo $RESPONSE | jq -r '.id')
      echo "Started critical test run: $RUN_ID"
      # Poll for completion (see basic example above)
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID

API Reference

View complete API documentation with all available endpoints, request/response schemas, and parameters

Running on Self-Hosted GitLab Runners

Run tests on your own GitLab runners using the Stably SDK for local execution with Stably’s AI features.

Prerequisites

A GitLab repository with .gitlab-ci.yml
Node.js 20+ configured in your pipeline
Playwright tests in your repository

Quick Setup

1. Install Stably Playwright SDK

Update your project dependencies to use Stably’s Playwright SDK alongside Playwright:

npm install --save-dev @playwright/test @stablyai/playwright-test

Commit the updated package.json and lock file to your repository.

2. Update Test Imports

Replace all imports from @playwright/test with @stablyai/playwright-test in your test files:

// Before
import { test, expect } from "@playwright/test";

// After
import { test, expect } from "@stablyai/playwright-test";

3. Configure GitLab CI/CD Variables

Add your Stably API key as a GitLab CI/CD variable:

Get your Team API Key from Stably Settings
Go to your repository at Settings > CI/CD > Variables
Add a variable named STABLY_API_KEY with your API key
Mark it as “Masked” to protect the secret

4. Update Your GitLab CI/CD Configuration

Update your pipeline to use the Stably CLI:

.gitlab-ci.yml

test:
  script:
    - npm ci
    - npx stably test
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID

Using npx stably test automatically injects the Stably reporter for enhanced analytics. You can also use npx playwright test directly if preferred.

Run specific Playwright projects:

.gitlab-ci.yml

test:
  script:
    - npm ci
    - npx stably test --project=smoke --project=critical
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID

Running Tests in Parallel (Self-Hosted)

When using self-hosted runners, leverage GitLab’s parallel keyword to distribute tests:

.gitlab-ci.yml

stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  script:
    - npm ci
    - npx stably test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
  timeout: 1h

For maximum parallelism (hundreds and thousands of parallel tests), use Stably Cloud instead of self-hosted runners.

Add Stably Reporter (Optional, but highly recommended)

To integrate with Stably’s cloud platform and view test results in the dashboard, configure the reporter in your playwright.config.ts:

playwright.config.ts

import { defineConfig } from "@playwright/test";
import { stablyReporter } from "@stablyai/playwright-test";

export default defineConfig({
  reporter: [
    ["list"],
    stablyReporter({ apiKey: process.env.STABLY_API_KEY, projectId: "your_project_id_here" }),
  ],
});

In your GitLab CI/CD configuration, add the environment variables:

test:
  script:
    - npm ci
    - npx stably test
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID

Find your Project ID in your project settings on the Stably Dashboard and add it to your GitLab CI/CD variables at Settings > CI/CD > Variables.

Once configured, view comprehensive results in the Stably dashboard:

Test execution timeline showing parallel run progress
Screenshots and videos for failed tests
AI-generated failure analysis and suggested fixes
Historical trends and flakiness detection
Detailed logs from each parallel job

All test results from parallel jobs are automatically aggregated into a single unified report, making it easy to review the complete test run regardless of how many shards executed.

Complete Pipeline Examples

Basic Setup
Parallel with Artifacts
Multiple Browsers
Matrix Jobs
With Stably Reporter

Minimal configuration for single-threaded execution:

.gitlab-ci.yml

stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  script:
    - npm ci
    - npx stably test
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID

Advanced configuration with test artifacts:

.gitlab-ci.yml

stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  script:
    - npm ci
    - npx stably test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
  artifacts:
    when: always
    paths:
      - playwright-report/
      - test-results/
    expire_in: 30 days
  timeout: 1h

Run tests across different browsers in parallel:

.gitlab-ci.yml

stages:
  - test

.test-template:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 2
  script:
    - npm ci
    - npx stably test --project=$BROWSER --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
  timeout: 1h

test:chromium:
  extends: .test-template
  variables:
    BROWSER: chromium

test:firefox:
  extends: .test-template
  variables:
    BROWSER: firefox

test:webkit:
  extends: .test-template
  variables:
    BROWSER: webkit

Use GitLab’s matrix feature for complex parallelization:

.gitlab-ci.yml

stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel:
    matrix:
      - BROWSER: [chromium, firefox, webkit]
        NODE_VERSION: ["18", "20"]
  before_script:
    - apt-get update && apt-get install -y curl
    - curl -fsSL https://deb.nodesource.com/setup_${NODE_VERSION}.x | bash -
    - apt-get install -y nodejs
  script:
    - npm ci
    - npx stably test --project=$BROWSER
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
  timeout: 1h

Complete setup with Stably reporter for dashboard integration:

.gitlab-ci.yml

stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  script:
    - npm ci
    - npx stably test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
  artifacts:
    when: always
    paths:
      - playwright-report/
      - test-results/
    expire_in: 30 days
  timeout: 1h

Pipeline-Specific Configuration

Cache Dependencies

Speed up your pipeline by caching node_modules:

test:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
      - ~/.cache/ms-playwright/
  script:
    - npm ci
    - npx stably test

Run Only on Specific Branches

Limit test execution to specific branches or merge requests:

test:
  script:
    - npx stably test
  only:
    - main
    - merge_requests
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID

Custom Docker Image

Use a custom Docker image with specific Playwright version:

test:
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  before_script:
    - node --version
    - npm --version
  script:
    - npm ci
    - npx stably test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID

Troubleshooting

Missing Playwright Browsers

If you see errors about missing browsers, ensure you’re using the Playwright Docker image or install browsers explicitly:

script:
  - npm ci
  - npx stably install --with-deps
  - npx stably test

Timeout Issues

For long-running test suites, increase the job timeout:

test:
  timeout: 2h
  script:
    - npx stably test

Permission Errors

If you encounter permission errors with the cache or artifacts, add:

test:
  before_script:
    - chmod -R 777 node_modules/

Next Steps

Explore AI assertions for intelligent test validation
Set up Slack notifications for test results
Configure GitLab CI for pipeline status and reporting

Getting started

Stably Web Editor

Stably CLI

Stably Cloud Runner

Stably SDK

Stably Test Reporter

Configuration

Troubleshooting

Migration Guides

Resources

Run Tests on GitLab

Running on Stably Cloud (Recommended)

Prerequisites

Setup

Running Specific Projects

API Reference

Running on Self-Hosted GitLab Runners

Prerequisites

Quick Setup

1. Install Stably Playwright SDK

2. Update Test Imports

3. Configure GitLab CI/CD Variables

4. Update Your GitLab CI/CD Configuration

Running Tests in Parallel (Self-Hosted)

Add Stably Reporter (Optional, but highly recommended)

Complete Pipeline Examples

Pipeline-Specific Configuration

Cache Dependencies

Run Only on Specific Branches

Custom Docker Image

Troubleshooting

Missing Playwright Browsers

Timeout Issues

Permission Errors

Next Steps

Getting started

Stably Web Editor

Stably CLI

Stably Cloud Runner

Stably SDK

Stably Test Reporter

Configuration

Troubleshooting

Migration Guides

Resources

​Running on Stably Cloud (Recommended)

​Prerequisites

​Setup

​Running Specific Projects

API Reference

​Running on Self-Hosted GitLab Runners

​Prerequisites

​Quick Setup

​1. Install Stably Playwright SDK

​2. Update Test Imports

​3. Configure GitLab CI/CD Variables

​4. Update Your GitLab CI/CD Configuration

​Running Tests in Parallel (Self-Hosted)

​Add Stably Reporter (Optional, but highly recommended)

​Complete Pipeline Examples

​Pipeline-Specific Configuration

​Cache Dependencies

​Run Only on Specific Branches

​Custom Docker Image

​Troubleshooting

​Missing Playwright Browsers

​Timeout Issues

​Permission Errors

​Next Steps

Running on Stably Cloud (Recommended)

Prerequisites

Setup

Running Specific Projects

Running on Self-Hosted GitLab Runners

Prerequisites

Quick Setup

1. Install Stably Playwright SDK

2. Update Test Imports

3. Configure GitLab CI/CD Variables

4. Update Your GitLab CI/CD Configuration

Running Tests in Parallel (Self-Hosted)

Add Stably Reporter (Optional, but highly recommended)

Complete Pipeline Examples

Pipeline-Specific Configuration

Cache Dependencies

Run Only on Specific Branches

Custom Docker Image

Troubleshooting

Missing Playwright Browsers

Timeout Issues

Permission Errors

Next Steps