Stably AI - AI-Powered Testing Platform Documentation

Stably provides two ways to run your Playwright tests on GitLab CI/CD:

Stably Cloud: Run tests on Stably’s infrastructure with high parallelism and AI-powered features
Self-Hosted Runners: Run tests on your own GitLab runners using the Stably SDK

Both approaches support your existing playwright.config.ts configuration and Playwright projects.

Running on Stably Cloud (Recommended)

Run your tests on Stably’s cloud infrastructure for maximum scalability and AI-powered features.

Prerequisites

Playwright tests deployed to Stably Cloud
Stably API key from settings page
GitLab repository with CI/CD enabled

Setup

Deploy Your Tests

First, deploy your tests to Stably Cloud:

npx stably deploy

Configure GitLab Variables

Add your credentials as GitLab CI/CD variables:

Go to Settings > CI/CD > Variables
Add STABLY_API_KEY with your API key (mark as masked)
Add STABLY_PROJECT_ID with your project ID (optional, mark as masked)

Create Pipeline Configuration

Create or update .gitlab-ci.yml:

.gitlab-ci.yml

stages:
  - test

stably-tests:
  stage: test
  image: node:18
  script:
    - npx stably cloud test --project=smoke
  variables:
    STABLY_API_KEY: $STABLY_API_KEY

Running Specific Projects

Organize your tests using Playwright projects and run them selectively:

.gitlab-ci.yml

stages:
  - test

smoke-tests:
  stage: test
  image: node:18
  script:
    - npx stably cloud test --project=smoke
  variables:
    STABLY_API_KEY: $STABLY_API_KEY

regression-tests:
  stage: test
  image: node:18
  script:
    - npx stably cloud test --project=regression --project=e2e
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
  only:
    - main

critical-tests:
  stage: test
  image: node:18
  script:
    - npx stably cloud test --grep @p0
  variables:
    STABLY_API_KEY: $STABLY_API_KEY

Using the API

For more control, use Stably’s REST API:

.gitlab-ci.yml

stages:
  - test

stably-api-test:
  stage: test
  image: curlimages/curl:latest
  script:
    - |
      curl -X 'POST' \
        "https://api.stably.ai/v1/projects/${STABLY_PROJECT_ID}/runs" \
        -H "accept: application/json" \
        -H "Authorization: Bearer ${STABLY_API_KEY}" \
        -H "Content-Type: application/json" \
        -d '{
          "projects": ["smoke", "critical"],
          "grep": "@p0"
        }'
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID

Running on Self-Hosted GitLab Runners

Run tests on your own GitLab runners using the Stably SDK for local execution with Stably’s AI features.

Prerequisites

A GitLab repository with .gitlab-ci.yml
Node.js 20+ configured in your pipeline
Playwright tests in your repository

Quick Setup

1. Install Stably Playwright SDK

Update your project dependencies to use Stably’s Playwright SDK alongside Playwright:

npm install --save-dev @playwright/test @stablyai/playwright-test

Commit the updated package.json and lock file to your repository.

2. Update Test Imports

Replace all imports from @playwright/test with @stablyai/playwright-test in your test files:

// Before
import { test, expect } from "@playwright/test";

// After
import { test, expect } from "@stablyai/playwright-test";

3. Configure GitLab CI/CD Variables

Add your Stably API key as a GitLab CI/CD variable:

Get your Team API Key from Stably Settings
Go to your repository at Settings > CI/CD > Variables
Add a variable named STABLY_API_KEY with your API key
Mark it as “Masked” to protect the secret

4. Update Your GitLab CI/CD Configuration

Update your pipeline to use the Stably CLI:

.gitlab-ci.yml

test:
  script:
    - npm ci
    - npx stably test
  variables:
    STABLY_API_KEY: $STABLY_API_KEY

Using npx stably test automatically injects the Stably reporter for enhanced analytics. You can also use npx playwright test directly if preferred.

Run specific Playwright projects:

.gitlab-ci.yml

test:
  script:
    - npm ci
    - npx stably test --project=smoke --project=critical
  variables:
    STABLY_API_KEY: $STABLY_API_KEY

Running Tests in Parallel (Self-Hosted)

When using self-hosted runners, leverage GitLab’s parallel keyword to distribute tests:

.gitlab-ci.yml

stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  script:
    - npm ci
    - npx stably test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
  timeout: 1h

For maximum parallelism (up to 100 workers), use Stably Cloud instead of self-hosted runners.

Add Stably Reporter (Optional, but highly reccomended)

To integrate with Stably’s cloud platform and view test results in the dashboard, configure the reporter in your playwright.config.ts:

playwright.config.ts

import { defineConfig } from "@playwright/test";
import { stablyReporter } from "@stablyai/playwright-test";

export default defineConfig({
  reporter: [
    ["list"],
    stablyReporter({ apiKey: process.env.STABLY_API_KEY, projectId: "your_project_id_here" }),
  ],
});

In your GitLab CI/CD configuration, add the environment variables:

test:
  script:
    - npm ci
    - npx playwright test
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID

Find your Project ID in your project settings on the Stably Dashboard and add it to your GitLab CI/CD variables at Settings > CI/CD > Variables.

Once configured, view comprehensive results in the Stably dashboard:

Test execution timeline showing parallel run progress
Screenshots and videos for failed tests
AI-generated failure analysis and suggested fixes
Historical trends and flakiness detection
Detailed logs from each parallel job

All test results from parallel jobs are automatically aggregated into a single unified report, making it easy to review the complete test run regardless of how many shards executed.

Complete Pipeline Examples

Basic Setup
Parallel with Artifacts
Multiple Browsers
Matrix Jobs
With Stably Reporter

Minimal configuration for single-threaded execution:

.gitlab-ci.yml

stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  script:
    - npm ci
    - npx playwright test
  variables:
    STABLY_API_KEY: $STABLY_API_KEY

Advanced configuration with test artifacts:

.gitlab-ci.yml

stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  script:
    - npm ci
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
  artifacts:
    when: always
    paths:
      - playwright-report/
      - test-results/
    expire_in: 30 days
  timeout: 1h

Run tests across different browsers in parallel:

.gitlab-ci.yml

stages:
  - test

.test-template:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 2
  script:
    - npm ci
    - npx playwright test --project=$BROWSER --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
  timeout: 1h

test:chromium:
  extends: .test-template
  variables:
    BROWSER: chromium

test:firefox:
  extends: .test-template
  variables:
    BROWSER: firefox

test:webkit:
  extends: .test-template
  variables:
    BROWSER: webkit

Use GitLab’s matrix feature for complex parallelization:

.gitlab-ci.yml

stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel:
    matrix:
      - BROWSER: [chromium, firefox, webkit]
        NODE_VERSION: ["18", "20"]
  before_script:
    - apt-get update && apt-get install -y curl
    - curl -fsSL https://deb.nodesource.com/setup_${NODE_VERSION}.x | bash -
    - apt-get install -y nodejs
  script:
    - npm ci
    - npx playwright test --project=$BROWSER
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
  timeout: 1h

Complete setup with Stably reporter for dashboard integration:

.gitlab-ci.yml

stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  script:
    - npm ci
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
  artifacts:
    when: always
    paths:
      - playwright-report/
      - test-results/
    expire_in: 30 days
  timeout: 1h

Pipeline-Specific Configuration

Cache Dependencies

Speed up your pipeline by caching node_modules:

test:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
      - ~/.cache/ms-playwright/
  script:
    - npm ci
    - npx playwright test

Run Only on Specific Branches

Limit test execution to specific branches or merge requests:

test:
  script:
    - npx playwright test
  only:
    - main
    - merge_requests
  variables:
    STABLY_API_KEY: $STABLY_API_KEY

Custom Docker Image

Use a custom Docker image with specific Playwright version:

test:
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  before_script:
    - node --version
    - npm --version
  script:
    - npm ci
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY

Troubleshooting

Missing Playwright Browsers

If you see errors about missing browsers, ensure you’re using the Playwright Docker image or install browsers explicitly:

script:
  - npm ci
  - npx playwright install --with-deps
  - npx playwright test

Timeout Issues

For long-running test suites, increase the job timeout:

test:
  timeout: 2h
  script:
    - npx playwright test

Permission Errors

If you encounter permission errors with the cache or artifacts, add:

test:
  before_script:
    - chmod -R 777 node_modules/

Next Steps

Explore AI assertions for intelligent test validation
Set up Slack notifications for test results
Configure GitLab integration for merge request comments and pipeline status

Getting started

Stably 2.0

Stably Classic

Troubleshooting

Run Tests on GitLab

Running on Stably Cloud (Recommended)

Prerequisites

Setup

Running Specific Projects

Using the API

Running on Self-Hosted GitLab Runners

Prerequisites

Quick Setup

1. Install Stably Playwright SDK

2. Update Test Imports

3. Configure GitLab CI/CD Variables

4. Update Your GitLab CI/CD Configuration

Running Tests in Parallel (Self-Hosted)

Add Stably Reporter (Optional, but highly reccomended)

Complete Pipeline Examples

Pipeline-Specific Configuration

Cache Dependencies

Run Only on Specific Branches

Custom Docker Image

Troubleshooting

Missing Playwright Browsers

Timeout Issues

Permission Errors

Next Steps

Getting started

Stably 2.0

Stably Classic

Troubleshooting

​Running on Stably Cloud (Recommended)

​Prerequisites

​Setup

​Running Specific Projects

​Using the API

​Running on Self-Hosted GitLab Runners

​Prerequisites

​Quick Setup

​1. Install Stably Playwright SDK

​2. Update Test Imports

​3. Configure GitLab CI/CD Variables

​4. Update Your GitLab CI/CD Configuration

​Running Tests in Parallel (Self-Hosted)

​Add Stably Reporter (Optional, but highly reccomended)

​Complete Pipeline Examples

​Pipeline-Specific Configuration

​Cache Dependencies

​Run Only on Specific Branches

​Custom Docker Image

​Troubleshooting

​Missing Playwright Browsers

​Timeout Issues

​Permission Errors

​Next Steps

Running on Stably Cloud (Recommended)

Prerequisites

Setup

Running Specific Projects

Using the API

Running on Self-Hosted GitLab Runners

Prerequisites

Quick Setup

1. Install Stably Playwright SDK

2. Update Test Imports

3. Configure GitLab CI/CD Variables

4. Update Your GitLab CI/CD Configuration

Running Tests in Parallel (Self-Hosted)

Add Stably Reporter (Optional, but highly reccomended)

Complete Pipeline Examples

Pipeline-Specific Configuration

Cache Dependencies

Run Only on Specific Branches

Custom Docker Image

Troubleshooting

Missing Playwright Browsers

Timeout Issues

Permission Errors

Next Steps