Skip to main content
Stably integrates seamlessly with your existing GitLab CI/CD pipelines. Use Stably’s Playwright SDK to unlock AI-powered testing capabilities including auto-heal, AI assertions, and centralized reporting—all while maintaining 100% compatibility with your existing Playwright tests.

Prerequisites

  • A GitLab repository with an existing .gitlab-ci.yml file (or the ability to create one)
  • Node.js 18+ configured in your pipeline
  • Playwright tests in your repository

Quick Setup

1. Install Stably Playwright SDK

Update your project dependencies to use Stably’s Playwright SDK instead of the standard Playwright package:
npm remove @playwright/test
npm install --save-dev @stablyai/playwright-test
Or using other package managers:
npm remove @playwright/test
npm install --save-dev @stablyai/playwright-test
Commit the updated package.json and lock file to your repository.

2. Update Test Imports

Replace all imports from @playwright/test with @stablyai/playwright-test in your test files:
// Before
import { test, expect } from "@playwright/test";

// After
import { test, expect } from "@stablyai/playwright-test";

3. Configure GitLab CI/CD Variables

Add your Stably API key as a GitLab CI/CD variable:
  1. Get your Team API Key from Stably Settings
  2. Go to your repository at Settings > CI/CD > Variables
  3. Add a variable named STABLY_API_KEY with your API key
  4. Mark it as “Masked” to protect the secret

4. Update Your GitLab CI/CD Configuration

Your pipeline remains the same—just ensure the STABLY_API_KEY environment variable is set:
.gitlab-ci.yml
test:
  script:
    - npm ci
    - npx playwright test
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
The Stably SDK is a drop-in replacement for Playwright, so you continue using npx playwright test instead of a different command. All your existing Playwright CLI flags and options work the same way.

Running Tests in Parallel

Stably works with GitLab’s parallel keyword to run your tests across multiple parallel jobs, reducing overall execution time and providing faster feedback.

Example: Parallel Execution

Here’s a complete pipeline that runs tests across 4 parallel jobs:
.gitlab-ci.yml
stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  script:
    - npm ci
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
  timeout: 1h

How Parallel Execution Works

By using GitLab’s parallel execution:
  • Multiple jobs are created automatically (4 in the example above)
  • Each job receives a unique shard of tests to execute using $CI_NODE_INDEX and $CI_NODE_TOTAL
  • Tests run simultaneously across all jobs, reducing total execution time
  • Results are collected and aggregated in the Stably dashboard
You can adjust the number of parallel jobs by changing the parallel value. For example, to run 8 parallel jobs:
test:
  parallel: 8
  script:
    - npm ci
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

Add Stably Reporter (Optional)

To integrate with Stably’s cloud platform and view test results in the dashboard, add the Stably reporter:

1. Install the Reporter

npm install --save-dev @stablyai/playwright-reporter

2. Configure in playwright.config.ts

playwright.config.ts
export default defineConfig({
  reporter: [
    ["list"],
    ["@stablyai/playwright-reporter"],
  ],
});

3. Add Environment Variables

In your GitLab CI/CD configuration, add the STABLY_PROJECT_ID variable:
test:
  script:
    - npm ci
    - npx playwright test
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
Find your Project ID in your project settings on the Stably Dashboard and add it to your GitLab CI/CD variables at Settings > CI/CD > Variables.
Once configured, view comprehensive results in the Stably dashboard:
  • Test execution timeline showing parallel run progress
  • Screenshots and videos for failed tests
  • AI-generated failure analysis and suggested fixes
  • Historical trends and flakiness detection
  • Detailed logs from each parallel job
All test results from parallel jobs are automatically aggregated into a single unified report, making it easy to review the complete test run regardless of how many shards executed.

Complete Pipeline Examples

  • Basic Setup
  • Parallel with Artifacts
  • Multiple Browsers
  • Matrix Jobs
  • With Stably Reporter
Minimal configuration for single-threaded execution:
.gitlab-ci.yml
stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  script:
    - npm ci
    - npx playwright test
  variables:
    STABLY_API_KEY: $STABLY_API_KEY

Pipeline-Specific Configuration

Cache Dependencies

Speed up your pipeline by caching node_modules:
test:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
      - ~/.cache/ms-playwright/
  script:
    - npm ci
    - npx playwright test

Run Only on Specific Branches

Limit test execution to specific branches or merge requests:
test:
  script:
    - npx playwright test
  only:
    - main
    - merge_requests
  variables:
    STABLY_API_KEY: $STABLY_API_KEY

Custom Docker Image

Use a custom Docker image with specific Playwright version:
test:
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  before_script:
    - node --version
    - npm --version
  script:
    - npm ci
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY

Troubleshooting

Missing Playwright Browsers

If you see errors about missing browsers, ensure you’re using the Playwright Docker image or install browsers explicitly:
script:
  - npm ci
  - npx playwright install --with-deps
  - npx playwright test

Timeout Issues

For long-running test suites, increase the job timeout:
test:
  timeout: 2h
  script:
    - npx playwright test

Permission Errors

If you encounter permission errors with the cache or artifacts, add:
test:
  before_script:
    - chmod -R 777 node_modules/

Next Steps

I