> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stably.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Tests on GitLab

> Run Stably-powered Playwright tests on GitLab using self-hosted runners or Stably Cloud via the API.

Stably provides two ways to run your Playwright tests on GitLab CI/CD:

1. **Stably Cloud**: Run tests on Stably's infrastructure via the REST API with high parallelism and AI-powered features
2. **Self-Hosted Runners**: Run tests on your own GitLab runners using the Stably SDK

Both approaches support your existing `playwright.config.ts` configuration and Playwright projects.

## Running on Stably Cloud (Recommended)

Run your tests on Stably's cloud infrastructure for maximum scalability and AI-powered features. Since GitLab doesn't have a native Stably action, you trigger cloud runs via the [REST API](/api-reference/introduction).

### Prerequisites

* Playwright tests published to Stably Cloud from the Web UI
* Stably API key from [settings page](https://app.stably.ai/settings?tab=api-key)
* GitLab repository with CI/CD enabled

### Setup

<Steps>
  <Step title="Configure GitLab Variables">
    Add your credentials as GitLab CI/CD variables:

    1. Go to `Settings > CI/CD > Variables`
    2. Add `STABLY_API_KEY` with your API key (mark as protected and masked)
    3. Add `STABLY_PROJECT_ID` with your project ID (mark as masked)
  </Step>

  <Step title="Create Pipeline Configuration">
    Create or update `.gitlab-ci.yml`:

    ```yaml .gitlab-ci.yml theme={null}
    stages:
      - test

    stably-tests:
      stage: test
      image: alpine:latest
      before_script:
        - apk add --no-cache curl jq
      script:
        - |
          # Trigger test run
          RESPONSE=$(curl -s -X POST \
            "https://api.stably.ai/v1/projects/${STABLY_PROJECT_ID}/runs" \
            -H "accept: application/json" \
            -H "Authorization: Bearer ${STABLY_API_KEY}" \
            -H "Content-Type: application/json" \
            -d '{}')

          RUN_ID=$(echo $RESPONSE | jq -r '.id')
          echo "Started test run: $RUN_ID"

          # Poll for completion
          while true; do
            STATUS_RESPONSE=$(curl -s -X GET \
              "https://api.stably.ai/v1/projects/${STABLY_PROJECT_ID}/runs/${RUN_ID}" \
              -H "Authorization: Bearer ${STABLY_API_KEY}" \
              -H "accept: application/json")

            STATUS=$(echo $STATUS_RESPONSE | jq -r '.status')

            if [ "$STATUS" = "FINISHED" ] || [ "$STATUS" = "COMPLETED" ]; then
              echo "Test run completed"

              PASSED=$(echo $STATUS_RESPONSE | jq -r '.passed // 0')
              FAILED=$(echo $STATUS_RESPONSE | jq -r '.failed // 0')
              echo "Passed: $PASSED, Failed: $FAILED"

              if [ "$FAILED" -gt 0 ]; then
                echo "❌ Some tests failed"
                exit 1
              else
                echo "✅ All tests passed"
                exit 0
              fi
            fi

            echo "Status: $STATUS — waiting..."
            sleep 30
          done
      variables:
        STABLY_API_KEY: $STABLY_API_KEY
        STABLY_PROJECT_ID: $STABLY_PROJECT_ID
    ```
  </Step>
</Steps>

### Running Specific Projects

Organize your tests using [Playwright projects](/use-cases/defining-test-groups) and run them selectively by passing `projects` and `grep` in the API request body:

```yaml .gitlab-ci.yml theme={null}
stages:
  - test

smoke-tests:
  stage: test
  image: alpine:latest
  before_script:
    - apk add --no-cache curl jq
  script:
    - |
      RESPONSE=$(curl -s -X POST \
        "https://api.stably.ai/v1/projects/${STABLY_PROJECT_ID}/runs" \
        -H "accept: application/json" \
        -H "Authorization: Bearer ${STABLY_API_KEY}" \
        -H "Content-Type: application/json" \
        -d '{"projects": ["smoke"]}')

      RUN_ID=$(echo $RESPONSE | jq -r '.id')
      echo "Started smoke test run: $RUN_ID"
      # Poll for completion (see basic example above)
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID

regression-tests:
  stage: test
  image: alpine:latest
  before_script:
    - apk add --no-cache curl jq
  script:
    - |
      RESPONSE=$(curl -s -X POST \
        "https://api.stably.ai/v1/projects/${STABLY_PROJECT_ID}/runs" \
        -H "accept: application/json" \
        -H "Authorization: Bearer ${STABLY_API_KEY}" \
        -H "Content-Type: application/json" \
        -d '{"projects": ["regression", "e2e"]}')

      RUN_ID=$(echo $RESPONSE | jq -r '.id')
      echo "Started regression test run: $RUN_ID"
      # Poll for completion (see basic example above)
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
  only:
    - main

critical-tests:
  stage: test
  image: alpine:latest
  before_script:
    - apk add --no-cache curl jq
  script:
    - |
      RESPONSE=$(curl -s -X POST \
        "https://api.stably.ai/v1/projects/${STABLY_PROJECT_ID}/runs" \
        -H "accept: application/json" \
        -H "Authorization: Bearer ${STABLY_API_KEY}" \
        -H "Content-Type: application/json" \
        -d '{"grep": "@p0"}')

      RUN_ID=$(echo $RESPONSE | jq -r '.id')
      echo "Started critical test run: $RUN_ID"
      # Poll for completion (see basic example above)
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
```

<Card title="API Reference" icon="book" href="/api-reference/introduction">
  View complete API documentation with all available endpoints, request/response schemas, and parameters
</Card>

## Running on Self-Hosted GitLab Runners

Run tests on your own GitLab runners using the Stably SDK for local execution with Stably's AI features.

### Prerequisites

* A GitLab repository with `.gitlab-ci.yml`
* Node.js 20+ configured in your pipeline
* Playwright tests in your repository

### Quick Setup

### 1. Install Stably Playwright SDK

Update your project dependencies to use Stably's Playwright SDK alongside Playwright:

<CodeGroup>
  ```bash npm theme={null}
  npm install --save-dev @playwright/test @stablyai/playwright-test
  ```

  ```bash yarn theme={null}
  yarn add --dev @playwright/test @stablyai/playwright-test
  ```

  ```bash pnpm theme={null}
  pnpm add -D @playwright/test @stablyai/playwright-test
  ```
</CodeGroup>

Commit the updated `package.json` and lock file to your repository.

### 2. Update Test Imports

Replace all imports from `@playwright/test` with `@stablyai/playwright-test` in your test files:

```typescript theme={null}
// Before
import { test, expect } from "@playwright/test";

// After
import { test, expect } from "@stablyai/playwright-test";
```

### 3. Configure GitLab CI/CD Variables

Add your Stably API key as a GitLab CI/CD variable:

1. Get your **Team API Key** from [Stably Settings](https://app.stably.ai/settings?tab=api-key)
2. Go to your repository at `Settings > CI/CD > Variables`
3. Add a variable named `STABLY_API_KEY` with your API key
4. Mark it as "Masked" to protect the secret

### 4. Update Your GitLab CI/CD Configuration

Update your pipeline to use the Stably CLI:

```yaml .gitlab-ci.yml theme={null}
test:
  script:
    - npm ci
    - npx stably test
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
```

<Info>
  Using `npx stably test` automatically injects the Stably reporter for enhanced analytics. You can also use `npx playwright test` directly if preferred.
</Info>

Run specific Playwright projects:

```yaml .gitlab-ci.yml theme={null}
test:
  script:
    - npm ci
    - npx stably test --project=smoke --project=critical
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
```

## Running Tests in Parallel (Self-Hosted)

When using self-hosted runners, leverage [GitLab's parallel keyword](https://docs.gitlab.com/ee/ci/yaml/#parallel) to distribute tests:

```yaml .gitlab-ci.yml theme={null}
stages:
  - test

test:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  script:
    - npm ci
    - npx stably test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
  timeout: 1h
```

<Note>
  For maximum parallelism (hundreds and thousands of parallel tests), use [Stably Cloud](#running-on-stably-cloud-recommended) instead of self-hosted runners.
</Note>

## Add Stably Reporter (Optional, but highly recommended)

To integrate with Stably's cloud platform and view test results in the dashboard, configure the reporter in your `playwright.config.ts`:

```typescript playwright.config.ts theme={null}
import { defineConfig } from "@playwright/test";
import { stablyReporter } from "@stablyai/playwright-test";

export default defineConfig({
  reporter: [
    ["list"],
    stablyReporter({ apiKey: process.env.STABLY_API_KEY, projectId: "your_project_id_here" }),
  ],
});
```

In your GitLab CI/CD configuration, add the environment variables:

```yaml theme={null}
test:
  script:
    - npm ci
    - npx stably test
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
```

<Note>
  * Find your **Project ID** in your project settings on the Stably Dashboard and add it to your GitLab CI/CD variables at `Settings > CI/CD > Variables`.
</Note>

Once configured, view comprehensive results in the [Stably dashboard](https://app.stably.ai):

* **Test execution timeline** showing parallel run progress
* **Screenshots and videos** for failed tests
* **AI-generated failure analysis** and suggested fixes
* **Historical trends** and flakiness detection
* **Detailed logs** from each parallel job

All test results from parallel jobs are automatically aggregated into a single unified report, making it easy to review the complete test run regardless of how many shards executed.

## Complete Pipeline Examples

<Tabs>
  <Tab title="Basic Setup">
    Minimal configuration for single-threaded execution:

    ```yaml .gitlab-ci.yml theme={null}
    stages:
      - test

    test:
      stage: test
      image: mcr.microsoft.com/playwright:v1.48.0-jammy
      script:
        - npm ci
        - npx stably test
      variables:
        STABLY_API_KEY: $STABLY_API_KEY
        STABLY_PROJECT_ID: $STABLY_PROJECT_ID
    ```
  </Tab>

  <Tab title="Parallel with Artifacts">
    Advanced configuration with test artifacts:

    ```yaml .gitlab-ci.yml theme={null}
    stages:
      - test

    test:
      stage: test
      image: mcr.microsoft.com/playwright:v1.48.0-jammy
      parallel: 4
      script:
        - npm ci
        - npx stably test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
      variables:
        STABLY_API_KEY: $STABLY_API_KEY
        STABLY_PROJECT_ID: $STABLY_PROJECT_ID
      artifacts:
        when: always
        paths:
          - playwright-report/
          - test-results/
        expire_in: 30 days
      timeout: 1h
    ```
  </Tab>

  <Tab title="Multiple Browsers">
    Run tests across different browsers in parallel:

    ```yaml .gitlab-ci.yml theme={null}
    stages:
      - test

    .test-template:
      stage: test
      image: mcr.microsoft.com/playwright:v1.48.0-jammy
      parallel: 2
      script:
        - npm ci
        - npx stably test --project=$BROWSER --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
      variables:
        STABLY_API_KEY: $STABLY_API_KEY
        STABLY_PROJECT_ID: $STABLY_PROJECT_ID
      timeout: 1h

    test:chromium:
      extends: .test-template
      variables:
        BROWSER: chromium

    test:firefox:
      extends: .test-template
      variables:
        BROWSER: firefox

    test:webkit:
      extends: .test-template
      variables:
        BROWSER: webkit
    ```
  </Tab>

  <Tab title="Matrix Jobs">
    Use GitLab's matrix feature for complex parallelization:

    ```yaml .gitlab-ci.yml theme={null}
    stages:
      - test

    test:
      stage: test
      image: mcr.microsoft.com/playwright:v1.48.0-jammy
      parallel:
        matrix:
          - BROWSER: [chromium, firefox, webkit]
            NODE_VERSION: ["18", "20"]
      before_script:
        - apt-get update && apt-get install -y curl
        - curl -fsSL https://deb.nodesource.com/setup_${NODE_VERSION}.x | bash -
        - apt-get install -y nodejs
      script:
        - npm ci
        - npx stably test --project=$BROWSER
      variables:
        STABLY_API_KEY: $STABLY_API_KEY
        STABLY_PROJECT_ID: $STABLY_PROJECT_ID
      timeout: 1h
    ```
  </Tab>

  <Tab title="With Stably Reporter">
    Complete setup with Stably reporter for dashboard integration:

    ```yaml .gitlab-ci.yml theme={null}
    stages:
      - test

    test:
      stage: test
      image: mcr.microsoft.com/playwright:v1.48.0-jammy
      parallel: 4
      script:
        - npm ci
        - npx stably test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
      variables:
        STABLY_API_KEY: $STABLY_API_KEY
        STABLY_PROJECT_ID: $STABLY_PROJECT_ID
      artifacts:
        when: always
        paths:
          - playwright-report/
          - test-results/
        expire_in: 30 days
      timeout: 1h
    ```
  </Tab>
</Tabs>

## Pipeline-Specific Configuration

### Cache Dependencies

Speed up your pipeline by caching node\_modules:

```yaml theme={null}
test:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
      - ~/.cache/ms-playwright/
  script:
    - npm ci
    - npx stably test
```

### Run Only on Specific Branches

Limit test execution to specific branches or merge requests:

```yaml theme={null}
test:
  script:
    - npx stably test
  only:
    - main
    - merge_requests
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
```

### Custom Docker Image

Use a custom Docker image with specific Playwright version:

```yaml theme={null}
test:
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel: 4
  before_script:
    - node --version
    - npm --version
  script:
    - npm ci
    - npx stably test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  variables:
    STABLY_API_KEY: $STABLY_API_KEY
    STABLY_PROJECT_ID: $STABLY_PROJECT_ID
```

## Troubleshooting

### Missing Playwright Browsers

If you see errors about missing browsers, ensure you're using the Playwright Docker image or install browsers explicitly:

```yaml theme={null}
script:
  - npm ci
  - npx stably install --with-deps
  - npx stably test
```

### Timeout Issues

For long-running test suites, increase the job timeout:

```yaml theme={null}
test:
  timeout: 2h
  script:
    - npx stably test
```

### Permission Errors

If you encounter permission errors with the cache or artifacts, add:

```yaml theme={null}
test:
  before_script:
    - chmod -R 777 node_modules/
```

## Next Steps

* Explore [AI assertions](/stably-sdk/ai-assertions) for intelligent test validation
* Set up [Slack notifications](/integrations/slack) for test results
* Configure [GitLab CI](/run-tests/gitlab-ci) for pipeline status and reporting