Integrate with Test Case Management Systems

Many teams manage their test cases in a dedicated test case management system (TCMS) like Zephyr Scale, Xray, or TestRail while using Stably to author, run, and auto-heal their Playwright tests. This guide shows how to report Stably test results back to your TCMS so that QA leads and PMs see a unified view of test health. The approach depends on where your tests execute:

Execution Model	How Results Are Available	Best Integration Path
Your own CI (GitHub Actions, GitLab, etc.)	Playwright JUnit XML + Stably Reporter	Use Playwright’s built-in JUnit reporter — most TCMS tools import JUnit XML natively
Stably Cloud	Stably REST API	Poll the API for results and push to your TCMS via a CI step or webhook script

Option 1: Running in Your Own CI with Playwright

When you run stably test or npx playwright test in your own CI environment, Playwright executes locally and you have full control over reporter output. This is the simplest integration path because most TCMS tools — including Zephyr Scale — natively import JUnit XML.

Step 1: Add the JUnit Reporter

Add Playwright’s built-in JUnit reporter alongside the Stably reporter in your playwright.config.ts:

playwright.config.ts

import { defineConfig, stablyReporter } from "@stablyai/playwright-test";

export default defineConfig({
  reporter: [
    ["list"],
    // Stably reporter — streams results to Stably dashboard
    stablyReporter({
      apiKey: process.env.STABLY_API_KEY,
      projectId: process.env.STABLY_PROJECT_ID,
    }),
    // JUnit reporter — generates XML for your TCMS
    ["junit", { outputFile: "test-results/junit-results.xml" }],
  ],
  use: {
    trace: "on",
  },
});

This produces a standard junit-results.xml file after every test run, alongside streaming results to the Stably dashboard.

Step 2: Upload Results to Zephyr Scale

After tests complete, upload the JUnit XML to Zephyr Scale using their CLI or API. Here’s an example for GitHub Actions:

Zephyr Scale CLI
Zephyr Scale API

.github/workflows/stably-zephyr.yml

name: Stably Tests → Zephyr Scale

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install dependencies
        run: npm ci

      - name: Install browsers
        run: npx stably install

      - name: Run Stably tests
        env:
          STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
          STABLY_PROJECT_ID: ${{ secrets.STABLY_PROJECT_ID }}
        run: npx stably test

      - name: Upload results to Zephyr Scale
        if: always()
        run: |
          npx @smartbear/zephyr-scale-cli \
            --mode junit \
            --projectKey "${{ vars.ZEPHYR_PROJECT_KEY }}" \
            --junitFile test-results/junit-results.xml \
            --autoCreateTestCases
        env:
          ZEPHYR_API_TOKEN: ${{ secrets.ZEPHYR_API_TOKEN }}

.github/workflows/stably-zephyr-api.yml

name: Stably Tests → Zephyr Scale (API)

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install dependencies
        run: npm ci

      - name: Install browsers
        run: npx stably install

      - name: Run Stably tests
        env:
          STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
          STABLY_PROJECT_ID: ${{ secrets.STABLY_PROJECT_ID }}
        run: npx stably test

      - name: Upload results to Zephyr Scale
        if: always()
        run: |
          curl -X POST \
            "https://api.zephyrscale.smartbear.com/v2/automations/executions/junit?projectKey=${{ vars.ZEPHYR_PROJECT_KEY }}&autoCreateTestCases=true" \
            -H "Authorization: Bearer ${{ secrets.ZEPHYR_API_TOKEN }}" \
            -F "file=@test-results/junit-results.xml"

The --autoCreateTestCases flag (or autoCreateTestCases=true query parameter) tells Zephyr to automatically create test cases in your project for any test names it hasn’t seen before. This means you don’t need to manually pre-create every test case in Zephyr — they’re created on first import.

How Test Names Map to Zephyr

Playwright’s JUnit XML uses this naming structure:

<testcase name="should complete purchase" classname="checkout.spec.ts">

Zephyr Scale uses the name attribute to match or create test cases. To keep things clean:

Use descriptive, stable test names in your Playwright tests
Avoid dynamically generated test names that change between runs
Use Playwright tags to organize tests, and map those to Zephyr folders or labels

Other TCMS Tools

The JUnit XML approach works with any TCMS that supports JUnit import:

TCMS	JUnit Import Method
Zephyr Scale	CLI tool, REST API, or Jira plugin
Xray	REST API or Jira plugin
TestRail	CLI tool (`trcli`) or API
qTest	API or Pulse integration

Option 2: Running on Stably Cloud

When tests run on Stably Cloud, you don’t have direct access to the file system where tests execute, so you can’t grab a JUnit XML file. Instead, use the Stably REST API to retrieve structured test results and push them to your TCMS.

The Stably API requires an API key. Get yours from the API Key Dashboard.

Step 1: Trigger a Cloud Run and Get the Run ID

Trigger a run using any method — the Web Editor, CLI, API, or scheduled runs. All methods produce a runId you can use to fetch results.

# Trigger via API
curl -X POST "https://api.stably.ai/v1/projects/$STABLY_PROJECT_ID/runs" \
  -H "Authorization: Bearer $STABLY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "playwrightProjectName": "my-project" }'

# Response: { "runId": "abc123" }

Step 2: Poll for Results

Poll the run status until it completes:

curl "https://api.stably.ai/v1/projects/$STABLY_PROJECT_ID/runs/$RUN_ID" \
  -H "Authorization: Bearer $STABLY_API_KEY"

Once the run finishes, the response includes detailed test results:

{
  "status": "FAILED",
  "startedAt": "2025-09-15T12:00:00.000Z",
  "finishedAt": "2025-09-15T12:02:30.000Z",
  "branchName": "main",
  "results": {
    "testCases": [
      {
        "title": "should complete purchase",
        "status": "PASSED",
        "durationMs": 12340
      },
      {
        "title": "should show error on declined card",
        "status": "FAILED",
        "durationMs": 8760
      }
    ]
  }
}

Possible test case statuses: PASSED, FAILED, TIMEDOUT, SKIPPED, INTERRUPTED, FLAKY.

Step 3: Push Results to Zephyr Scale

Write a script that maps Stably results to Zephyr’s test execution API. Here’s a complete example:

sync-to-zephyr.sh

#!/bin/bash
set -euo pipefail

STABLY_PROJECT_ID="${STABLY_PROJECT_ID:?Set STABLY_PROJECT_ID}"
STABLY_API_KEY="${STABLY_API_KEY:?Set STABLY_API_KEY}"
ZEPHYR_API_TOKEN="${ZEPHYR_API_TOKEN:?Set ZEPHYR_API_TOKEN}"
ZEPHYR_PROJECT_KEY="${ZEPHYR_PROJECT_KEY:?Set ZEPHYR_PROJECT_KEY}"
RUN_ID="${1:?Usage: $0 <runId>}"

# 1. Fetch results from Stably API
RESULTS=$(curl -sf \
  "https://api.stably.ai/v1/projects/$STABLY_PROJECT_ID/runs/$RUN_ID" \
  -H "Authorization: Bearer $STABLY_API_KEY")

STATUS=$(echo "$RESULTS" | jq -r '.status')
if [ "$STATUS" = "RUNNING" ] || [ "$STATUS" = "QUEUED" ]; then
  echo "Run $RUN_ID is still $STATUS. Wait for it to complete."
  exit 1
fi

# 2. Convert Stably results to JUnit XML
echo "$RESULTS" | jq -r '
  .results.testCases as $tests |
  ($tests | length) as $total |
  ($tests | map(select(.status == "FAILED" or .status == "TIMEDOUT")) | length) as $failures |
  "<?xml version=\"1.0\" encoding=\"UTF-8\"?>",
  "<testsuites>",
  "  <testsuite name=\"stably-cloud\" tests=\"\($total)\" failures=\"\($failures)\">",
  ($tests[] |
    if .status == "PASSED" or .status == "FLAKY" then
      "    <testcase name=\"\(.title)\" time=\"\((.durationMs // 0) / 1000)\"/>"
    else
      "    <testcase name=\"\(.title)\" time=\"\((.durationMs // 0) / 1000)\">",
      "      <failure message=\"Status: \(.status)\"/>",
      "    </testcase>"
    end
  ),
  "  </testsuite>",
  "</testsuites>"
' > /tmp/stably-junit.xml

echo "Generated JUnit XML with $(echo "$RESULTS" | jq '.results.testCases | length') test cases"

# 3. Upload to Zephyr Scale
curl -sf -X POST \
  "https://api.zephyrscale.smartbear.com/v2/automations/executions/junit?projectKey=$ZEPHYR_PROJECT_KEY&autoCreateTestCases=true" \
  -H "Authorization: Bearer $ZEPHYR_API_TOKEN" \
  -F "file=@/tmp/stably-junit.xml"

echo "Results uploaded to Zephyr Scale"

Using stably runs for Richer Data

The stably runs view --json command provides more detailed results including error messages, attempt counts, and file locations — useful if your TCMS supports richer metadata:

stably runs view <runId> --json | jq '.testCases[] | {
  title: .title,
  status: .status,
  location: .location,
  durationMs: .durationMs,
  error: .attempts[-1].errorMessage
}'

Automating the Sync in CI

Add the sync script as a post-run step in your CI pipeline:

.github/workflows/stably-cloud-zephyr.yml

name: Stably Cloud → Zephyr Scale

on:
  schedule:
    - cron: "0 2 * * *"  # nightly

jobs:
  test-and-sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Trigger Stably Cloud run
        id: trigger
        run: |
          RESPONSE=$(curl -sf -X POST \
            "https://api.stably.ai/v1/projects/${{ secrets.STABLY_PROJECT_ID }}/runs" \
            -H "Authorization: Bearer ${{ secrets.STABLY_API_KEY }}" \
            -H "Content-Type: application/json" \
            -d '{}')
          echo "run_id=$(echo $RESPONSE | jq -r '.runId')" >> "$GITHUB_OUTPUT"

      - name: Wait for completion
        run: |
          for i in $(seq 1 60); do
            STATUS=$(curl -sf \
              "https://api.stably.ai/v1/projects/${{ secrets.STABLY_PROJECT_ID }}/runs/${{ steps.trigger.outputs.run_id }}" \
              -H "Authorization: Bearer ${{ secrets.STABLY_API_KEY }}" \
              | jq -r '.status')
            echo "Status: $STATUS"
            if [ "$STATUS" != "RUNNING" ] && [ "$STATUS" != "QUEUED" ]; then
              break
            fi
            sleep 30
          done

      - name: Sync results to Zephyr
        run: bash sync-to-zephyr.sh "${{ steps.trigger.outputs.run_id }}"
        env:
          STABLY_PROJECT_ID: ${{ secrets.STABLY_PROJECT_ID }}
          STABLY_API_KEY: ${{ secrets.STABLY_API_KEY }}
          ZEPHYR_API_TOKEN: ${{ secrets.ZEPHYR_API_TOKEN }}
          ZEPHYR_PROJECT_KEY: ${{ vars.ZEPHYR_PROJECT_KEY }}

Mapping Test Statuses

Stably and Zephyr use different status vocabularies. Here’s how they map:

Stably Status	Zephyr Scale Status	Notes
`PASSED`	Pass	Direct mapping
`FAILED`	Fail	Direct mapping
`FLAKY`	Pass	Passed on retry — typically treated as pass in TCMS
`TIMEDOUT`	Fail	Test exceeded timeout
`SKIPPED`	Not Executed	Test was skipped
`INTERRUPTED`	Blocked	Run was cancelled mid-execution

When using JUnit XML import, Zephyr automatically maps <testcase> (no failure element) to Pass and <testcase> with <failure> to Fail. Skipped tests use the <skipped/> element.

Best Practices

Use Stable Test Names

Your TCMS matches test cases by name. Avoid dynamically generated names that change between runs — this creates duplicate entries in Zephyr.

Run the Sync on Every CI Run

Automate the TCMS sync so results are always up to date. Don’t rely on manual uploads.

Use autoCreateTestCases

Let Zephyr auto-create test cases on first import. This avoids the overhead of manually creating entries for every Playwright test.

Keep One Source of Truth

Author and maintain tests in Playwright with Stably. Use the TCMS as a reporting and visibility layer, not as the place where test definitions live.

Next Steps

Stably Test Reporter

Stream results to the Stably dashboard for AI-powered debugging

API Reference

Full REST API documentation for programmatic access

Run Tests on Cloud

Execute tests on Stably’s cloud infrastructure

CI/CD Integration

Set up tests in your deployment pipeline

​Option 1: Running in Your Own CI with Playwright

​Step 1: Add the JUnit Reporter

​Step 2: Upload Results to Zephyr Scale

​How Test Names Map to Zephyr

​Other TCMS Tools

​Option 2: Running on Stably Cloud

​Step 1: Trigger a Cloud Run and Get the Run ID

​Step 2: Poll for Results

​Step 3: Push Results to Zephyr Scale

​Using stably runs for Richer Data

​Automating the Sync in CI

​Mapping Test Statuses

​Best Practices

Use Stable Test Names

Run the Sync on Every CI Run

Use autoCreateTestCases

Keep One Source of Truth

​Next Steps

Stably Test Reporter

API Reference

Run Tests on Cloud

CI/CD Integration

Option 1: Running in Your Own CI with Playwright

Step 1: Add the JUnit Reporter

Step 2: Upload Results to Zephyr Scale

How Test Names Map to Zephyr

Other TCMS Tools

Option 2: Running on Stably Cloud

Step 1: Trigger a Cloud Run and Get the Run ID

Step 2: Poll for Results

Step 3: Push Results to Zephyr Scale

Using stably runs for Richer Data

Automating the Sync in CI

Mapping Test Statuses

Best Practices

Next Steps