Alerting - Test Notifications

Overview

Stably provides flexible notification options to keep your team informed about test results. Configure notifications at the project level in your playwright.config.ts file.

Important: Notifications require running tests with npx stably test, not npx playwright test.

Configuration

Using Stably’s defineConfig

Import defineConfig from @stablyai/playwright-test to get proper TypeScript typings for Stably-specific configuration:

playwright.config.ts

import { defineConfig } from '@stablyai/playwright-test';

export default defineConfig({
  testDir: './tests',
  projects: [
    {
      name: 'smoke',
      testMatch: '/smoke/**',
      stably: {
        notifications: {
          email: {
            to: ['[email protected]']
          },
          slack: {
            channelName: '#stably-notifications'
          }
        }
      }
    }
  ]
});

Using Stably’s defineConfig is fully compatible with Playwright. We just add extra types for Stably-specific features!

Notification Types

Email Notifications

Send test results via email to team members:

playwright.config.ts

import { defineConfig } from '@stablyai/playwright-test';

export default defineConfig({
  projects: [
    {
      name: 'critical',
      testMatch: '/critical/**',
      stably: {
        notifications: {
          email: {
            to: [
              '[email protected]',
              '[email protected]'
            ]
          }
        }
      }
    }
  ]
});

Slack Notifications

Post test results to a Slack channel:

playwright.config.ts

import { defineConfig } from '@stablyai/playwright-test';

export default defineConfig({
  projects: [
    {
      name: 'smoke',
      testMatch: '/smoke/**',
      stably: {
        notifications: {
          slack: {
            channelName: '#test-results'
          }
        }
      }
    }
  ]
});

Slack integration must be configured in your Stably dashboard settings before channels will receive notifications.

Combined Notifications

Configure both email and Slack notifications:

playwright.config.ts

import { defineConfig } from '@stablyai/playwright-test';

export default defineConfig({
  projects: [
    {
      name: 'release',
      testMatch: '/release/**',
      stably: {
        notifications: {
          email: {
            to: ['[email protected]']
          },
          slack: {
            channelName: '#releases'
          }
        }
      }
    }
  ]
});

Project-Specific Notifications

Different projects can have different notification settings:

playwright.config.ts

import { defineConfig } from '@stablyai/playwright-test';

export default defineConfig({
  testDir: './tests',
  projects: [
    {
      name: 'smoke',
      testMatch: '/smoke/**',
      stably: {
        notifications: {
          slack: {
            channelName: '#smoke-tests'
          }
        }
      }
    },
    {
      name: 'critical',
      testMatch: '/critical/**',
      stably: {
        notifications: {
          email: {
            to: ['[email protected]']
          },
          slack: {
            channelName: '#critical-alerts'
          }
        }
      }
    },
    {
      name: 'regression',
      testMatch: '/regression/**',
      stably: {
        notifications: {
          email: {
            to: ['[email protected]']
          }
        }
      }
    }
  ]
});

Notification Content

Stably notifications include:

Test Run Summary

Total tests executed
Pass/fail counts
Test duration
Project name and run information

Failure Details

For failed tests:

Test name and location
Error message and stack trace
Screenshot (when available)
Link to detailed results in dashboard

Contextual Information

Commit SHA (when run from CI)
Branch name
PR number (for PR runs)
Environment information

Quick Actions

View full results in dashboard
Rerun failed tests
Compare with previous runs

Use Cases

Production Monitoring
Team Notifications
Release Validation
Developer Feedback

Alert the on-call team immediately when critical tests fail:

{
  name: 'production-critical',
  testMatch: '/critical/**',
  stably: {
    notifications: {
      email: {
        to: ['[email protected]', '[email protected]'],
        notifyOnResult: 'failures-only'
      },
      slack: {
        channelName: '#production-alerts',
        notifyOnResult: 'failures-only'
      }
    }
  }
}

Keep the entire QA team informed of regression test results:

{
  name: 'regression',
  testMatch: '/regression/**',
  stably: {
    notifications: {
      slack: {
        channelName: '#qa-team',
        notifyOnResult: 'all'
      }
    }
  }
}

Notify release managers and stakeholders of release test results:

{
  name: 'release',
  testMatch: '/release/**',
  stably: {
    notifications: {
      email: {
        to: [
          '[email protected]',
          '[email protected]'
        ],
        notifyOnStart: true,
        notifyOnResult: 'all'
      },
      slack: {
        channelName: '#releases',
        notifyOnStart: true,
        notifyOnResult: 'all'
      }
    }
  }
}

Send smoke test results to the development team:

{
  name: 'smoke',
  testMatch: '/smoke/**',
  stably: {
    notifications: {
      slack: {
        channelName: '#dev-team',
        notifyOnResult: 'failures-only'
      }
    }
  }
}

Integration Setup

Slack Integration

Install Slack App

Navigate to Integrations > Slack in your Stably dashboard

Authorize Workspace

Click “Add to Slack” and authorize Stably to post to your workspace

Invite Bot to Channels

Invite the @Stably bot to any channels you want to receive notifications

Configure in playwright.config.ts

Add the channel name to your project configuration using channelName

Run tests with stably test

Use npx stably test to run your tests

Email Setup

Email notifications work automatically. Just specify recipient email addresses in your configuration.

Best Practices

Target the Right People

Send notifications to the teams responsible for each test category

Use Separate Channels

Create dedicated Slack channels for different test types to avoid noise

Critical Tests First

Prioritize notifications for critical path and production monitoring tests

Balance Frequency

Avoid alert fatigue by configuring appropriate notification frequency

Troubleshooting

Notifications Not Working

Problem: Configured notifications are not being sentSolution:

Make sure you’re using npx stably test — notifications do not work with npx playwright test
If you haven’t logged in via stably login, verify STABLY_API_KEY and STABLY_PROJECT_ID environment variables are set
Check that the stably config is properly nested under your project configuration

Slack Notifications Not Arriving

Problem: Configured Slack channel not receiving notificationsSolution:

Verify the Stably bot is invited to the channel
Check channelName matches exactly (including #)
Ensure Slack integration is connected in dashboard
Verify the channel is public or the bot has access

Email Notifications Not Received

Problem: Email recipients not getting notificationsSolution:

Check spam/junk folders
Verify email addresses are correct in configuration
Ensure emails are properly formatted
Check email provider isn’t blocking notifications

Too Many Notifications

Problem: Team overwhelmed by notification volumeSolution:

Use notifyOnResult: 'failures-only' to only notify on failures
Disable notifyOnStart if start notifications aren’t needed
Configure notifications only for critical test projects

Configuration Reference

Notification Schema

type NotificationResultPolicy = "all" | "failures-only";

type NotificationConfig = {
  email?: {
    to: string[];                                      // Array of email addresses
    notifyOnStart?: boolean;                           // Send notification when test run starts
    notifyOnResult?: NotificationResultPolicy | null;  // "all" or "failures-only"
  } | null;
  slack?: {
    channelName: string;                               // Slack channel name (include #)
    notifyOnStart?: boolean;                           // Send notification when test run starts
    notifyOnResult?: NotificationResultPolicy;         // "all" or "failures-only"
  } | null;
};

Configuration Options

Option	Type	Default	Description
`to`	`string[]`	—	(Email only) Array of recipient email addresses
`channelName`	`string`	—	(Slack only) The Slack channel to post notifications to (include `#`)
`notifyOnStart`	`boolean`	`false`	Send a notification when the test run begins
`notifyOnResult`	`"all"` \| `"failures-only"`	`"all"`	Control when result notifications are sent: `"all"` sends for every run, `"failures-only"` sends only when tests fail

Complete Example

playwright.config.ts

import { defineConfig } from '@stablyai/playwright-test';

export default defineConfig({
  testDir: './tests',
  
  projects: [
    {
      name: 'smoke',
      testMatch: '/smoke/**',
      stably: {
        notifications: {
          slack: {
            channelName: '#smoke-tests',
            notifyOnStart: true,
            notifyOnResult: 'all'
          }
        }
      }
    },
    {
      name: 'critical',
      testMatch: '/critical/**',
      stably: {
        notifications: {
          email: {
            to: ['[email protected]'],
            notifyOnResult: 'failures-only'
          },
          slack: {
            channelName: '#critical-alerts',
            notifyOnStart: true,
            notifyOnResult: 'failures-only'
          }
        }
      }
    },
    {
      name: 'regression',
      testMatch: '/regression/**',
      stably: {
        notifications: {
          email: {
            to: ['[email protected]', '[email protected]'],
            notifyOnResult: 'all'
          }
        }
      }
    }
  ]
});

Then run tests with npx stably test.

Next Steps

Slack Integration

Set up Slack integration for your workspace

Define Test Groups

Organize tests using Playwright projects

Schedule Tests

Configure automatic test schedules

Testing Best Practices

Learn testing strategies and workflows

Getting started

Stably Web Editor

Stably CLI

Stably Cloud Runner

Stably SDK

Stably Test Reporter

Stably Classic

Troubleshooting

Alerts & Notifications

Overview

Configuration

Using Stably’s defineConfig

Notification Types

Email Notifications

Slack Notifications

Combined Notifications

Project-Specific Notifications

Notification Content

Use Cases

Integration Setup

Slack Integration

Email Setup

Best Practices

Target the Right People

Use Separate Channels

Critical Tests First

Balance Frequency

Troubleshooting

Configuration Reference

Notification Schema

Configuration Options

Complete Example

Next Steps

Slack Integration

Define Test Groups

Schedule Tests

Testing Best Practices

Getting started

Stably Web Editor

Stably CLI

Stably Cloud Runner

Stably SDK

Stably Test Reporter

Stably Classic

Troubleshooting

​Overview

​Configuration

​Using Stably’s defineConfig

​Notification Types

​Email Notifications

​Slack Notifications

​Combined Notifications

​Project-Specific Notifications

​Notification Content

​Use Cases

​Integration Setup

​Slack Integration

​Email Setup

​Best Practices

Target the Right People

Use Separate Channels

Critical Tests First

Balance Frequency

​Troubleshooting

​Configuration Reference

​Notification Schema

​Configuration Options

​Complete Example

​Next Steps

Slack Integration

Define Test Groups

Schedule Tests

Testing Best Practices

Overview

Configuration

Using Stably’s defineConfig

Notification Types

Email Notifications

Slack Notifications

Combined Notifications

Project-Specific Notifications

Notification Content

Use Cases

Integration Setup

Slack Integration

Email Setup

Best Practices

Troubleshooting

Configuration Reference

Notification Schema

Configuration Options

Complete Example

Next Steps