Alerting - Test Notifications

Overview

Stably provides flexible notification options to keep your team informed about test results. Configure notifications at the project level in your playwright.config.ts file.

Configuration

Using Stably’s defineConfig

Import defineConfig from @stablyai/playwright-test to get proper TypeScript typings for Stably-specific configuration:

playwright.config.ts

import { defineConfig } from '@stablyai/playwright-test';

export default defineConfig({
  testDir: './tests',
  projects: [
    {
      name: 'smoke',
      testMatch: '/smoke/**',
      stably: {
        notifications: {
          email: {
            to: ['[email protected]']
          },
          slack: {
            channels: ['#stably-notifications']
          }
        }
      }
    }
  ]
});

Using Stably’s defineConfig is fully compatible with Playwright. We just add extra types for Stably-specific features!

Notification Types

Email Notifications

Send test results via email to team members:

playwright.config.ts

import { defineConfig } from '@stablyai/playwright-test';

export default defineConfig({
  projects: [
    {
      name: 'critical',
      testMatch: '/critical/**',
      stably: {
        notifications: {
          email: {
            to: [
              '[email protected]',
              '[email protected]'
            ]
          }
        }
      }
    }
  ]
});

Slack Notifications

Post test results to Slack channels:

playwright.config.ts

import { defineConfig } from '@stablyai/playwright-test';

export default defineConfig({
  projects: [
    {
      name: 'smoke',
      testMatch: '/smoke/**',
      stably: {
        notifications: {
          slack: {
            channels: [
              '#test-results',
              '#engineering-alerts'
            ]
          }
        }
      }
    }
  ]
});

Slack integration must be configured in your Stably dashboard settings before channels will receive notifications.

Combined Notifications

Configure both email and Slack notifications:

playwright.config.ts

import { defineConfig } from '@stablyai/playwright-test';

export default defineConfig({
  projects: [
    {
      name: 'release',
      testMatch: '/release/**',
      stably: {
        notifications: {
          email: {
            to: ['[email protected]']
          },
          slack: {
            channels: ['#releases', '#qa-team']
          }
        }
      }
    }
  ]
});

Project-Specific Notifications

Different projects can have different notification settings:

playwright.config.ts

import { defineConfig } from '@stablyai/playwright-test';

export default defineConfig({
  testDir: './tests',
  projects: [
    {
      name: 'smoke',
      testMatch: '/smoke/**',
      stably: {
        notifications: {
          slack: {
            channels: ['#smoke-tests']
          }
        }
      }
    },
    {
      name: 'critical',
      testMatch: '/critical/**',
      stably: {
        notifications: {
          email: {
            to: ['[email protected]']
          },
          slack: {
            channels: ['#critical-alerts']
          }
        }
      }
    },
    {
      name: 'regression',
      testMatch: '/regression/**',
      stably: {
        notifications: {
          email: {
            to: ['[email protected]']
          }
        }
      }
    }
  ]
});

Notification Content

Stably notifications include:

Test Run Summary

Total tests executed
Pass/fail counts
Test duration
Project name and run information

Failure Details

For failed tests:

Test name and location
Error message and stack trace
Screenshot (when available)
Link to detailed results in dashboard

Contextual Information

Commit SHA (when run from CI)
Branch name
PR number (for PR runs)
Environment information

Quick Actions

View full results in dashboard
Rerun failed tests
Compare with previous runs

Use Cases

Production Monitoring
Team Notifications
Release Validation
Developer Feedback

Alert the on-call team immediately when critical tests fail:

{
  name: 'production-critical',
  testMatch: '/critical/**',
  stably: {
    notifications: {
      email: {
        to: ['[email protected]', '[email protected]']
      },
      slack: {
        channels: ['#production-alerts']
      }
    }
  }
}

Keep the entire QA team informed of regression test results:

{
  name: 'regression',
  testMatch: '/regression/**',
  stably: {
    notifications: {
      slack: {
        channels: ['#qa-team']
      }
    }
  }
}

Notify release managers and stakeholders of release test results:

{
  name: 'release',
  testMatch: '/release/**',
  stably: {
    notifications: {
      email: {
        to: [
          '[email protected]',
          '[email protected]'
        ]
      },
      slack: {
        channels: ['#releases']
      }
    }
  }
}

Send smoke test results to the development team:

{
  name: 'smoke',
  testMatch: '/smoke/**',
  stably: {
    notifications: {
      slack: {
        channels: ['#dev-team', '#test-results']
      }
    }
  }
}

Integration Setup

Slack Integration

Install Slack App

Navigate to Integrations > Slack in your Stably dashboard

Authorize Workspace

Click “Add to Slack” and authorize Stably to post to your workspace

Invite Bot to Channels

Invite the @Stably bot to any channels you want to receive notifications

Configure in playwright.config.ts

Add channel names to your project configuration as shown above

Email Setup

Email notifications work automatically. Just specify recipient email addresses in your configuration.

Best Practices

Target the Right People

Send notifications to the teams responsible for each test category

Use Separate Channels

Create dedicated Slack channels for different test types to avoid noise

Critical Tests First

Prioritize notifications for critical path and production monitoring tests

Balance Frequency

Avoid alert fatigue by configuring appropriate notification frequency

Troubleshooting

Slack Notifications Not Arriving

Problem: Configured Slack channels not receiving notificationsSolution:

Verify the Stably bot is invited to the channel
Check channel name matches exactly (including #)
Ensure Slack integration is connected in dashboard
Verify the channel is public or the bot has access

Email Notifications Not Received

Problem: Email recipients not getting notificationsSolution:

Check spam/junk folders
Verify email addresses are correct in configuration
Ensure emails are properly formatted
Check email provider isn’t blocking notifications

Too Many Notifications

Problem: Team overwhelmed by notification volumeSolution:

Create separate channels for different test types
Configure notifications only for critical tests
Use scheduled summaries instead of per-run alerts
Adjust notification settings per project

Configuration Reference

Notification Schema

type NotificationConfig = {
  email?: {
    to: string[];  // Array of email addresses
  };
  slack?: {
    channels: string[];  // Array of Slack channel names (include #)
  };
};

Complete Example

playwright.config.ts

import { defineConfig } from '@stablyai/playwright-test';

export default defineConfig({
  testDir: './tests',
  
  projects: [
    {
      name: 'smoke',
      testMatch: '/smoke/**',
      stably: {
        notifications: {
          slack: {
            channels: ['#smoke-tests']
          }
        }
      }
    },
    {
      name: 'critical',
      testMatch: '/critical/**',
      stably: {
        notifications: {
          email: {
            to: ['[email protected]']
          },
          slack: {
            channels: ['#critical-alerts', '#sre-team']
          }
        }
      }
    },
    {
      name: 'regression',
      testMatch: '/regression/**',
      stably: {
        notifications: {
          email: {
            to: ['[email protected]', '[email protected]']
          }
        }
      }
    }
  ]
});

Next Steps

Slack Integration

Set up Slack integration for your workspace

Define Test Groups

Organize tests using Playwright projects

Schedule Tests

Configure automatic test schedules

Testing Best Practices

Learn testing strategies and workflows

Getting started

Stably 2.0

Stably Classic

Troubleshooting

Alerts & Notifications

Overview

Configuration

Using Stably’s defineConfig

Notification Types

Email Notifications

Slack Notifications

Combined Notifications

Project-Specific Notifications

Notification Content

Use Cases

Integration Setup

Slack Integration

Email Setup

Best Practices

Target the Right People

Use Separate Channels

Critical Tests First

Balance Frequency

Troubleshooting

Configuration Reference

Notification Schema

Complete Example

Next Steps

Slack Integration

Define Test Groups

Schedule Tests

Testing Best Practices

Getting started

Stably 2.0

Stably Classic

Troubleshooting

​Overview

​Configuration

​Using Stably’s defineConfig

​Notification Types

​Email Notifications

​Slack Notifications

​Combined Notifications

​Project-Specific Notifications

​Notification Content

​Use Cases

​Integration Setup

​Slack Integration

​Email Setup

​Best Practices

Target the Right People

Use Separate Channels

Critical Tests First

Balance Frequency

​Troubleshooting

​Configuration Reference

​Notification Schema

​Complete Example

​Next Steps

Slack Integration

Define Test Groups

Schedule Tests

Testing Best Practices

Overview

Configuration

Using Stably’s defineConfig

Notification Types

Email Notifications

Slack Notifications

Combined Notifications

Project-Specific Notifications

Notification Content

Use Cases

Integration Setup

Slack Integration

Email Setup

Best Practices

Troubleshooting

Configuration Reference

Notification Schema

Complete Example

Next Steps