Run Tests
Run Configuration
Configure Stably test groups to run on schedules, trigger from CI/CD pipelines, or execute manually with customizable test execution settings.
Stably group tests into test groups to run them together. You can configure the test group to run on a schedule, trigger it from a CI/CD pipeline, or run it manually.
Add Tests to a Test Group
To add tests to a test group, click on theAdd Tests button on a test group on the Web UI.
View Group Run Result
Results for tests in a group are reported together.Modify Group Configuration
1. Execution Options
Individual Test Retries
- Function: Number of times a test is ran again after a failure
- Default Value: 0
- Purpose: Automatically retry failed tests to handle flaky failures
Action Delay
- Function: Adds a delay to each Playwright operation
- Default Value: 750 ms
- Purpose: Slow down test execution to handle timing issues or simulate real user behavior
Test Timeout
- Function: Length of time before each test timeouts
- Default Value: 8 mins
- Purpose: Prevent tests from hanging indefinitely and ensure timely test completion
2. Parallelism Settings
Parallelism
- Function: Whether your tests are ran sequentially or in parallel
- Default Value: Manual with 1 worker
- Options:
- Auto (RECOMMENDED): Enables high parallelism with smart worker count detection. This allows up to 100 workers to run in parallel. Reach out if you need more workers.
- Manual: Choose specific number of workers
- Purpose: Optimize test execution speed by running multiple tests simultaneously
Ensure tests in your test suites are not interdependent before running in parallel
3. Setup/Teardown
Setup
- Function: Test that run before the main test suite execution
- Purpose: Initialize test environment, create test data, or set up prerequisites
Teardown
- Function: Test that run after the main test suite execution
- Purpose: Clean up test environment, remove test data, or perform cleanup tasks
Before Each
- Function: Tests that run before each individual test in the suite
- Purpose: Set up conditions specific to each test case
After Each
- Function: Tests that run after each individual test in the suite
- Purpose: Clean up after each test case
4. Environment Variables
Top-level Environment
- Function: Inherit environment variables from a selected environment
- Purpose: Use predefined environment configurations
Custom Variables
- Function: Add custom environment variables specific to this suite
- Purpose: Override or add environment variables for this test suite
5. Notifications
Notification Settings
- Options:
- Use project level settings: Use the project notification settings
- Use suite specific settings: Override the notification settings for this suite (RECOMMENDED)
- Disable notifications: No notifications will be sent
- Purpose: Configure how test results are communicated to the team
Email Notifications
- Purpose: Send email notifications for test results
Slack Notifications
- Purpose: Send Slack notifications for test results
Skipping Tests
To temporarily exclude certain tests from execution in a test suite, first navigate to the relevant test suite, then either select them and click theSkip Tests button, or click the Skip button in the test’s dropdown menu.