Testing Guides
Release Testing
Run comprehensive regression test suites on staging environments before production releases
Overview
Release testing in Stably runs your existing regression test suites against staging environments before production deployments. These test suites include all of Stably’s AI-powered features like AI Auto-Heal, AI Steps, and intelligent test maintenance. Key Features:- Execute comprehensive regression test suites on staging environments
- Leverage AI Auto-Heal to automatically adapt tests to UI changes
- Use AI Steps for intelligent test interactions
- Run tests manually on-demand or automatically in CI/CD pipelines
- Get detailed test reports with failure analysis and screenshots
- Block production deployments when critical tests fail
Perfect for: Validating staging environments with the same test suites you use for production monitoring, but triggered for release validation
Setup Process
1
Create Release Test Suite
Set up a comprehensive regression test suite for release validation:
Test Selection
Test Selection
Include tests that cover:
- Critical User Journeys: Login, core workflows, checkout processes
- Core Functionality: Features that could break with new releases
- Integration Points: API endpoints, third-party services, external dependencies
- Edge Cases: Error handling, boundary conditions, negative scenarios
AI-Powered Features
AI-Powered Features
Your release test suites automatically include Stably’s AI capabilities:
- AI Auto-Heal: Tests automatically adapt to minor UI changes during releases
- AI Steps: Intelligent interactions that handle dynamic content
- AI Assertions: Smart validation that adapts to content changes
- Automatic Maintenance: Tests stay current with application changes
Test Organization
Test Organization
- Group related tests into logical test suites
- Configure setup/teardown for test data preparation
- Set appropriate timeouts for staging environment performance
- Configure retry logic for handling flaky tests
2
Configure Environment Variables
Set up environment-specific variables to target your staging environment:
Environment Configuration
Environment Configuration
- Navigate to Core Configuration > Test Data
- Create environment-specific variables:
- BASE_URL: Your staging environment URL (e.g.,
https://staging.yourapp.com) - API_ENDPOINT: Staging API endpoints
- TEST_CREDENTIALS: Staging-specific login credentials
- DATABASE_URL: Staging database connection (if needed)
- BASE_URL: Your staging environment URL (e.g.,
- Create separate environment scopes for staging vs production
- Mark sensitive data (API keys, passwords) as sensitive
3
Configure Test Suite Settings
Optimize your test suite configuration for release testing:
Execution Configuration
Execution Configuration
- Environment: Select staging environment scope
- Parallelism: Set to “Auto” for faster execution on large regression suites
- Individual Test Retries: Configure 1-2 retries for handling flaky tests
- Test Timeout: Adjust for staging performance (default: 8 minutes)
- Action Delay: Increase if staging is slower than production (default: 750ms)
AI Auto-Heal Settings
AI Auto-Heal Settings
Configure AI Auto-Heal behavior for release testing:
- Healing Sensitivity: Set appropriate level for UI changes
- Exploration Cycles: Limit cycles for faster execution
- Auto-Update: Enable to keep tests current with releases
Notifications
Notifications
Set up alerts for test results:
- Email Notifications: Alert team members of test results
- Slack Integration: Post results to team channels
- Suite-Specific Settings: Override project notifications for releases
- Failure Alerts: Immediate notification for critical failures
Running Release Tests
Manual Execution
For on-demand release validation:1
Navigate to Test Suite
Go to your release test suite in the Stably dashboard
2
Verify Configuration
Before running, confirm:
- Environment is set to staging
- All critical tests are included (not skipped)
- Test data and credentials are current
- AI Auto-Heal is properly configured
3
Execute Test Run
Click Run Tests to start execution:
- Monitor progress in real-time dashboard
- Watch AI Auto-Heal adapt to any UI changes
- Review test results as they complete
- Check failure details and screenshots
4
Review Results
Once testing completes:
- Review comprehensive test reports
- Investigate any failures with AI-generated insights
- Document findings for release decision
- Share results with stakeholders
Automated CI/CD Execution
For pipeline integration and automated release validation:1
Set Up CI Integration
Configure your CI/CD pipeline to run release tests automatically:
GitHub Actions Integration
GitHub Actions Integration
- Get your API key from Settings > API Keys
- Add
STABLY_API_KEYas a repository secret in GitHub - From your test suite page, click Add to CI to get the workflow code
- Create your release workflow (e.g.,
.github/workflows/release.yaml)
Other CI/CD Platforms
Other CI/CD Platforms
For other CI/CD platforms, use the Stably API directly:
2
Configure URL Replacement
Use URL replacement to test staging with production test suites:
URL Replacement Configuration
URL Replacement Configuration
- Original URL: The production URL configured in your test suite
- Replacement URL: Your staging environment URL
- This allows you to reuse production test suites for staging validation
- AI features work the same way on staging as production
- Ensure staging environment mirrors production configuration
3
Set Up Release Gates
Configure your pipeline to block releases on test failures:
- Make the test step required for deployment
- Configure branch protection rules
- Set up approval processes for overriding failures
- Document the release process for your team
AI-Powered Release Testing
AI Auto-Heal During Releases
AI Auto-Heal automatically adapts your tests during release validation:Automatic Adaptation
Automatic Adaptation
- UI Changes: Tests automatically adapt to button text changes, layout updates
- Dynamic Content: AI handles content that changes between staging and production
- Element Updates: Automatically finds elements when selectors change
- Workflow Changes: Adapts to modified user flows and navigation
Configuration for Releases
Configuration for Releases
- Healing Sensitivity: Set conservative healing to avoid false positives
- Exploration Limits: Balance thorough testing with execution time
- Manual Review: Configure when healing changes need approval
- Rollback Options: Revert healing changes if they cause issues
AI Steps for Complex Scenarios
AI Steps provide intelligent interactions during release testing:- Dynamic Forms: Handle forms that change structure between releases
- Complex Workflows: Navigate multi-step processes intelligently
- Conditional Logic: Adapt to different paths based on application state
- Error Handling: Intelligently handle unexpected states or errors
Release Testing Workflow
Trigger Conditions
Release tests can be triggered:On-Demand Execution
- Before major releases
- After significant staging deployments
- When investigating issues
- For compliance or audit requirements
Related Features
- PR Testing - Automated pull request validation
- Scheduled Checks - Regular automated monitoring
- AI Auto-Heal - Automatic test maintenance
- AI Step - Intelligent test interactions
- CI Integration - Setting up CI/CD integration
- Run Configuration - Test execution configuration
- Test Data - Managing test environments and variables