Anti Bot-Detection

Configure settings to handle bot detection systems that may interfere with automated testing. Stably’s AI runs in a headless browser, which is often detected as a bot by various security systems.

​

Overview

Many applications use bot detection services to protect against automated attacks. However, these same systems can interfere with legitimate automated testing. This guide covers how to configure Stably to work with common bot detection systems.

​

Cloudflare Bot Detection

If you see a Cloudflare bot detection page in your test failures, you likely have Cloudflare Bot Detection enabled.

​

Cloudflare Access Users

You can allowlist Stably’s User-Agent in your Cloudflare settings:

1. Navigate to Browser Settings in your Stably workspace
2. Configure User-Agent to use a standard browser user agent instead of the default Stably identifier
3. Update Cloudflare Settings to allowlist the configured user agent

​

Cloudflare Zero Trust Users

In Zero Trust environments, each action initiated by Stably needs to contain a service token:

​

Clerk Bot Detection

If you encounter Clerk bot detection pages during test execution, you have Clerk Bot Detection enabled in your application.

​

Solution 1: Disable Clerk Bot Detection on Staging

Recommended approach for staging environments:

• Clerk doesn’t provide granular User-Agent allowlisting
• The most effective solution is to disable Clerk Bot Detection in your staging environment
• This allows automated testing while maintaining production security

​

Solution 2: Session Token API Integration

For production testing or when bot detection must remain enabled:

Create an API endpoint that generates session tokens for Stably. This approach uses Clerk’s testing framework as described in their testing documentation.

​

Implementation Example

import { clerkClient } from "@clerk/clerk-express";

const clerk = clerkClient({
  secretKey: process.env.CLERK_SECRET_KEY,
}); 

app.post("/session", async (req, res) => {
  const { clerkToken } = req.body;
  const session = await clerk.sessions.create({
    userId: clerkToken,
  });
  res.json(session);
});

​

Configuration Steps

​

IP Allowlisting

For applications with strict bot detection that can’t be bypassed through user agent or header configuration, IP allowlisting provides a reliable solution.

​

When to Use IP Allowlisting

IP allowlisting is effective when:

• Bot detection systems are too aggressive to bypass with configuration changes
• You need guaranteed test execution without detection interference
• Your security policies allow IP-based exceptions for testing
• Other methods (user agent, headers, tokens) have proven insufficient

​

Implementation Steps

​

Configuration Examples

​

Cloudflare IP Access Rules

# Add to Cloudflare IP Access Rules
Action: Allow
IP Address: [Stably IP Range]
Zone: yourapp.com
Note: Stably AI Testing Platform

​

WAF Rule Example

{
  "action": "allow",
  "expression": "(ip.src in {192.168.1.0/24 203.0.113.0/24})",
  "description": "Allow Stably testing IPs"
}

​

Considerations and Limitations

Security Implications:

• IP allowlisting reduces security for allowlisted addresses
• Regularly review and update the allowlisted IP ranges
• Monitor traffic from allowlisted IPs for any suspicious activity

Maintenance Requirements:

• Stably IP addresses may change with infrastructure updates
• Set up monitoring to detect when allowlisted IPs become inactive
• Establish a process for updating IP allowlists when Stably provides new ranges

Network Dependencies:

• Ensure your CDN/security provider supports IP allowlisting
• Verify that allowlisting works across all geographic regions where tests run
• Test failover scenarios to ensure continued access during IP changes

​

General Bot Detection Best Practices

​

User Agent Configuration

• Use Standard User Agents: Configure a standard browser user agent instead of the default Stably identifier
• Rotate User Agents: Consider rotating between different standard user agents
• Environment-Specific: Use different user agents for different testing environments

​

HTTP Headers

Common headers that help bypass bot detection:

• Accept: Standard browser accept headers
• Accept-Language: Language preferences
• Accept-Encoding: Compression preferences
• Referer: Appropriate referer headers for navigation flows

​

Testing Strategy

1. Start with Staging: Always test bot detection configurations in staging environments first
2. Monitor Failures: Watch for bot detection patterns in test failures
3. Gradual Implementation: Implement bypasses incrementally to identify the minimal required configuration
4. Environment Separation: Use different configurations for different environments

​

Troubleshooting

​

Common Issues

Tests failing with bot detection pages:

• Verify user agent configuration
• Check HTTP headers are properly set
• Ensure service tokens are valid and not expired

Intermittent bot detection:

• Review rate limiting settings
• Consider adding delays between actions
• Check for IP-based restrictions

Configuration not taking effect:

• Verify settings are saved in Browser Settings
• Check that configurations are applied to the correct environment
• Restart test execution after configuration changes

​

Getting Help

If you continue to experience bot detection issues:

1. Document the Detection: Capture screenshots of bot detection pages
2. Gather Configuration: Note your current user agent and header settings
3. Contact Support: Reach out with your configuration details and error patterns