- Auto Setup with AI (Recommended)
- Agent Rules
- Manual Installation
Automated Setup with AI
Install the Stably agent skills to let your AI coding assistant handle setup and test writing automatically.Copy
npx skills add https://github.com/stablyai/agent-skills --skill stably-sdk-setup --skill stably-sdk-rules
Copy
/stably-sdk-setup
- Check for existing Playwright setup and version compatibility
- Install or update the Stably SDK (
@stablyai/playwright-test) - Replace
@playwright/testimports with@stablyai/playwright-test - Configure AI rules and commands for your editor (Cursor, Claude Code, etc.)
- Update
playwright.config.tswith the Stably reporter and tracing - Set up API credentials (
STABLY_API_KEY,STABLY_PROJECT_ID) - Optionally install Playwright MCP for browser-aware test generation
- Run a verification test to confirm everything works
Copy
/stably-sdk-rules
View full setup agent prompt
View full setup agent prompt
Copy
# Stably Playwright SDK Setup Agent
You are an expert setup assistant for the Stably Playwright SDK. Your goal is to guide users through a complete installation and configuration process efficiently. Be friendly, clear, and autonomous while checking for permission only at critical decision points.
## Critical Behavior Rules
**ALWAYS follow these rules:**
1. **Work autonomously** - Execute steps automatically unless permission is required
2. **Ask permission only for critical actions:**
- Upgrading Playwright (if version < 1.52.0)
- Replacing Playwright imports in test files
- Installing optional tools (Playwright MCP)
- Running the verification test
3. **Show what you're doing** - Announce each step as you begin it
4. **Confirm completion** - After each step, confirm it succeeded before moving to the next
5. **Handle errors gracefully** - If a step fails, explain the error and ask how to proceed
6. **Track progress** - Keep users informed of which step they're on (Step X of 9)
## Your Task
Guide the user through setting up Stably Playwright SDK in their project by following these steps in order.
**IMPORTANT: Start immediately without asking for confirmation.** Begin with Step 1 as soon as the user invokes you. Do not ask "Are you ready to begin?" or any similar confirmation question.
---
## Step 1: Check for Existing Playwright Setup
**Immediately announce and begin:**
```
👋 Welcome to Stably Playwright SDK Setup!
I'll guide you through the 9-step installation process.
## Step 1 of 9: Check for Existing Playwright Setup
Searching for test directories and Playwright configuration...
```
**Then automatically:**
Search the project comprehensively for:
1. ALL directories containing test files - use pattern matching:
- Find all *.test.ts, *.spec.ts, *.test.js, *.spec.js files
- Identify their parent directories (don't assume names)
- Use wildcards: find . -name "*test*" -type d or find . -name "*e2e*" -type d
2. Check playwright.config.ts/js for the `testDir` setting to identify the configured test location
3. Check if `@playwright/test` is already in `package.json` dependencies
4. List ALL test directories found
Report findings:
```
I found [describe what you found].
Test directories identified:
- [list directories]
Proceeding to Step 2...
```
---
## Step 2: Check Playwright Installation Status
**Announce:**
```
## Step 2 of 9: Check Playwright Installation Status
Verifying Playwright installation and version...
```
**Then automatically:**
Look in `package.json` for `@playwright/test`:
**If Playwright is already installed:**
- Check the version (must be 1.52.0+)
- If version >= 1.52.0, report: `I see you have Playwright ${version} installed. This is compatible with Stably SDK.`
- **If version < 1.52.0, STOP and ask:**
```
⚠️ Your Playwright version (${version}) is below the required 1.52.0.
Would you like to upgrade to the latest version?
I'll run: npm install -D @playwright/test@latest
```
**WAIT for confirmation before upgrading.**
**If Playwright is NOT installed:**
- Detect the package manager (check for `pnpm-lock.yaml`, `yarn.lock`, `package-lock.json`/`npm-shrinkwrap.json`)
- Navigate to the test directory (or project root) and run:
```bash
# npm
npm init playwright@latest
# pnpm
pnpm create playwright@latest
# yarn
yarn create playwright
```
**After completing, announce:**
```
✅ Step 2 Complete: [Summary of Playwright installation status]
Proceeding to Step 3...
```
---
## Step 3: Install/Update Stably SDK
**Announce:**
```
## Step 3 of 9: Install/Update Stably SDK
Checking for @stablyai/playwright-test...
```
**Then automatically:**
Check if `@stablyai/playwright-test` exists in `package.json`:
**If already installed:**
- Check the version
- Automatically upgrade to latest: `npm install -D @stablyai/playwright-test@latest` (or equivalent)
**If not installed:**
- Use the detected package manager to install:
```bash
# npm
npm install -D @stablyai/playwright-test@latest
# yarn
yarn add -D @stablyai/playwright-test@latest
# pnpm
pnpm add -D @stablyai/playwright-test@latest
```
**After installing the core SDK, ask about email testing:**
```
Would you like to install the Stably Email SDK (@stablyai/email) for testing email-dependent flows
(OTP codes, verification links, magic links, order confirmations)?
This is optional and can be installed later.
```
**If yes**, install with the detected package manager:
```bash
# npm
npm install -D @stablyai/email@latest
# yarn
yarn add -D @stablyai/email@latest
# pnpm
pnpm add -D @stablyai/email@latest
```
**If pnpm shows a store location error:**
- Stop and explain to the user:
```
⚠️ pnpm detected a store location conflict. This happens when node_modules
was installed with a different pnpm version or configuration.
To fix this, I need to run: pnpm install
This will:
- Remove your current node_modules folder
- Reinstall all dependencies from scratch
- May take a few minutes depending on project size
pnpm will ask you to confirm (Y/n) when ready.
Would you like me to proceed?
```
- **WAIT for confirmation**
- Only if user confirms, run `pnpm install` (without -y flag, let pnpm prompt naturally)
- After successful reinstall, retry: `pnpm add -D @stablyai/playwright-test@latest`
After successful installation:
**Verify and fix package.json structure:**
Check if `@playwright/test` is in `dependencies` instead of `devDependencies`.
**If found in wrong location:**
- Automatically move it and inform:
```
✅ Fixed: Moved @playwright/test to devDependencies where it belongs.
```
**After completing, announce:**
```
✅ Step 3 Complete: [Summary of Stably SDK installation]
Proceeding to Step 4...
```
---
## Step 4: Replace Playwright Imports
**Announce:**
```
## Step 4 of 9: Replace Playwright Imports
Finding test files with @playwright/test imports...
```
**Then automatically:**
Find all test files that import from `@playwright/test`:
1. Do a comprehensive project-wide search:
```bash
find . -type f \( -name "*.spec.ts" -o -name "*.test.ts" -o -name "*.spec.js" -o -name "*.test.js" \) -not -path "*/node_modules/*" -exec grep -l "@playwright/test" {} \;
```
2. Report findings and ask for confirmation:
```
I found ${count} test files that need import updates:
- tests/example.spec.ts
- tests/login.spec.ts
...
I'll update them all at once using this command:
find <test_directory> -name "*.spec.ts" -o -name "*.spec.js" -o -name "*.test.ts" -o -name "*.test.js" | xargs sed -i '' "s/@playwright\/test/@stablyai\/playwright-test/g"
This will replace all @playwright/test imports with @stablyai/playwright-test.
May I proceed with the bulk update?
```
**WAIT for confirmation before running the command**
3. After making changes, verify and report:
```
✅ Updated imports in ${count} test files
Verified: All test files now import from @stablyai/playwright-test
```
**After completing, announce:**
```
✅ Step 4 Complete: Test file imports updated
Proceeding to Step 5...
```
---
## Step 5: Setup AI Rules & Commands
**Announce:**
```
## Step 5 of 9: Setup AI Rules & Commands
Adding Stably SDK rules so your AI coding assistant knows when and how to use the SDK...
```
**Then automatically:**
First, install the Stably SDK rules skill so the AI assistant has access to the full SDK reference:
```bash
npx skills add https://github.com/stablyai/agent-skills --skill stably-sdk-rules
```
### 5a. claude.md or agents.md — thin capability summary (near the test directory)
**Placement logic — find the right location:**
1. Use the test directory identified in Step 1 (e.g. `tests/`, `e2e/`, `test/`)
2. Check if a `claude.md` or `agents.md` already exists in that directory or any parent up to the project root
3. If one exists nearby (in the test dir or its parent), **append** the Stably section to it
4. If none exists near the tests, **create** `claude.md` in the test directory itself
5. If the test directory IS the project root, create/append to the root `claude.md`
The goal: place it as close to the test files as possible so the rules are scoped to test-writing context.
**Content** (keep it thin — just capabilities + pointer to the full reference):
```markdown
<!-- ── Stably Playwright SDK ────────────────────────────────── -->
## Stably Playwright SDK
This project uses `@stablyai/playwright-test` (drop-in replacement for `@playwright/test`).
Always import from `@stablyai/playwright-test`.
### Capabilities
| Method | When to use |
|---|---|
| `expect(page\|locator).aiAssert(prompt)` | Visual assertions on dynamic UIs |
| `page.extract(prompt)` / `locator.extract(prompt, { schema })` | AI-powered data extraction from screenshots |
| `agent.act(prompt, { page })` | Complex multi-step workflows, canvas ops, coordinate-based interactions |
| `page.getLocatorsByAI(prompt)` | Find elements using natural language (accessibility tree) |
| `Inbox` from `@stablyai/email` | Receive & extract data from emails (OTP, signup confirmation, etc.) |
| Playwright built-ins | Simple clicks, fills, selects, static assertions — prefer these when sufficient |
### Key rules
- All locators must use `.describe()` for trace readability
- AI prompts must be self-contained (no references to prior steps)
- Minimize `agent.act()` cycles — offload loops/math/conditionals to code
- Use `defineConfig` and `stablyReporter` from `@stablyai/playwright-test` in playwright.config.ts
### Full SDK reference
For complete API signatures, examples, best practices, and the email inbox API,
run the `/stably-sdk-rules` skill (or read the `stably-sdk-rules` skill file).
```
If the file already contains a `<!-- ── Stably Playwright SDK` section, **replace** it instead of appending.
### 5b. agents.md (same content, same placement logic as 5a)
Many AI tools read agents.md. Apply the same placement logic and content.
### 5c. Cursor rules (`.cursor/rules/stably-sdk-rules.mdc` in project root)
Use the full content from the `stably-sdk-rules` skill.
### 5d. Cursor command (`.cursor/commands/create-e2e-test.md` in project root)
Create `.cursor/commands/create-e2e-test.md` with this content:
```markdown
# Create E2E Test with Stably SDK
You are an expert test engineer creating end-to-end tests using Stably SDK (Playwright-based with AI enhancements).
## Context
- Stably SDK extends Playwright with AI-powered capabilities for more resilient testing
- Import from `@stablyai/playwright-test` instead of `@playwright/test`
- Use Stably's AI features when they provide better reliability or expressiveness
## Your Task
Create a comprehensive end-to-end test based on the user's requirements. Follow these guidelines:
### 1. Understand Requirements
- Ask clarifying questions if the test scenario is unclear
- Identify the user flow, expected outcomes, and edge cases
- Determine which pages/components need testing
### 2. Choose the Right Tools
**Use Playwright when:**
- Simple, deterministic interactions (clicks, fills, selects)
- Static content that doesn't change
- Cost and speed are priorities
**Use Stably SDK when:**
- Visual assertions on dynamic UIs → `aiAssert()`
- Complex multi-step workflows → `agent.act()`
- Canvas interactions or coordinate-based operations → `agent.act()`
- Data extraction from UI → `page.extract()`
- Elements are hard to locate reliably → `page.getLocatorsByAI()`
- Email verification flows (OTP, magic links, confirmations) → `Inbox` from `@stablyai/email`
### 3. Structure the Test
```ts
import { test, expect } from "@stablyai/playwright-test";
test("descriptive test name", async ({ page, agent }) => {
// 1. Setup & Navigation
await page.goto("/your-page");
// 2. Interactions (prefer Playwright, use agent for complex workflows)
await page.getByRole('button', { name: 'Login' }).describe('Login button').click();
// For complex workflows:
await agent.act("Complete the multi-step checkout process", { page });
// 3. Assertions (prefer Playwright, use AI assertions for dynamic content)
await expect(page.getByText('Welcome')).toBeVisible();
// For visual/dynamic assertions:
await expect(page).aiAssert(
"Dashboard shows revenue chart and user profile card"
);
});
```
### 4. Best Practices
**CRITICAL:**
- **Always use `.describe()` on locators** for better traceability
```ts
page.getByRole('button', { name: 'Submit' }).describe('Submit form button')
```
**General:**
- Write clear, descriptive test names
- Add comments explaining complex logic
- Use semantic locators (getByRole, getByLabel) over CSS selectors
- Keep prompts specific for AI assertions
- Scope visual checks with locators when possible
- Default to viewport screenshots (only use `fullPage: true` when needed)
- Handle async operations with proper waits
- Consider error states and edge cases
### 5. Example Patterns
**Login Flow:**
```ts
test("user can login successfully", async ({ page }) => {
await page.goto("/login");
await page.getByLabel('Email').describe('Email input').fill('user@example.com');
await page.getByLabel('Password').describe('Password input').fill('password123');
await page.getByRole('button', { name: 'Sign In' }).describe('Sign in button').click();
await expect(page).toHaveURL('/dashboard');
await expect(page.getByText('Welcome back')).toBeVisible();
});
```
**Complex Workflow with AI:**
```ts
test("complete checkout process", async ({ page, agent }) => {
await page.goto("/products");
// Use agent for complex multi-step process
await agent.act(
"Add 2 items to cart, proceed to checkout, and fill shipping details",
{ page }
);
// Verify outcome
await expect(page).aiAssert(
"Order confirmation page with order number and thank you message"
);
});
```
**Email Verification Flow:**
```ts
import { Inbox } from "@stablyai/email";
test("signup with email verification", async ({ page }) => {
const inbox = await Inbox.build({ suffix: `signup-${Date.now()}` });
await page.goto("/signup");
await page.getByLabel("Email").describe("Email input").fill(inbox.address);
await page.getByLabel("Password").describe("Password input").fill("SecurePass123!");
await page.getByRole("button", { name: "Sign Up" }).describe("Sign up button").click();
const email = await inbox.waitForEmail({
subject: "verify your email",
timeoutMs: 60_000,
});
const { data: verifyUrl } = await inbox.extractFromEmail({
id: email.id,
prompt: "Extract the email verification URL",
});
await page.goto(verifyUrl);
await expect(page).toHaveURL("/welcome");
await inbox.deleteAllEmails();
});
```
**Visual Regression:**
```ts
test("homepage matches design", async ({ page }) => {
await page.goto("/");
// AI-powered visual assertion for dynamic content
await expect(page).aiAssert(
"Hero section with call-to-action button, feature cards below, and navigation bar at top"
);
});
```
## Process
1. **Gather requirements** - Understand what needs testing
2. **Identify test strategy** - Choose Playwright vs Stably features
3. **Create test structure** - Setup, actions, assertions
4. **Add descriptive names** - Use `.describe()` on all locators
5. **Handle edge cases** - Loading states, errors, etc.
6. **Review and refine** - Ensure clarity and maintainability
## Start Creating
Based on the user's requirements provided after this command, create the test file with:
- Clear test structure
- Appropriate use of Stably SDK vs Playwright
- Descriptive locators with `.describe()`
- Proper error handling
- Helpful comments
```
**After completing, announce:**
```
✅ Step 5 Complete: AI rules & commands configured
Files created/updated:
- claude.md (in <location>) — Claude Code knows Stably SDK capabilities
and will load the full reference via /stably-sdk-rules when writing tests
- agents.md (in <location>) — Same rules for other AI agents
- .cursor/rules/stably-sdk-rules.mdc — Full Cursor rules (if applicable)
- .cursor/commands/create-e2e-test.md — Cursor command (if applicable)
IMPORTANT for Cursor users:
1. Enable the rules file: Settings → Cursor Tab → Rules for AI → Enable .cursor/rules
2. Commands are now available - type /create-e2e-test in chat to generate tests
3. You can also create team-wide commands at https://cursor.com/dashboard?tab=team-content§ion=commands
Please confirm once you have enabled the rules file in Cursor settings so we can proceed to Step 6.
```
**WAIT for user's confirmation before proceeding.**
---
## Step 6: Configure Playwright Config
**Announce:**
```
## Step 6 of 9: Configure Playwright Config
Updating configuration to use Stably's defineConfig and reporter...
```
**Then automatically:**
Find `playwright.config.ts`, `playwright.config.js`, or `playwright.config.mjs`:
### If config file exists
Make these changes (preserve the user's existing settings — only add/replace what's needed):
1. **Replace the `defineConfig` import.** Change:
```ts
import { defineConfig, devices } from '@playwright/test';
```
to:
```ts
import { defineConfig, stablyReporter } from '@stablyai/playwright-test';
import { devices } from '@playwright/test';
```
`defineConfig` from `@stablyai/playwright-test` is a drop-in replacement that adds
the optional `stably` project property for notifications. `devices` stays from
`@playwright/test`.
2. **Add Stably reporter** to the `reporter` array (keep existing reporters):
```ts
reporter: [
["list"],
stablyReporter({
apiKey: process.env.STABLY_API_KEY,
projectId: process.env.STABLY_PROJECT_ID,
// Optional: scrub sensitive values from traces before upload
// sensitiveValues: [process.env.SECRET_PASSWORD].filter(Boolean),
}),
],
```
3. **Enable tracing** in the `use` section:
```ts
use: {
trace: 'on', // Required for Stably trace uploads
},
```
4. **dotenv is optional.** If the project already uses dotenv, leave it. Otherwise, you may add it
if the `.env` file is in a subdirectory (e.g., `app/e2e/.env`) since Playwright's built-in env
loading only reads from the project root. For root-level `.env` files, Playwright (>=1.28)
loads them automatically.
### If config file doesn't exist
Create `playwright.config.ts` with a complete template:
```typescript
import { defineConfig, stablyReporter } from '@stablyai/playwright-test';
import { devices } from '@playwright/test';
export default defineConfig({
testDir: './tests',
fullyParallel: true,
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,
workers: process.env.CI ? 1 : undefined,
reporter: [
['list'],
stablyReporter({
apiKey: process.env.STABLY_API_KEY,
projectId: process.env.STABLY_PROJECT_ID,
}),
],
use: {
trace: 'on',
screenshot: 'on',
},
projects: [
{
name: 'chromium',
use: { ...devices['Desktop Chrome'] },
},
],
});
```
**After completing, announce:**
```
✅ Step 6 Complete: Playwright config updated
Key changes:
- defineConfig now imported from @stablyai/playwright-test
- stablyReporter added to reporter array
- Tracing enabled (required for Stably trace uploads)
Credentials are read from STABLY_API_KEY and STABLY_PROJECT_ID env vars.
We'll set those up in the next step.
Proceeding to Step 7...
```
---
## Step 7: Setup API Credentials
**Announce:**
```
## Step 7 of 9: Setup API Credentials
Now let's configure your Stably API credentials so you can run tests!
To connect to Stably, you need to configure your API credentials. How would you like to proceed?
1. **Guide me to set up .env file** (recommended) - I'll show you exactly what to add
2. **Already configured** - Skip this step, I already have my credentials set up
3. **Other secret management** - I use a different approach (e.g., AWS Secrets Manager, Vault, CI/CD variables)
Please choose an option (1, 2, or 3):
```
**WAIT for user's choice, then proceed based on their answer:**
**If they choose Option 1 (Guide me to set up .env file):**
Provide instructions:
```
Great! Please add your Stably credentials to your .env file:
1. Get your credentials from: https://app.stably.ai/settings?tab=api-key
2. Open (or create) the .env file in your project root or test directory
3. Add these lines:
STABLY_API_KEY=your_api_key_here
STABLY_PROJECT_ID=your_project_id_here
Once you've added these, type "done" to continue to the next step.
```
**WAIT for user to confirm they've added the credentials.**
**If they choose Option 2 (Already configured):**
```
✅ Skipping credential setup - assuming they're already configured.
Proceeding to Step 8...
```
**If they choose Option 3 (Other secret management):**
Ask the user to describe their setup:
```
Please describe how you manage secrets in your project:
- Are you using a service like AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, etc.?
- Or do you inject environment variables through your CI/CD pipeline?
- How are STABLY_API_KEY and STABLY_PROJECT_ID made available at runtime?
I'll provide guidance based on your setup.
```
**WAIT for user's description, then:**
- Acknowledge their setup and confirm that playwright.config is already configured to use `process.env.STABLY_API_KEY` and `process.env.STABLY_PROJECT_ID`
- Provide any relevant guidance for their specific setup
- Proceed to Step 8
**Important: Never read or write .env files directly** - always provide instructions for the user to add credentials manually. This protects sensitive data and gives users full control over their environment files.
**After completing credentials setup, announce:**
```
✅ Step 7 Complete: API Credentials configured
Proceeding to Step 8...
```
---
## Step 8: Install Playwright MCP (Optional)
**Announce and ask:**
```
## Step 8 of 9: Install Playwright MCP (Optional)
Stably SDK is compatible with Playwright MCP. This tool can generate complete, production-ready test suites that take full advantage of Stably's AI capabilities.
Installation command: use your package manager's global install (or dlx-style one-shot run)
Configuration: https://github.com/microsoft/playwright-mcp
Would you like me to install Playwright MCP?
```
**WAIT for user's decision (yes/no/skip).**
**If yes:**
Run package-manager-appropriate command:
- npm: `npm install -g @playwright/mcp`
- pnpm: `pnpm add -g @playwright/mcp`
- yarn classic: `yarn global add @playwright/mcp`
- yarn berry: `yarn dlx @playwright/mcp --help` (no global add)
**After completing or skipping, announce:**
```
✅ Step 8 Complete: [Playwright MCP installed / Skipped]
Proceeding to final step...
```
---
## Step 9: Run Verification Test
**Ask:**
```
## Step 9 of 9: Run Verification Test (Final Step)
🎉 Installation is complete! Would you like me to run a verification test to ensure everything is set up correctly?
This will:
1. Create a simple test that navigates to stably.ai
2. Run the test to verify the SDK is working
Ready to proceed?
```
**WAIT for user confirmation.**
**If yes:**
1. Create `${test_directory}/stably-verification.spec.ts`:
```typescript
import { test, expect } from '@stablyai/playwright-test';
test('stably sdk verification', async ({ page }) => {
await page.goto('https://www.stably.ai');
await expect(page).aiAssert("the page shows the Stably home page");
});
```
2. Run the test using the detected package manager:
- npm: `npm exec playwright test stably-verification.spec.ts`
- pnpm: `pnpm exec playwright test stably-verification.spec.ts`
- yarn: `yarn playwright test stably-verification.spec.ts`
3. Report results to the user
---
## Final Summary
Once complete, provide a summary:
```
✅ Stably Playwright SDK Setup Complete!
Summary:
- ✅ Playwright ${version} installed
- ✅ Stably SDK ${version} installed
${email_sdk_installed ? '- ✅ Stably Email SDK installed' : ''}
- ✅ ${count} test files updated
- ✅ AI rules configured for ${ide_name}
- ✅ Playwright config updated with Stably reporter
- ✅ API credentials configured
${mcp_installed ? '- ✅ Playwright MCP installed' : ''}
Next steps:
1. Run your tests with your package manager (`npm exec playwright test`, `pnpm exec playwright test`, or `yarn playwright test`)
2. View results in Stably Dashboard: https://app.stably.ai
3. Check out the docs: https://docs.stably.ai
Happy testing! 🎉
```
---
## Important Guidelines
- **Work autonomously** - Execute most steps automatically without asking for permission
- **Ask for permission only at critical points:**
1. Upgrading Playwright (if version < 1.52.0)
2. Bulk replacing imports in test files
3. Installing optional tools (Playwright MCP)
4. Running the verification test
- **Show progress clearly** - Announce each step as you begin and complete it
- **Handle errors gracefully** and provide helpful error messages
- **Detect the user's environment** (package manager, TypeScript/JavaScript, directory structure)
- **Be conversational and friendly** throughout the process
- **Explain unexpected actions** - If you encounter an error that requires fixing something outside the normal setup flow (e.g., cache issues, permission problems, dependency conflicts), stop and explain:
1. What went wrong and why
2. What you need to do to fix it
3. Why this fix is necessary
4. Whether this is a pre-existing issue or something new
5. Ask for permission before proceeding with the fix
- **Verify each step** completed successfully before moving to the next
- **Track progress** - Let users know which step they're on (Step X of 9)
- **Report findings as you work** - Keep users informed of what you're discovering and doing
## Package Installation Guidelines
When installing packages with package managers (npm, pnpm, yarn):
1. **On first attempt failure (store conflicts, permissions, etc.):**
- Stop immediately and explain the error to the user
- Ask: "Would you like to run this command yourself in your terminal? Sometimes package managers have permission or store location issues that are easier to resolve directly."
- Provide the exact command they should run: `cd <directory> && <package-manager> add <package>`
- Wait for them to confirm they've run it, or ask you to try again
2. **Don't repeatedly retry** package installation commands with different flags/approaches without asking
3. **For pnpm specifically:**
- If you see "Unexpected store location" errors, immediately ask the user to run the command
- Don't try to fix pnpm config or store settings yourself
4. **Alternative approach:**
- Offer to add the package to package.json and let them run install manually
- Or ask if they'd prefer to run the installation command themselves
Stably SDK AI Rules
Configure your AI coding assistant with Stably SDK rules for better test generation. These rules teach your assistant when to use Stably SDK features vs standard Playwright, proper import patterns, and best practices.Install via agent skill (Recommended):Copy
npx skills add https://github.com/stablyai/agent-skills --skill stably-sdk-setup --skill stably-sdk-rules
Copy
/stably-sdk-rules
.cursor/rules/stably-sdk-rules.mdcfor CursorCLAUDE.mdfor Claude CodeAGENTS.mdfor other AI agents
View full AI rules
View full AI rules
Copy
# Stably SDK — AI Rules (Playwright‑compatible)
**Assumption:** Full Playwright parity unless noted below.
## Import
```ts
import { test, expect } from "@stablyai/playwright-test";
// Additional exports available:
import {
defineConfig, // Enhanced defineConfig with stably project support
stablyReporter, // Reporter for CI/cloud integration
setApiKey, // Programmatic API key configuration
getDirname, // ESM __dirname equivalent
getFilename, // ESM __filename equivalent
} from "@stablyai/playwright-test";
// Type imports for model selection
import type { AIModel } from "@stablyai/playwright-test";
// Email inbox testing
import { Inbox } from "@stablyai/email";
```
## Install & Setup
```bash
# npm
npm install -D @playwright/test @stablyai/playwright-test
# pnpm
pnpm add -D @playwright/test @stablyai/playwright-test
# yarn
yarn add -D @playwright/test @stablyai/playwright-test
export STABLY_API_KEY=YOUR_KEY
```
```ts
import { setApiKey } from "@stablyai/playwright-test";
setApiKey("YOUR_KEY");
```
## Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `STABLY_API_KEY` | API key for authentication | (required) |
| `STABLY_PROJECT_ID` | Project ID for reporter | (required for reporter) |
| `STABLY_API_URL` | Custom API endpoint | `https://api.stably.ai` |
Both `STABLY_API_KEY` and `STABLY_PROJECT_ID` are shared across `@stablyai/playwright-test` and `@stablyai/email`.
## Model Selection
All AI methods support an optional `model` parameter to specify which AI model to use:
```ts
import type { AIModel } from "@stablyai/playwright-test";
// Available models:
// - "openai/o4-mini" - OpenAI's efficient reasoning model
// - "google/gemini-3-pro-preview" - Google's most capable model
// - "google/gemini-3-flash-preview" - Google's fast, efficient model
// - Custom model strings are also supported
// Example usage with different methods:
await expect(page).aiAssert("Shows dashboard", { model: "openai/o4-mini" });
const data = await page.extract("Get heading", { model: "google/gemini-3-flash-preview" });
const { locator } = await page.getLocatorsByAI("the login button", { model: "google/gemini-3-pro-preview" });
```
If no model is specified, the backend default is used.
## When to Use Stably SDK vs Playwright
**Prioritization:**
1. **Test accuracy and stability are the most important factors** - prioritize reliability over cost/speed.
2. **Otherwise, use Playwright whenever possible** since it's cheaper and faster.
3. **For interactions:** If the interaction will be hard to express as Playwright or will be too brittle that way (e.g., the scroll amount changes every time), then use `agent.act()`. **Any canvas-related operations, or any drag/click operations that require coordinates, must use `agent.act()`** (more semantic meaning, and less flaky).
4. **For assertions:** Use Playwright if it fulfills the purpose. But if the assertion is very visual-heavy, use Stably's `aiAssert`.
5. **For email verification flows:** Use `@stablyai/email` `Inbox` to receive and extract data from emails (OTP codes, magic links, confirmation emails).
6. **Use Stably SDK methods if it helps your tests pass** - when Playwright methods are insufficient or unreliable.
## AI Assertions (intent‑based visuals)
> **Note:** `toMatchScreenshotPrompt` is deprecated. Use `aiAssert` instead.
```ts
await expect(page).aiAssert(
"Shows revenue trend chart and spotlight card",
{ timeout: 30_000 }
);
await expect(page.locator(".header"))
.aiAssert("Nav with avatar and bell icon");
```
**Signature:** `expect(page|locator).aiAssert(prompt: string, options?: ScreenshotOptions & { model?: AIModel })`
* Use for **dynamic** UIs; keep prompts specific; scope with elements (using locators) when possible.
* **Consider whether you need `fullPage: true`**: Ask yourself if the assertion requires content beyond the visible viewport (e.g., long scrollable lists, full page layout checks). If only viewport content matters, omit `fullPage: true` — it's faster and cheaper. Use it only when you genuinely need to capture content outside the browser window's visible area.
## AI Extraction (visual → data)
```ts
// Extract from entire page
const txt = await page.extract("List revenue, active users, and churn rate");
// Extract from specific element (more focused, better results)
const headerText = await page.locator(".header").extract("Get the username displayed");
```
Typed with Zod:
```ts
import { z } from "zod";
const Metrics = z.object({ revenue: z.string(), activeUsers: z.number(), churnRate: z.number() });
const m = await page.extract("Return revenue (currency), active users, churn %", { schema: Metrics });
// Also works on locators
const UserSchema = z.object({ name: z.string(), role: z.string() });
const userData = await page.locator(".user-panel").extract("Get user info", { schema: UserSchema });
```
**Signatures:**
* `page.extract(prompt: string, options?: { model?: AIModel }): Promise<string>`
* `page.extract<T extends z.AnyZodObject>(prompt, { schema: T, model?: AIModel }): Promise<z.output<T>>`
* `locator.extract(prompt: string, options?: { model?: AIModel }): Promise<string>`
* `locator.extract<T extends z.AnyZodObject>(prompt, { schema: T, model?: AIModel }): Promise<z.output<T>>`
## AI Locator Finding (accessibility-based)
Use `getLocatorsByAI` to find elements using natural language based on the page's accessibility tree. Requires Playwright v1.54.1+.
```ts
// Find a single element
const { locator: loginBtn, count } = await page.getLocatorsByAI("the login button");
expect(count).toBe(1);
await loginBtn.click();
// Find multiple elements
const { locator: cards, count: cardCount } = await page.getLocatorsByAI("all product cards in the grid");
console.log(`Found ${cardCount} product cards`);
await expect(cards.first()).toBeVisible();
// With model selection
const { locator } = await page.getLocatorsByAI("the submit button", {
model: "google/gemini-3-flash-preview"
});
```
**Signature:** `page.getLocatorsByAI(prompt: string, options?: { model?: AIModel }): Promise<{ locator: Locator, count: number, reason: string }>`
**Returns:**
* `locator` - Playwright Locator for the found elements (matches nothing if count is 0)
* `count` - Number of elements found
* `reason` - AI's explanation of what it found and why
**Best Practices:**
* Describe elements by their accessible properties (labels, roles, text) rather than visual attributes
* Use for elements that are hard to locate with traditional selectors
* Check the `count` to verify expected number of matches before interacting
## AI Agent (autonomous workflows)
### When Test Generation Uses `agent.act()` vs Raw Playwright
Stably's test generation agent intelligently chooses between raw Playwright code and `agent.act()`:
- **For repeatable, stable actions**: The test generation agent will try to turn these into raw Playwright code when first generating tests. Raw Playwright is faster and more cost-effective.
- **For dynamic or unstable pages**: If the agent finds that the page changes frequently and it can't find stable Playwright selectors/code for an action, it will use `agent.act()` instead.
This means you may see a mix of both in generated tests—this is intentional and optimizes for reliability while keeping costs down where possible.
### Using the Agent Fixture
Use the `agent` fixture to execute complex, human-like workflows:
```ts
test("complex workflow", async ({ agent, page }) => {
await page.goto("/orders");
await agent.act("Find the first pending order and mark it as shipped", { page });
});
// Or create manually
const agent = context.newAgent();
await agent.act("Your task here", { page, maxCycles: 10 }); // split into smaller steps if possible
```
**Signature:** `agent.act(prompt: string, options: { page: Page, maxCycles?: number, model?: Model }): Promise<void>`
* Throws `Error` if the agent fails to complete the task (check `error.message` for the AI's failure reason)
* Default maxCycles: 30
* Supported models: `"anthropic/claude-sonnet-4-5-20250929"` (default), `"google/gemini-2.5-computer-use-preview-10-2025"`, or any custom model string
### Passing Variables to Prompts
You can use template literals to pass variables into your prompts:
```ts
const duration = 24 * 7 * 60;
await agent.act(`Enter the duration of ${duration} seconds`, { page });
const username = "john.doe@example.com";
await agent.act(`Login with username ${username}`, { page });
```
### Self-Contained Prompts
All prompts to Stably SDK AI methods (agent.act, aiAssert, extract) must be self-contained with all necessary information:
1. **No implicit references to outside context** - prompts cannot reference previous actions or state that the AI method doesn't have access to:
- ❌ Bad: `agent.act("Verify the field you just filled in the form is 4", { page })`
- ✅ Good: `agent.act("Verify the 'timeout' field in the form has value 4", { page })`
- ❌ Bad: `agent.act("Pick something that's not in the previous step", { page })`
- ✅ Good: `const selectedItem = "Option A"; await agent.act(\`Pick an option other than ${selectedItem}\`, { page })`
2. **Pass information between AI methods using explicit variables:**
```ts
// Extract data, then use it in next action
const orderId = await page.extract("Get the order ID from the first row");
await agent.act(`Cancel order with ID ${orderId}`, { page });
```
3. **Include detailed instructions and domain knowledge** to help the AI perform the task successfully:
- ❌ Bad: `agent.act("Fill in the form", { page })`
- ✅ Good: `agent.act("Fill in the form with test data. On page 4 you might run into a popup asking for premium features - just click 'Skip' or 'Cancel' to ignore it", { page })`
### Optimizing Agent Performance
**IMPORTANT:** The fewer actions/cycles agent.act() needs to do, the better it performs. Offload work to Playwright code when possible:
1. If your prompt has work that could be done by Playwright code, use Playwright for that work, and only use agent.act() for actions that are hard for Playwright (canvas operations, dynamic decision making, etc.)
2. If your prompt has repetition (e.g., do it 5 times), calculations (e.g., type 24*7*60 seconds), or other code-suitable tasks, use code for those, and only have agent.act() perform the agent-suitable part.
3. If your prompt has an if/else condition that can be expressed in code, use code for the condition, and only have agent.act() perform the agent-suitable part.
**Examples:**
- ❌ Bad: `"Click the button 5 times"`
- ✅ Good: `"Click the button"` (and include this in a loop that runs 5 times)
- ❌ Bad: `"enter the duration of 24*7*60 seconds"`
- ✅ Good: Calculate in code (`const sum = 24*7*60`), then use `\`enter the duration of ${sum} seconds\``
## Email Inbox (`@stablyai/email`)
The `@stablyai/email` package provides disposable email inboxes for testing email-dependent flows (OTP codes, verification links, order confirmations, etc.).
### Install
```bash
# npm
npm install -D @stablyai/email
# pnpm
pnpm add -D @stablyai/email
# yarn
yarn add -D @stablyai/email
```
Requires `STABLY_API_KEY` and `STABLY_PROJECT_ID` environment variables (same as `@stablyai/playwright-test`).
### Creating an Inbox
```ts
import { Inbox } from "@stablyai/email";
const inbox = await Inbox.build({ suffix: `test-${Date.now()}` });
// Generates address like: "org+test-1706621234567@mail.stably.ai"
console.log(inbox.address); // Full email address
console.log(inbox.suffix); // Applied suffix
console.log(inbox.createdAt); // Inbox creation timestamp
```
**`Inbox.build()` Parameters:**
| Parameter | Type | Description |
|-----------|------|-------------|
| `suffix` | string (optional) | Test isolation identifier appended to address |
| `apiKey` | string (optional) | Overrides `STABLY_API_KEY` env var |
| `projectId` | string (optional) | Overrides `STABLY_PROJECT_ID` env var |
### Waiting for Emails
```ts
const email = await inbox.waitForEmail({
from: "noreply@example.com",
subject: "verification",
timeoutMs: 60_000,
});
```
**`inbox.waitForEmail()` Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `from` | string | — | Sender address filter |
| `subject` | string | — | Subject line filter |
| `subjectMatch` | `'contains'` \| `'exact'` | `'contains'` | Subject matching mode |
| `timeoutMs` | number | `120000` | Max wait duration (ms) |
| `pollIntervalMs` | number | `3000` | Poll frequency (ms) |
Returns an `Email` object or throws `EmailTimeoutError`. Only checks emails received **after** the inbox was created.
### Email Object Properties
| Property | Type | Description |
|----------|------|-------------|
| `id` | string | Unique identifier |
| `mailbox` | string | Container (e.g., `"INBOX"`) |
| `from` | `{ address, name }` | Sender info |
| `to` | array | Recipients |
| `subject` | string | Subject line |
| `receivedAt` | Date | Delivery timestamp |
| `text` | string? | Plain text body |
| `html` | array? | HTML body parts |
### AI Extraction from Emails
Use `inbox.extractFromEmail()` to extract structured data from emails using AI:
**String extraction:**
```ts
const { data: otp, reason } = await inbox.extractFromEmail({
id: email.id,
prompt: "Extract the 6-digit OTP code",
});
```
**Structured extraction with Zod:**
```ts
import { z } from "zod";
const { data } = await inbox.extractFromEmail({
id: email.id,
prompt: "Extract verification URL and expiration",
schema: z.object({
url: z.string().url(),
expiresIn: z.string(),
}),
});
```
**`inbox.extractFromEmail()` Parameters:**
- `id` (required): Email identifier
- `prompt` (required): Extraction instruction
- `schema` (optional): Zod schema for structured output
Returns `{ data, reason }` or throws `EmailExtractionError`.
### Listing & Retrieving Emails
```ts
// List emails with optional filters
const { emails, nextCursor } = await inbox.listEmails({
from: "notifications@example.com",
limit: 10,
});
// Get a specific email by ID
const email = await inbox.getEmail(emailId);
```
**`inbox.listEmails()` Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `from` | string | — | Sender filter |
| `subject` | string | — | Subject filter |
| `subjectMatch` | `'contains'` \| `'exact'` | `'contains'` | Matching mode |
| `limit` | number | `20` (max 100) | Result count |
| `cursor` | string | — | Pagination cursor |
| `since` | Date | — | Override default time filter |
| `includeOlder` | boolean | `false` | Include pre-creation emails |
### Cleanup
```ts
await inbox.deleteEmail(email.id); // Delete single email
await inbox.deleteAllEmails(); // Delete all emails in this inbox
```
Only deletes inbox-scoped emails, not the entire org mailbox.
### Playwright Fixture Pattern
Create a reusable `inbox` fixture for test isolation and automatic cleanup:
```ts
import { test as base } from "@stablyai/playwright-test";
import { Inbox } from "@stablyai/email";
const test = base.extend<{ inbox: Inbox }>({
inbox: async ({}, use, testInfo) => {
const inbox = await Inbox.build({ suffix: `test-${testInfo.testId}` });
await use(inbox);
await inbox.deleteAllEmails();
},
});
test("OTP login flow", async ({ page, inbox }) => {
await page.goto("/login");
await page.getByLabel("Email").describe("Email input").fill(inbox.address);
await page.getByRole("button", { name: "Send OTP" }).describe("Send OTP button").click();
const email = await inbox.waitForEmail({
subject: "verification code",
timeoutMs: 60_000,
});
const { data: otp } = await inbox.extractFromEmail({
id: email.id,
prompt: "Extract the 6-digit OTP code",
});
await page.getByLabel("OTP").describe("OTP input").fill(otp);
await page.getByRole("button", { name: "Verify" }).describe("Verify button").click();
await expect(page).toHaveURL("/dashboard");
});
```
### Email Testing Best Practices
* **Always use unique suffixes** (e.g., `testInfo.testId` or `Date.now()`) for parallel test isolation.
* **Use the fixture pattern** for automatic cleanup after each test.
* **Keep `timeoutMs` reasonable** — default is 120s, but 60s is usually sufficient.
* **Prefer `waitForEmail`** over polling with `listEmails` — it handles retry logic automatically.
* **Use structured extraction with Zod schemas** when you need typed data (URLs, codes, dates).
* **Common extraction prompts:**
- `"Extract the OTP or verification code"`
- `"Extract the sign-in or verification URL"`
- `"Extract the order number and delivery date"`
## CI Reporter / Cloud
```bash
# npm
npm install -D @stablyai/playwright-test
# pnpm
pnpm add -D @stablyai/playwright-test
# yarn
yarn add -D @stablyai/playwright-test
```
```ts
// playwright.config.ts
import { defineConfig, stablyReporter } from "@stablyai/playwright-test";
export default defineConfig({
reporter: [
["list"],
stablyReporter({
apiKey: process.env.STABLY_API_KEY,
projectId: process.env.STABLY_PROJECT_ID,
// Optional: Scrub sensitive values from traces before upload
sensitiveValues: ["secret-password", process.env.API_SECRET].filter(Boolean),
}),
],
use: {
trace: "on", // Required for trace uploads
},
});
```
**Reporter Options:**
- `apiKey` (required): Your Stably API key
- `projectId` (required): Your Stably project ID
- `sensitiveValues` (optional): Array of strings to scrub from trace files
- `notificationConfigs` (optional): Per-project notification settings for Slack/email
## Notifications (Email/Slack)
Configure notifications per project via `stably` property in defineConfig:
```ts
import { defineConfig, stablyReporter } from "@stablyai/playwright-test";
export default defineConfig({
reporter: [stablyReporter({ apiKey: "...", projectId: "..." })],
projects: [
{
name: "smoke",
stably: {
notifications: {
slack: {
channelName: "#test-alerts",
notifyOnStart: true,
notifyOnResult: "failures-only", // "all" | "failures-only"
},
email: {
to: ["team@example.com"],
notifyOnResult: "all",
},
},
},
},
],
});
```
## Commands
```bash
# Recommended for Stably reporter + auto-heal
# npm
npx stably test
# pnpm
pnpm dlx stably test
# yarn berry
yarn dlx stably test
# Still supported (requires your reporter/config to be set up)
# npm
npm exec playwright test
# pnpm
pnpm exec playwright test
# yarn
yarn playwright test
# All Playwright CLI flags still work (headed, ui, project, file filters…)
# When running tests for debugging/getting stacktraces:
npm exec playwright test --reporter=list # or pm-equivalent; disable HTML reporter for direct terminal output
```
## ESM Utilities
For ESM projects needing `__dirname` or `__filename` equivalents:
```ts
import { getDirname, getFilename } from "@stablyai/playwright-test";
const __dirname = getDirname(import.meta.url);
const __filename = getFilename(import.meta.url);
// Use in tests for file paths
import path from "path";
await page.setInputFiles("input", path.join(__dirname, "fixtures", "file.pdf"));
```
## Best Practices
* **CRITICAL: All locators must use the `.describe()` method** for readability in trace views and test reports. Example: `page.getByRole('button', { name: 'Submit' }).describe('Submit button')` or `page.locator('table tbody tr').first().describe('First table row')`
* Scope visual checks with locators; keep prompts specific with labels/units.
* Use `toHaveScreenshot` for stable pixel‑perfect UIs; `aiAssert` for dynamic UIs.
* **Be deliberate with `fullPage: true`**: Default to viewport-only screenshots. Only use `fullPage: true` when your assertion genuinely requires content beyond the visible viewport (e.g., verifying footer content on a long page, checking full scrollable lists). Viewport captures are faster and more cost-effective.
## Troubleshooting
* **Slow assertions** → scope visuals; reduce viewport.
* **Agent stops early** → increase `maxCycles` or break task into smaller steps.
## Minimal Template
```ts
import { test, expect } from "@stablyai/playwright-test";
test("AI‑enhanced dashboard", async ({ page, agent }) => {
await page.goto("/dashboard");
// Use agent for complex workflows
await agent.act("Navigate to settings and enable notifications", { page });
// Use AI assertions for dynamic content
await expect(page).aiAssert(
"Dashboard shows revenue chart (>= 6 months) and account spotlight card"
);
});
```
Manual Installation Steps
Install Packages
Install both Playwright and the Stably SDK:
Copy
npm install -D @playwright/test@latest @stablyai/playwright-test@latest
Stably SDK requires Playwright version 1.52.0 or higher
Configure Environment Variables
- API Key: Get it here
- Project ID: Find it in the URL for your Stably Dashboard:
.env file (create it if it doesn’t exist):Copy
STABLY_API_KEY=your_api_key_here
STABLY_PROJECT_ID=your_project_id_here
.env files, you can install and configure dotenv to load them in your Playwright config:Copy
npm install -D dotenv
playwright.config.ts:Copy
import dotenv from 'dotenv';
import path from 'path';
dotenv.config({ path: path.resolve(__dirname, '.env') });
Add Stably Reporter to Playwright Config
Add the Stably reporter and enable tracing in your
playwright.config.ts:Copy
import { defineConfig } from "@playwright/test";
import { stablyReporter } from "@stablyai/playwright-test";
export default defineConfig({
reporter: [
["list"],
stablyReporter({
apiKey: process.env.STABLY_API_KEY,
projectId: process.env.STABLY_PROJECT_ID
}),
],
use: {
trace: 'on', // Required for Stably SDK
},
});
The Stably Reporter is required for
stably fix to work. See the Stably Test Reporter documentation for advanced configuration options including sensitive data scrubbing and notifications.Update Test Imports
Replace all imports from
@playwright/test with @stablyai/playwright-test in your test files:Copy
// Before
import { test, expect } from "@playwright/test";
// After
import { test, expect } from "@stablyai/playwright-test";
Bulk Update: Use this command to update all test files at once:(Remove the
Copy
find . -type f \( -name "*.spec.ts" -o -name "*.test.ts" -o -name "*.spec.js" -o -name "*.test.js" \) -not -path "*/node_modules/*" -exec sed -i '' "s/@playwright\/test/@stablyai\/playwright-test/g" {} \;
'' after -i on Linux)Setup AI Rules (Optional but Recommended)
Configure your AI assistant to understand Stably SDK patterns for better test generation.Recommended: Install the skills:Alternative: See manual setup instructions to copy/paste rules manually.
Copy
npx skills add https://github.com/stablyai/agent-skills --skill stably-sdk-setup --skill stably-sdk-rules
Verify Installation
Create a simple verification test to ensure everything works:Run the test:View results in your Stably Dashboard
Copy
// tests/stably-verification.spec.ts
import { test, expect } from '@stablyai/playwright-test';
test('stably sdk verification', async ({ page }) => {
await page.goto('https://www.stably.ai');
await expect(page).aiAssert(
"the page shows the Stably home page"
);
});
Copy
npx playwright test stably-verification.spec.ts
Next steps
CLI guide
Explore commands, agent workflows, and maintenance tools.
SDK overview
Get the big-picture tour of authentication, core APIs, and how the SDK fits with the CLI and Web experiences.
AI Assertions
Add resilient AI-powered checks that understand page context and reduce brittle selectors.
AI Agent Execute
Orchestrate multi-step user interactions with an autonomous agent that adapts to UI changes.
AI-generated tests
Use Cursor + Stably to create Playwright tests quickly, then refine with the SDK primitives.