We Found a Production Database Bug in 5 Minutes — Here's How

We built Agent Scrimmage to stress-test AI agents before customers do. To prove it worked, we tested our own AI assistant first — the JobSite Viewer assistant, a Claude Code agent built for a construction SaaS platform.

JobSite Viewer is a project management tool for contractors. The AI assistant handles invoicing, project lookups, client management, scheduling, and more. It connects to a PostgreSQL database via Supabase, accesses file systems, and manages persistent memory across conversations.

We connected the agent via its API endpoint and hit "Run Discovery."

Discovery completed in under 30 seconds. It extracted:

• 42 confirmed capabilities (invoice creation, project lookups, client management, scheduling, time tracking)
• 28 limitations (no direct database writes, no email sending, no payment processing)
• 16 tools (Supabase queries, file system access, memory management, MCP server connections)
• 8 guardrails (data validation, permission checks, destructive action blocks)
• Industry classification: Field Service / Construction

The readiness score came back at 63/100 — "Needs Work." The discovery found gaps in error handling, missing guardrails for edge cases, and several capabilities the agent claimed but couldn't verify.

But the real discovery came next.

We ran the first batch of targeted scenarios — conversations designed to test the specific capabilities discovery had found. One scenario asked the agent to look up a project and cross-reference it with recent invoices.

The agent started its response normally, queried the database for the project record, then attempted to join invoice data from a replicated read replica. That's when it crashed.

The error: a database replication lag issue where the read replica was returning stale data that didn't match the primary. The agent's query assumed consistency between primary and replica — an assumption that held during development and manual testing, but failed under the specific query pattern our scenario triggered.

The crash would have affected every API customer using the agent. It had been in production for weeks. No customer had triggered the exact query pattern yet — but it was only a matter of time.

The development team had tested the agent manually — chatting with it, asking it to perform tasks, checking responses. The agent worked perfectly in these tests because:

1. 1.Manual testers asked simple, predictable questions
2. 2.They never combined operations in the specific sequence that triggered the replication lag
3. 3.They tested during low-traffic periods when replication was caught up
4. 4.They didn't test with realistic data volumes or concurrent operations

Our scenario was different. It combined a project lookup with an invoice cross-reference with a date range filter — three operations that, when executed in sequence, exposed the replication timing issue. This pattern would naturally occur when a real customer asks "Show me all invoices for the downtown renovation project from last quarter."

The fix was straightforward: add a consistency check before cross-referencing data across tables, and fall back to the primary database when replica staleness is detected. The development team implemented the fix in under an hour.

We re-ran the same scenarios. The agent handled the cross-reference query correctly, returning accurate data with a note about the data freshness. The readiness score improved from 63 to 71.

Subsequent simulation batches found additional issues — a missing guardrail for bulk delete operations, inconsistent date formatting in invoice exports, and a prompt injection vulnerability in the project search function. Each was caught before any customer encountered it.

This experience crystallized why we built Agent Scrimmage:

1. 1.Discovery-first testing is essential. Without mapping capabilities first, we would have tested the wrong things. Generic prompts like "Can you help me with an invoice?" would have passed. The bug required a specific, multi-step operation that only targeted scenario testing would trigger.
2. 2.AI agents need more than conversation testing. The bug wasn't in the agent's conversation quality — it was in the infrastructure the agent depended on. Testing only the chat interface misses the entire backend failure surface.
3. 3.Production bugs hide in realistic scenarios. The database replication issue only appeared when operations were combined in sequences that real users would naturally perform. Synthetic, isolated test prompts can't reproduce this.
4. 4.Speed matters. Finding this bug took 5 minutes with Agent Scrimmage. Finding it from a customer report would have taken days of debugging, plus the customer trust damage.

If you're building an AI agent that connects to databases, APIs, or external services, you need to test it the way your customers will use it — not the way your developers demo it.