Supabase Evals: Benchmark Architecture & Agent Behavior Analysis (2026)

Date: August 1, 2026

Repository: supabase/evals (Apache-2.0)

Target Agents: Claude Code, Codex, OpenCode

1. Executive Summary

Supabase open-sourced Supabase Evals, an evaluation framework and benchmark suite designed to test how effectively AI coding agents build, debug, deploy, and maintain cloud database backends. Rather than relying on simple unit tests or synthetic code snippets, Supabase Evals executes agents against live containerized environments (PostgreSQL, local Supabase CLI, @supabase/lite management API, and real MCP servers).

2. Evaluation Matrix & Harness Architecture

Matrix Dimensions

The benchmark spans three core operational vectors:

1. Products: Database, Auth, Storage, Edge Functions, Realtime, Cron, Queues, Vectors, Data API.

2. Topics: RLS, Security, Migrations, SQL, SDK, Observability, Self-hosting, Testing, Declarative Schemas.

3. Lifecycle Stages: Build, Deploy, Investigate, Resolve.

Harness & Scorer Structure

Each evaluation scenario is self-contained in a dedicated directory:

• PROMPT.md: Frontmatter metadata (stage, suite, interface, product, topic) + task instructions.
• EVAL.ts: Programmatic scoring script exporting a LocalStackScorer.
• Optional local/ or remote/ directories supplying initial git workspaces or database states.

Scoring Mechanism:

• Combines deterministic checks (e.g. verifying supabase db diff tool calls, database information schema queries, container logs) with LLM-as-a-judge assertions.
• Agents receive one retry attempt upon initial failure before final grading.

3. Key Agent Pitfalls & Behavioral Findings

1. Imperative SQL vs. Declarative Migrations

Issue:* Agents routinely attempt to hand-write SQL migration scripts rather than making declarative schema edits and generating diffs via supabase db diff.

Harness Assertion:* EVAL.ts specifically inspects tool call logs for supabase db diff execution.

Remediation:* Explicit skill instructions forcing agents to alter declarative .sql schemas first, then run supabase db diff.

2. Raw Auth Implementation vs. SDK Helpers

Issue:* Agents write custom JWT or session verification logic manually rather than using standard framework helpers (e.g. @supabase/server).

Remediation:* Provide SDK selection guidelines in agent skill prompts.

3. Disparate Documentation Retrieval Habits

Codex / GPT-5.6:* Pulls ~8 documentation pages per task scenario on average.

Claude Code:* Fetches documentation in <40% of scenarios (~2 pages per scenario), relying heavily on context window knowledge or pre-loaded skills.

4. Skill Effectiveness Across Model Tiers

Top-Tier Models (Opus 5, Kimi K3):* Achieved 100% pass rates on Build stage scenarios without extra skills.

Mid-Tier Models (Sonnet 5, GPT-5.6 Sol, GPT-5.4 mini):* Experienced substantial performance boosts when supplied with targeted domain skills:

- Sonnet 5: 78% → 100%

- GPT-5.6 Sol: 89% → 100%

- GPT-5.4 mini: 78% → 89%

4. Operational Takeaways for Sovereign Agent Systems

• Deterministic Evaluation Gating: When building internal skills or prompt harnesses, enforce programmatic checks on intermediate tool execution (e.g., verifying specific CLI subcommands like diff or lint) rather than solely judging final file output.
• Skill Authoring Standard: Ensure skills for cloud/database tooling explicitly highlight native framework abstractions (@supabase/server, declarative migrations) to steer agents away from anti-patterns.