A while ago, in “Don’t Keep Paddling for AI in the Age of Exploration”, we discussed a core paradigm shift: in the AI era, the biggest paradigm upgrade for us engineers is actually to stop hands-on paddling for AI like before, and instead establish a clear authorization and tooling system to treat AI as virtual employees capable of working independently. But to be honest, when people actually try to implement this authorization, almost everyone hits the exact same wall. Many teams find when assigning engineering tasks to AI that whenever AI tries to call their in-house SDKs, CLI tools, or APIs, the process becomes extremely fragile. AI often misunderstands hidden constraints of API parameters, hand-writes non-standard database migration scripts in declarative architectures, or gets stuck halfway without being able to continue simply because error messages aren’t clear enough.
When encountering such execution friction, people’s first instinct is usually to assume the model isn’t smart enough yet, thinking they might as well wait for the next generation of more powerful LLMs. But if we actually take the time to inspect the Agent’s execution trajectories and debug logs, we’ll discover an awkward reality that has long been overlooked: often it’s really not that the model lacks intelligence, but rather that our own product is simply too difficult for AI to use in terms of interface design, error feedback, and documentation structure. Between authorizing AI to do work and AI actually getting the job done properly lies a hard barrier—the product’s own AI friendliness. If the products we build in the future are intended not only for human use but also for AI use, we need to have a serious conversation about how to quantify, evaluate, and step-by-step improve a product’s AI friendliness.
Looking back over the past few decades, when we built developer software and SaaS products, the user persona in our minds was really of just one kind: human engineers sitting in front of screens. All Web consoles, interactive interfaces, CLI output formats, and official documentation were designed around human habits, cognitive cadence, and search behaviors. Human developers have strong inference abilities when reading documentation—they can fill in ambiguous gaps using experience and habitually search Google or community forums when seeing rudimentary error messages. But as tools like Cursor, Claude Code, Codex, and OpenCode become daily fixtures in developer environments, the actual users of products have quietly split into two camps.
In “Beyond DRY: AI-Native Software Engineering”, we explored this trend: AI is no longer just an auxiliary plugin helping humans type code, but is becoming a first-class citizen user of software systems.
On July 31, 2026, Supabase open-sourced its evaluation framework supabase/evals on its official blog and shared their latest scores on their public results page. They mentioned a key detail in the article: Agents have become one of the primary ways developers use Supabase. Every day, these Agents frequently invoke Supabase’s CLI, MCP Server, Agent Skills, and Markdown-formatted official documentation. This means the interface side of modern products is actually facing two types of users: human engineers who rely on experience and intuition, and AI Agents that rely primarily on context inputs and lack real-world experience. Humans can tolerate implicit assumptions in docs, but Agents must rely on exact specifications, structured errors, and precise indexing. If a product lacks machine expressiveness, Agents will frequently trial-and-error and fall into dead loops when creating schemas, configuring database RLS permission policies, or deploying Edge Functions.
Faced with AI frequently encountering friction within their product, an engineering team’s first instinct is often to look for an off-the-shelf benchmark standard to figure out who is falling short. But this is precisely the first pitfall most people stumble into: their initial reaction is usually to check public PR leaderboards like SWE-bench, MMLU, or LMSYS, hoping to save the day by switching to a higher-ranked general model. But frankly, general model leaderboards measure the upper bound of a model’s general coding capability; they cannot answer whether a model can get work done correctly within a specific product architecture. Engineering details vary too widely across products—whether it is Supabase’s permission policy configuration, Stripe’s payment workflow integration, or Convex’s reactive data modeling, all fall far beyond what general leaderboards can measure.
Because of this, top AI-Native companies have long stopped relying on general leaderboards and have started building proprietary evaluation systems for their own products. For example, on March 2, 2026, Stripe published its payment integration benchmark and made public a real test environment covering API migrations, SDK upgrades, Checkout integration, Subscriptions management, and full-stack browser flows in the open-source repository stripe/ai/benchmarks; Convex organized 8 typical backend tasks in its open-source project get-convex/convex-evals and explained on its public leaderboard that they automatically run evaluations every 4 hours via configuration files, pin the test set using version hashes, and continuously tune official developer guides using test results. There is a critical pivot here: doing AI evaluation for your own product is not about releasing a horizontal PR leaderboard for competitive comparisons. Truly valuable Benchmarking looks inward—establishing a longitudinal regression mechanism tailored to your own product. We don’t compete with external rivals; we only watch whether AI task success rates in our own scenarios steadily rise as our product documentation, CLI guidance, or Skill specifications improve.
Once shifting from viewing general leaderboards to building
longitudinal regressions for one’s own product, a major mindset shift
occurs within the team: when automated evaluations fail in the sandbox,
people no longer habitually blame the AI for not being smart enough;
instead, they realize the true target under test is the product itself.
While running supabase/evals, Supabase
recorded several valuable fix cases: the team originally provided a
Postgres best practices Agent Skill, but the initial sandbox test
success rate was only about 10%; the team subsequently rewrote the
description text, boosting activation rate to around 60%; in database
migration tests where the Agent wrote messy migration scripts, the team
revised guidance in the Skill to converge Agent behavior into standard
flows; when the Agent confused concepts using the
@supabase/server package, the team added a “Which package
to choose” documentation guide; the team also tracked page counts and
navigation paths of different Agent toolchains (MCP Server vs. Web
retrieval) when reading docs, accurately distinguishing between
knowledge gaps and retrieval behavior deviations.
Similarly, when Stripe ran its own integration benchmarks, it explicitly stated that the evaluation system directly exposed and helped the team fix multiple previously unnoticed bugs in their official documentation. These examples vividly illustrate a key principle: test failures do not mean the model is defective; they indicate flaws in product interfaces and documentation design. An evaluation system is like installing an automated diagnostic tool for your product, where every failure points directly to what needs to be fixed on the product side.
However, when building this diagnostic tool in practice, many fall
into a second trap: confusing surface-level format checks with real task
execution and getting misled by surface scores from static grading
tools. For instance, tools like Fern Agent
Score or Mintlify
Agent Score, which emerged in April 2026, act like a Lighthouse scan
for documentation sites, mainly checking static prerequisites such as
whether llms.txt exists, path formats are normalized, or
OpenAPI descriptions are sufficient. In Fern’s scan, Supabase’s
documentation site was given a low score of 72/100; Mintlify similarly
gave an 81/100 score. Looking at these static scores alone, it would
seem Supabase is unfriendly to AI. Yet in Supabase’s own end-to-end
dynamic testing, the Agent’s task success rate for creating tables,
configuring permissions, and deploying edge functions inside real Docker
sandboxes was as high as 95% to 100%.
This seemingly contradictory contrast perfectly highlights the difference between static prerequisites and dynamic execution. Static prerequisites answer whether the machine entry point of a product is easy to find and whether the storefront is open—making them suitable for lint checks in CI workflows, but they cannot guarantee that an Agent can write complex engineering code correctly after receiving the documentation. Dynamic execution answers whether an Agent can truly perform complex tasks correctly inside a sandbox, requiring tasks to be dispatched in real containers or virtual machines and using deterministic test code to verify final database state and system behavior. Mintlify’s own doc URL discovery experiment published on July 17, 2026 (open-sourced at docs-url-discovery-bench), also confirmed this: across 2,400 real Agent retrievals on 20 documentation sites, the key to eliminating retrieval 404 errors was providing clear Indexes and URL Maps, rather than simply converting HTML to Markdown. Therefore, we must never replace end-to-end dynamic testing with static scanning scores. Static checks ensure entry criteria meet standards, while dynamic evaluations verify whether tasks are ultimately executed correctly.
Having confirmed the necessity of dynamic execution for validation, many teams enter a third pitfall when improving their products: blindly chasing the newest and most complex AI delivery concepts. In practice, it’s easy to develop a subtle technical snobbery, feeling that writing instructions in prompts is an outdated 2023 practice, and that entering the Agent era and AI-Native workflows means one must use the latest Agent Skill, MCP Server, or various Plugins. But the hard truth in engineering is that it doesn’t matter if a cat is black or white as long as it catches mice; what truly determines the quality of a technical solution is always the ultimate task success rate, not whether a technical concept sounds trendy enough.
To verify the practical effectiveness of different delivery routes, we can break down common approaches: one is providing no extra context documentation, relying solely on the model’s built-in prior knowledge; another is providing a default Agent Skill file, letting the Agent judge whether to use it; a third is adding strict constraints in prompt instructions to force the Agent to invoke a specific Skill; and yet another is compressing key indexes into a minimal text document (such as AGENTS.md) directly embedded in the context.
On January 27, 2026, Vercel published evaluation data for Next.js 16 APIs on its official blog (with evaluation code open-sourced at vercel/next.js/evals), providing a clear comparison across these approaches:
This test result revealed a surprising detail: in 56% of cases, the Agent did not voluntarily invoke the pre-configured default Agent Skill. In other words, without explicit instructions to forcefully intervene, configuring a default Skill for an Agent yields practical results virtually identical to the baseline state of providing no documentation at all.
This experiment is not meant to prove that the Skill mechanism itself is ineffective, but rather to reveal an easily overlooked truth: without quantitative measurement through Evaluation, a team might invest substantial effort maintaining a complex set of Skill specifications without ever realizing that the Agent fails to trigger them in most scenarios. Based on Evaluation data, Vercel ultimately made a pragmatic decision: abandon the cumbersome default Skill approach and instead provide a compressed AGENTS.md index in the project root directory. This validates our earlier point—don’t chase trendy technical concepts based on intuition alone; follow real data produced by Evaluation to find the AI context delivery approach that best suits your product today.
Having recognized the limits of general leaderboards, clarified the necessity of dynamic testing, and avoided the trap of blindly chasing trends, the remaining core question is: how do we actually get this AI friendliness evaluation running within our own team? Implementing this evaluation method within your team doesn’t need to be complex right out of the gate; following a three-step process makes it smooth. Step one: build a test suite derived from real pain points. Instead of inventing scenarios out of thin air, go directly through customer support tickets, high-frequency error reports in GitHub Issues, and places where community developers report getting stuck most often, rewriting these real friction points into 20 to 50 reproducible automated evaluation scenarios.
Step two: set up a two-layer test architecture combining static Lint
and dynamic Eval. The static check layer (cheap, CI-level) sits in CI
workflows like GitHub Actions, performing sub-second checks on Markdown
formatting, broken links, OpenAPI structures, and llms.txt
specifications on every commit; the dynamic evaluation layer (expensive,
daily/weekly) draws on Convex’s scheduling design to spin up
infrastructure inside isolated Docker containers to run end-to-end
tasks, using deterministic test scripts to verify database state and
generated code while locking down test suites with version hashes. Step
three: complete the attribution loop from test failure to product fix.
Whenever evaluation pass rates drop or hit roadblocks, don’t rush to
tweak LLM parameters; instead, dive into the Agent’s tool call
trajectories and document retrieval paths. Make repairs directly on the
product surface and verify behavioral convergence in the next Regression
run.
In the Age of Exploration, the prerequisite for empowering AI to work independently is equipping your product with a self-reflecting diagnostic tool for AI friendliness.
Evaluations are not for crafting PR leaderboards to flex against competitors, but for building a clear feedback loop between your product and AI. Stop agonizing over general model rankings and stop blindly chasing the latest technical buzzwords. Follow the empirical data from Evaluation to let your product truly sail smoothly in the upcoming AI era.