Can AI coding agents actually use your SDK?
AI agents now write most first integrations. SDKScore measures how often they get your SDK right: the generated code either compiles against your published types or it doesn't. No LLM judges, no vibes.
Pin the SDK
Every SDK is installed at an exact version into a throwaway project. Types come from the real package, not from docs.
Publish the tasks
Four realistic integration tasks per SDK. The prompts are public and identical for every model.
Run the agent headlessly
Each agent writes the file with no human in the loop, no retries, and no access to the grader.
Compile, don't judge
tsc --noEmit against the pinned types, plus published pattern checks for verified-deprecated APIs.
Leaderboard
Click any row for the per-task breakdown and the exact compiler errors.
| # | SDK | gpt-5.6-sol | haiku-4.5 | sonnet-5 | Overall |
|---|---|---|---|---|---|
| 01 | Octokit (GitHub REST) | 100 76–100 | 100 76–100 | 100 76–100 | 100 |
| 02 | Supabase | 100 76–100 | 100 76–100 | 100 76–100 | 100 |
| 03 | Hono | 100 76–100 | 92 65–99 | 92 65–99 | 95 |
| 04 | Stripe | 100 76–100 | 83 55–95 | 83 55–95 | 89 |
| 05 | Zod | 100 76–100 | 50 25–75 | 92 65–99 | 81 |
| 06 | Resend | 100 76–100 | 25 9–53 | 100 76–100 | 75 |
| 07 | Drizzle ORM | 100 76–100 | 25 9–53 | 92 65–99 | 72 |
| 08 | Anthropic | 25 9–53 | 75 47–91 | 100 76–100 | 67 |
| 09 | OpenAI | 75 47–91 | 25 9–53 | 92 65–99 | 64 |
| 10 | Vercel AI SDK | 50 25–75 | 0 0–24 | 75 47–91 | 42 |
Add your badge
If your SDK is on the board, it has a badge. It updates when the benchmark reruns. Nothing to host, nothing to install.
badges/stripe.svg
[](https://sdkscore.dev/#stripe)
Get your SDK scored
Adding an SDK is free and the results are published whether they flatter you or not. Send the package name and four integration tasks you actually care about.
- Public npm package with shipped TypeScript types
- Four tasks a real developer would write in week one
- You may contest any failing task in a public issue
Questions
Numbers in the answers come from the current run.
Why do AI coding agents write outdated SDK code?
A model only knows an SDK as it existed at its training cutoff, and SDKs ship on their own cadence. Renames, removed methods and reshaped option objects that land after the cutoff are invisible to it, so it keeps writing the last version it saw. That code often reads fine and still fails to compile against the published types.
Which SDKs do agents fail most?
Bottom of the July 2026 run: Vercel AI SDK at 42/100 and OpenAI at 64/100, averaged across gpt-5.6-sol, haiku-4.5 and sonnet-5. The list is regenerated from results.json on every rerun. The exact compiler diagnostics behind each failing task are published there too.
How are scores graded?
Generated code is compiled with tsc --noEmit against the SDK's pinned, real published types, then checked against published patterns for APIs the vendor documents as deprecated. A task passes only if it compiles clean and uses none of those APIs. No LLM judges and no rubric: the compiler is the only referee.
How do I get my SDK scored?
Open a public issue or email hello@sdkscore.dev with the npm package name, a version to pin, and four integration tasks a real developer writes in week one. Scoring is free and the results are published whether they flatter the SDK or not. The requirements are listed in Get your SDK scored.