Caliper brings pass@k reliability testing to Claude Code and Codex

Caliper is an open-source tool that runs pass@k evaluations against AI coding agents like Claude Code and OpenAI Codex, measuring how reliably they solve a given task across multiple attempts. For indie makers building on top of AI coding tools, this fills a real gap: instead of eyeballing whether an agent 'works,' you get a quantitative reliability signal. That kind of systematic benchmarking has mostly lived inside big labs — now it's a GitHub repo anyone can run.