Apibenchmarks Update

The Latency Arms Race Nobody Asked For

Two years ago, choosing an LLM API was simple. You picked OpenAI, you got GPT-4, you paid the bill, and you wrote your prompt. Nobody really cared about time-to-first-token because the use case was "summarize this PDF" and a 2-second delay felt perfectly fine. Then someone figured out you could put these things in customer-facing chat, and suddenly latency stopped being a nice-to-have and started being the whole game.

We're the team behind Apibenchmarks, and we've spent the last 14 months hammering on roughly 180 different model endpoints — every GPT variant, every Claude, every Gemini, plus a genuinely absurd number of open-weight models hosted on third parties. We're not paid by any of these providers. We just like knowing which boxes are slow, which boxes lie about their throughput, and where the price/performance curve actually bends. This post is a summary of what we found when we measured things honestly.

The short version: if you're optimizing for raw tokens-per-second on long generations, the picture in late 2025 looks nothing like it did even nine months ago. DeepSeek showed up, Qwen shipped quietly excellent small models, and the "premium provider" advantage evaporated for anything that isn't a giant reasoning model. Meanwhile, on time-to-first-token — the metric that actually matters for chat UIs — the spread between best and worst is now over 11x. Eleven times. From a user perspective, that's the difference between feeling snappy and feeling broken.

What We Actually Measured (And Why Your Run-of-the-Mill Latency Test Is Lying to You)

Most "API benchmarks" you see on Twitter are garbage, and we say that with love. The typical pipeline: hit the endpoint, time the full response, divide by token count, post a screenshot. That gives you a single number that tells you approximately nothing useful. Here's why.

A modern inference endpoint has at least three distinct latency phases. There's cold start, which is the time from "first request in a while" to "first token produced" — dominated by model loading, weight sharding across GPUs, KV-cache warmup, and any provider-specific bootstrap. Then there's warm TTFT (time to first token on a warm endpoint), which measures pure prefill cost on your prompt. Finally, there's inter-token latency, the average ms between subsequent tokens in the decode phase, which is bound by memory bandwidth and KV-cache size.

We measure all three, plus throughput, plus tail latencies at p50/p90/p99. We use the same 1,247-token prompt across all models (a chunky section of a fictional technical document with enough variation to avoid lucky-cache wins). We run 50 warm-up calls before timing anything, then collect 200 samples, with 5-second pauses between calls to avoid stepping on other tenants. Cold-start numbers come from a fresh endpoint with no requests for 10 minutes prior. Everything goes through the OpenAI-compatible chat completions interface because that's the only realistic way to compare apples to apples.

We also report standard deviation, because models with great averages and terrible consistency are unusable in production. A p99 that's 4x your p50 is a recipe for angry customers.

The Numbers: Cold Start vs Warm Latency Across 184+ Models

The table below shows a curated slice of our dataset — the models people actually deploy, run from a us-east-1 client hitting each provider's primary region. All numbers are in milliseconds. Lower is better. "Cold" is measured after 10 minutes of endpoint idle. "Warm TTFT" is the post-warmup time-to-first-token on the same 1,247-token prompt.

Some patterns jump out. The frontier proprietary models from OpenAI, Anthropic, and Google all cluster within ~80ms of each other on warm TTFT — the differences between them are statistical noise at typical prompt sizes. The real variance is at the cold-start end: spinning up a 405B from scratch across an 8-GPU node takes the hosting provider almost 10 seconds, while a small model on a single H100 is up in under 1.5.

Throughput (tokens per second during decode) tells an interesting story. The fastest model in our test is Gemini 2.5 Flash at 232 tok/s — partly because Flash is genuinely small, partly because Google has crazy good tensor serving infrastructure. The slowest is Llama 3.1 405B at 84 tok/s, because 405 billion parameters simply don't fit in memory bandwidth on a single host. If you're streaming long completions to a user and care about perceived speed, those numbers compound fast: a 2,000-token response takes 8.6 seconds on the 405B and 8.6 seconds before Flash even finishes.

Streaming Tokens Per Second: Where Throughput Gets Weird

Throughput isn't a constant, and pretending it is will wreck your capacity planning. We measured tokens-per-second at three different generation lengths — short (200 tokens), medium (1,000), and long (3,000) — and the ranking changes depending on where you look.

The "Drop %" column shows how much throughput degrades from short to long generations. Bigger, smarter models degrade faster — the 405B loses 45% of its tok/s when the KV cache grows large enough to spill attention computation across slower memory. Small models are remarkably stable. If you're generating user-facing responses that exceed 1,000 tokens, your effective throughput could be 30% lower than what the marketing page claims.

One more subtlety: inter-token latency is not normally distributed. The p99 inter-token latency is typically 2–3x the median. A model with a 25ms median inter-token latency will sometimes pause for 80ms between tokens, and your users will absolutely notice those pauses even if the average looks great.

Pricing vs Speed: The Uncomfortable Truth

Here's the part that made us squint at our spreadsheet for a while. Cost-per-million-tokens varies by an order of magnitude across these models, and the relationship to speed is not what you'd expect.

The rightmost column — dollars spent per second of generated output — is the ratio we actually care about for cost-sensitive