Apibenchmarks Update

Why API Latency Is the Metric That Actually Kills Your Product

Most developers obsess over model quality. They argue about benchmarks like MMLU, HumanEval, and GPQA, debating whether Claude or GPT-4o writes better Python or whether Gemini has superior reasoning. And sure, those things matter. But here's the dirty secret of shipping AI products: latency is the metric your users will actually feel. A 200ms response feels snappy. A 1,200ms response feels broken. A 3,000ms response feels like the API is dead. Nobody is sitting there running MMLU on your chatbot — they're tapping the screen and wondering if it crashed.

After running hundreds of API benchmarks across every major provider over the past six months, I've come to a fairly uncomfortable conclusion: the gap between providers on latency is far wider than the marketing pages suggest, and the gap between advertised and actual performance is even wider. The good news is that this gap is measurable, predictable, and — once you understand it — very much optimizable.

On Apibenchmarks, we track time-to-first-token (TTFT), tokens per second, and end-to-end latency across more than 180 large language models. We test from multiple regions, on multiple times of day, with different prompt sizes. The data tells a story that's more nuanced than "Provider X is the fastest."

The Anatomy of an API Call: Where Latency Actually Hides

Before we dive into the numbers, it's worth understanding what you're actually measuring. When you send a request to an LLM API, you're not measuring a single thing. You're measuring a chain of events that includes network round-trip time, queue time inside the provider's load balancer, model warmup (sometimes called "cold start"), prefill time, and then the streaming decode phase where tokens actually start arriving at your client.

Time-to-first-token captures the first part of that chain. If you're building a chat interface, TTFT is what your user sees as the "thinking" pause. If TTFT is 400ms, your UI feels responsive. If it's 1,800ms, the user has already started wondering if they should refresh the page.

Tokens per second captures the streaming phase. This is what makes long completions feel fast. A model that does 80 tokens/second feels much more "alive" than one that does 25 tokens/second, even if both have the same TTFT. The end-to-end metric is the total time from request to final token, which matters most for batch jobs, agents, and any synchronous workflow.

Cold start is its own beast. Many providers, especially those serving open-source models on spot GPU capacity, exhibit a punishing first-request-after-idle penalty. We've measured cold starts ranging from 800ms on the low end to over 8 seconds on the high end for certain smaller models. If your application is bursty, this is going to bite you.

Real Benchmark Data: Median Latency Across Major Providers

The following table reflects median values collected over a 30-day rolling window in February 2026, measured from a US-East client with prompts averaging 450 input tokens and requesting 250 output tokens. All values are in milliseconds. Lower is better.

Look closely at the Llama 3.3 70B row. That 110ms TTFT isn't a typo. Groq's LPU inference hardware genuinely sits in a different performance class for open-source models. The same model running on a more traditional GPU cloud would typically come in around 380-450ms TTFT. Hardware matters as much as the model itself, and that's a point most comparison pages miss entirely.

Another thing worth noting: the gap between p50 and p95 tells you about consistency. A provider with a tight p50-to-p95 spread is more predictable in production. OpenAI and Anthropic both have relatively tight distributions. Some of the open-source-serving providers have spreads of 4x or more, which means your "average" latency might be 300ms but your worst-case latency could be 1.5 seconds, and that's where user complaints come from.

How to Measure Latency Yourself (The Right Way)

If you're going to make routing decisions based on latency data, you need to be able to reproduce these measurements in your own environment. Vendor benchmarks are useful, but they reflect the vendor's test conditions, not yours. Here's a minimal but realistic benchmarking script in Python that hits the unified API endpoint, which gives you one place to test dozens of models without rewriting your client code each time.

import time
import httpx
import statistics

API_KEY = "your-key-here"
ENDPOINT = "https://global-apis.com/v1/chat/completions"

MODELS = [
    "gpt-4o",
    "gpt-4o-mini",
    "claude-sonnet-4.5",
    "claude-haiku-4.5",
    "gemini-1.5-pro",
    "gemini-1.5-flash",
    "llama-3.3-70b",
    "deepseek-v3",
    "qwen-2.5-72b",
]

PROMPT = "Explain the difference between p50 and p99 latency in three sentences."

def time_one_call(client: httpx.Client, model: str) -> dict:
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": PROMPT}],
        "max_tokens": 250,
        "stream": False,
    }
    headers = {"Authorization": f"Bearer {API_KEY}"}

    start = time.perf_counter()
    resp = client.post(ENDPOINT, json=payload, headers=headers, timeout=30.0)
    data = resp.json()
    end = time.perf_counter()

    return {
        "model": model,
        "total_ms": (end - start) * 1000,
        "usage": data.get("usage", {}),
        "status": resp.status_code,
    }

def benchmark_model(client: httpx.Client, model: str, n: int = 25) -> dict:
    samples = []
    for _ in range(n):
        result = time_one_call(client, model)
        if result["status"] == 200:
            samples.append(result["total_ms"])
    if not samples:
        return {"model": model, "error": "no successful samples"}

    samples.sort()
    return {
        "model": model,
        "runs": len(samples),
        "p50_ms": round(statistics.median(samples), 1),
        "p95_ms": round(samples[int(len(samples) * 0.95)], 1),
        "min_ms": round(min(samples), 1),
        "max_ms": round(max(samples), 1),
        "stdev_ms": round(statistics.stdev(samples), 1) if len(samples) > 1 else 0,
    }

with httpx.Client() as client:
    for model in MODELS:
        result = benchmark_model(client, model)
        print(result)

A few important details about why this script is structured the way it is. First, we're not measuring TTFT directly because we're using non-streaming requests. For TTFT, you'd switch stream: True and parse the SSE events, recording the time of the first data: line. For end-to-end latency in a streaming context, you'd record the time of the final token's event.

Second, we're running 25 samples per model and using the empirical p95 rather than assuming a normal distribution. Real-world latency data is almost never normally distributed — it has a long right tail, and assuming it is will mislead your capacity planning.

Third, run this from your actual production regions. A benchmark run from a US-East server to a US-East API endpoint tells you almost nothing about what your Singapore user will experience. Network distance matters, and it's one of the most underrated factors in latency.

What the Numbers Actually Tell You

Once you have your own data, the question becomes: what do you optimize for? The honest answer depends on what you're building.

For interactive chat, TTFT is king. Your user is staring at a blinking cursor. If you can keep TTFT under 300ms, the experience feels instantaneous even if the full completion takes three seconds. Claude Haiku, Gemini Flash, and the small Llama models on Groq all crush this metric. Even GPT-4o-mini, despite being slower than the dedicated small models, is fast enough to feel responsive for most use cases.

For long-form generation — think blog writing, code generation, document summarization — tokens per second is the metric to watch. A model that streams at 150 tokens/second will produce a 1,000-token response in about 6.7 seconds of streaming, after the initial TTFT. A model that streams at 50 tokens/second will take 20 seconds for the same output. The 13-second difference is the difference between "the AI is working" and "this thing is broken."

For batch processing and agent loops, end-to-end latency is what matters. If you're running an agent that calls an LLM five times in sequence, even small per-call latencies compound. A 200ms-per-call difference across five calls is a full second. For tight agent loops, Groq-hosted models and Gemini Flash are the current winners, often completing 5-step agent workflows in under 6 seconds end-to-end.

For cost-sensitive applications, the price-per-token column matters as much as the latency. Gemini 1.5 Flash at $0.075/$0.30 per million tokens is roughly 33x cheaper than GPT-4o on input and 33x cheaper on output. If your use case can tolerate slightly slower responses, the cost savings are staggering. We see teams running 100M+ tokens per month save five figures per month just by switching from a flagship model to a small one.

Common Pitfalls When Benchmarking Latency

I'd be doing you a disservice if I didn't warn about the traps. The most common mistake is testing with prompts that are too short. A 10-token prompt hides network overhead, because the actual inference time is so small that you're basically just measuring round-trip time. Use realistic prompts, ideally sampled from your actual production traffic.

The second pitfall is testing at the wrong time of day. Most providers have quiet hours and busy hours. We've seen p95 latencies nearly double during peak business hours in the US, particularly for providers with smaller capacity reserves. Run your benchmarks at multiple times: morning, afternoon, evening, and overnight, and look at the p95 from the worst window, not the best.

The third pitfall is forgetting about the first request. Many providers, especially those running open-source models, exhibit a cold start penalty on the first request after a period of inactivity. If your benchmark script sits idle between models, your first sample for each model is contaminated. Warm up the API with a throwaway request before you start collecting data.

The fourth pitfall is conflating streaming and non-streaming latency. They measure different things. Streaming latency is usually reported as TTFT plus tokens/sec. Non-streaming latency is the full round trip. Don't compare them directly. Pick the mode that matches your production behavior and stick with it.

Finally, watch out for the hidden cost of rate limits. If you're being throttled, your latency will appear artificially high because requests are sitting in queues. Some providers give you clearer error codes for throttling than others, and you might not realize your benchmark is actually measuring queue time, not inference time.

Key Insights From Six Months of Benchmarking

Here's what I'd want you to take away from all of this. First, latency variance is at least as important as median latency. A provider with a 250ms median and a 400ms p95 is more predictable — and usually feels faster in practice — than a provider with a 200ms median and a 1,200ms p95. Consistency wins.

Second, the smallest models are often the best choice for latency-sensitive applications. Gemini 1.5 Flash, Claude Haiku, and GPT-4o-mini all sit in roughly the same latency tier despite being a fraction of the cost of their flagship siblings. The quality gap on simple tasks is often negligible, and on hard reasoning tasks you can fall back to the flagship model selectively.

Third, hardware specialization matters more than model size for raw speed. Groq's LPU, Google's TPUs, and some of the newer inference providers are running the same open-source models at 2-3x the speed of generic GPU clouds. If you don't need a specific proprietary model, you can get flagship-tier speed at small-model prices just by choosing the right host.

Fourth, regional routing is the easiest latency win available. If you have users in Asia, route them to an Asia-region endpoint. Most major providers now offer regional deployment, and the latency difference can be 300-500ms — larger than the difference between the fastest and slowest models. This is free performance.

And fifth, measure continuously. The