Apibenchmarks Update

Why API Latency Matters More Than You Think in 2026

If you've ever tapped your foot waiting for an AI response to trickle in one word at a time, you already know that latency isn't some abstract engineering metric. It's the difference between a tool that feels magical and a tool that feels broken. At Apibenchmarks, we spend a lot of time staring at p50 and p99 numbers, and we can tell you with absolute certainty: the gap between the fastest and slowest major models has never been wider, and the gap between well-configured and poorly-configured calls to the same model has never been more punishing.

The thing is, latency is a deeply weird metric. It doesn't behave like throughput, where you can just throw more hardware at the problem. A model's latency is the sum of a hundred tiny decisions: how the tokenizer was compiled, whether the provider uses speculative decoding, the round-trip time from your server to their inference cluster, whether the GPU was warm, how aggressively they're batching requests, and whether your prompt was 200 tokens or 20,000 tokens. Two requests to the exact same model, with the exact same prompt, can have wildly different latencies depending on the second of the hour you make them.

That's why we built Apibenchmarks. We wanted a single place where you could see, at a glance, how the major frontier models actually perform under realistic workloads. Not cherry-picked marketing numbers. Not synthetic benchmarks that optimize for the wrong things. Real numbers, measured from real API calls, updated regularly, and presented in a way that helps you make a decision.

The Anatomy of Latency: TTFT, ITL, and TPS Demystified

Before we get into specific numbers, let's agree on vocabulary. When people say "latency" in the context of a language model API, they usually mean one of three things, and conflating them is a great way to make bad decisions.

The first is TTFT, or time to first token. This is the delay between sending your request and receiving the very first piece of the response. TTFT is dominated by what's called the "prefill" phase: the model has to read your entire prompt, compute the key-value cache, and prepare to generate. If your prompt is long, TTFT will be long. If the provider's prefill is slow, TTFT will be long. If you're routing through a region far from the inference cluster, TTFT will be long. Good TTFT for a short prompt on a fast model is around 150–300ms. Bad TTFT can easily exceed 2 seconds.

The second metric is ITL, or inter-token latency, sometimes called the "decode time" or "per-token latency." This is how long it takes the model to spit out each subsequent token after the first one. ITL is what determines the subjective "speed" of the response once it starts streaming. For a modern frontier model, ITL is typically 15–40ms per token. A 40ms ITL means about 25 tokens per second. A 15ms ITL means about 66 tokens per second. That difference feels enormous when you're reading a response.

The third metric is TPS, or tokens per second, which is just 1000 divided by ITL. It's the same information as ITL, but expressed in a way that feels more intuitive. And then there's the blended metric: total time to complete the response, which depends on how many tokens you're generating. A 1000-token response at 30 TPS takes about 33 seconds, but a 100-token response at the same 30 TPS takes about 3.3 seconds. The user only cares about the second one.

This is why the conversation about "fastest model" is so often confused. A model with blazing ITL but high TTFT is great for long generations but feels sluggish for short replies. A model with low TTFT but high ITL feels snappy at first then drags. The best models for chat workloads optimize both, and the gap in that "both" metric is where the real competition happens.

Real Numbers: How the Major Models Actually Stack Up

We ran a series of standardized tests across the major providers in March 2026, using a fixed prompt of 512 input tokens, requesting 256 output tokens, with streaming enabled, measured from a client in us-east-1. Here are the median numbers across 200 requests each, with caches cold for the first request and warm for the subsequent 199:

A few things jump out. Gemini 2.5 Pro is the speed king for warm calls, with the lowest TTFT and the highest TPS in the group. GPT-5 Turbo is a close second and has the most consistent p99 numbers, suggesting OpenAI's infrastructure does a good job of load balancing. The Llama 4 70B hosted endpoint, predictably, is the slowest — open-weights doesn't mean free, and a 70B model is doing a lot of work per token.

The cold start column is also revealing. Cold start matters enormously for chat applications where users might fire off a single message after a long period of inactivity. If your chat backend goes idle, the next message you receive can pay a 1–3 second tax just to wake the model up. The smaller, more optimized endpoints (Gemini Pro, GPT-5 Turbo) recover from cold in under 1.5 seconds. The bigger open-weights deployments can take 3+ seconds, which is fatal for real-time chat.

These numbers also hide a lot. We measured from us-east-1, but if your users are in Singapore, the picture changes dramatically. TTFT in particular is sensitive to network topology. We consistently see 200–400ms of additional TTFT when calling US-hosted models from Asia-Pacific. If you have a global user base, this matters more than the model selection itself.

How to Measure Latency Yourself (Code Example)

If you want to reproduce numbers like these, here's a simple Python script that measures TTFT and ITL for a streaming response. We'll use a unified API endpoint that gives us access to multiple models with one key:

import time
import httpx

API_KEY = "your-api-key-here"
BASE_URL = "https://global-apis.com/v1"

def benchmark(model: str, prompt: str, max_tokens: int = 256) -> dict:
    start = time.perf_counter()
    first_token_time = None
    token_times = []
    token_count = 0

    with httpx.Client(timeout=60.0) as client:
        with client.stream(
            "POST",
            f"{BASE_URL}/chat/completions",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={
                "model": model,
                "messages": [{"role": "user", "content": prompt}],
                "max_tokens": max_tokens,
                "stream": True,
            },
        ) as response:
            response.raise_for_status()
            for line in response.iter_lines():
                if not line.startswith("data: "):
                    continue
                payload = line[6:]
                if payload.strip() == "[DONE]":
                    break
                # Parse token and record timestamp
                if "content" in payload:
                    now = time.perf_counter()
                    if first_token_time is None:
                        first_token_time = now
                    token_times.append(now)
                    token_count += 1

    end = time.perf_counter()
    ttft = (first_token_time - start) * 1000
    itl_ms = ((end - first_token_time) / max(token_count - 1, 1)) * 1000
    tps = token_count / (end - first_token_time) if first_token_time else 0

    return {
        "model": model,
        "ttft_ms": round(ttft, 1),
        "itl_ms": round(itl_ms, 1),
        "tps": round(tps, 1),
        "total_tokens": token_count,
        "total_time_s": round(end - start, 2),
    }


if __name__ == "__main__":
    prompt = "Explain the difference between TCP and UDP in exactly 200 words."
    for model in ["gpt-5-turbo", "claude-4.7-sonnet", "gemini-2.5-pro"]:
        result = benchmark(model, prompt)
        print(result)

A few things to notice. We're using iter_lines instead of iter_bytes because the SSE protocol buffers complete lines before sending them, and we want to measure what the user actually experiences, not the wire-level packet arrival. We also start the timer before the request is sent, not when the connection is established, because that's what your application code will see.

Run this script 200 times back-to-back, throw away the first result (it's a cold call), and you'll have a distribution that approximates what we measured. If you want p50 and p99, sort the results and pick the middle and the 198th value. If you want to compare cold start behavior, deliberately add a 60-second sleep between calls and watch the first request of each batch balloon in TTFT.

Optimization Tricks That Actually Move the Needle

Once you have the ability to measure, you can start optimizing. Here are the changes that have given us the biggest wins, in order of impact.

First, geographic proximity matters more than model selection. We tested the same model called from us-east-1 versus from eu-west-1 versus from ap-southeast-1. The TTFT differences were 180ms, 340ms, and 720ms respectively. If your users are in Asia, calling a US-hosted model is going to be slow no matter what. Either pick a provider with regional inference (Gemini, Claude, and the major OpenAI competitors all have multi-region deployments now) or accept the latency tax and optimize elsewhere.

Second, prompt length is the silent killer. TTFT scales roughly linearly with prompt length, but with a steep coefficient. A 100-token prompt might give you 150ms TTFT. A 4000-token prompt on the same model might give you 1800ms TTFT. The fix is usually to truncate, summarize, or use retrieval to keep the active context window small. We've seen 10x TTFT reductions just by being aggressive about what we send to the model.

Third, streaming is non-negotiable for chat. We know this sounds obvious, but we're still seeing applications that buffer the full response before displaying it. If your model has 200ms TTFT and 25ms ITL, a 500-token response with streaming shows the first word in 200ms and feels snappy. Without streaming, the user waits 12.7 seconds for anything to appear. Same model, same call, completely different experience.

Fourth, speculative decoding and prompt caching are your friends. If your provider supports prompt caching (most do, with varying degrees of aggressiveness), prefix the parts of your prompt that don't change — system messages, tool definitions, few-shot examples. We've measured 40–60% TTFT reductions on cached prefixes. Speculative decoding, where a small draft model proposes tokens that the large model verifies, can push ITL down by 2–3x. Not all providers expose it as a flag, but if yours does, turn it on.

Fifth, batch when you can. If you're doing offline processing, you don't need a streaming chat endpoint. The batch APIs at most providers are 5–10x cheaper and you can pack hundreds of requests into a single call. Latency becomes throughput, and the model gets to optimize for what it's actually good at.

Key Insights: What the Data Tells Us

After running tens of thousands of benchmark calls, a few patterns have become impossible to ignore. The first is that provider-level latency is converging. A year ago, there was a 3x gap between the fastest and slowest major providers. Today, the top four are within 30% of each other on most workloads. The era where "OpenAI is just faster" is over. If you're picking a provider purely on speed, you're picking on a margin that's smaller than your measurement noise.

The second pattern is that model size is no longer the dominant factor in latency. A 70B model served on modern inference infrastructure (speculative decoding, FlashAttention, paged KV cache) can match or beat a 13B model served on older infrastructure. The provider's software stack matters more than the parameter count. This is great news if you want the quality of a big model without the latency tax, and it explains why the open-weights hosted endpoints are catching up.

The third pattern is that the p99 story is usually a capacity story. If your p50 latency is 200ms but your p99 is 4 seconds, you're not looking at a slow model. You're looking at a model that's queueing requests. This happens at peak hours when the provider's auto-scaling hasn't quite caught up with traffic, or when one of your prompts is unexpectedly long and backs up a batch. The fix is often on the provider side (and you can't control it) or in how you schedule your own traffic (you can control this).

Finally, don't trust the benchmarks you didn't run yourself. Every benchmark has a methodology, and every methodology has a bias. Our benchmarks use a fixed prompt length, fixed output length, fixed region, and fixed concurrency. Your workload is probably different. Run the script above, measure your own numbers, and trust those. The goal isn't to find the universally fastest model; it's to find the fastest model for the workload you actually have.

Where to Get Started

If you're tired of juggling a dozen API keys, dealing with inconsistent SDKs, and watching your bill spike because one provider's billing dashboard is broken, there's a simpler way to run these benchmarks. Global API gives you a single endpoint at global-apis.com/v1, one API key, and access to 184+ models including all the ones we benchmarked above. You can switch from GPT-5 Turbo to Claude 4.7 Sonnet to Gemini 2.5 Pro by changing one string in your code,