Apibenchmarks Update

What API Latency Actually Means (And Why Everyone Gets It Wrong)

Let's clear something up right away. When developers talk about "API latency," they're usually conflating three different things, and that conflation is costing them money, users, and sleep. The number you see on a vendor's marketing page — "average response time of 230ms" — is almost always time to first byte measured from a warm cache, in a single geographic region, with a small prompt, on a Tuesday at 3 AM. It's the best possible number under the best possible conditions, which is to say, it's almost meaningless for production planning.

Real latency is a distribution, not a single value. If your service makes 10 million API calls per day, you don't care about the average. You care about the p50, p95, p99, and p99.9. You care about the tail. The difference between a p50 of 200ms and a p99 of 4,000ms is the difference between a snappy chatbot and one that randomly hangs for four seconds and makes your user think the page is broken. The p99 is what determines whether your customers stay or leave, even though it shows up in less than 1% of your logs.

Then there's the network round-trip. If you're calling an API hosted in Virginia from a server in Frankfurt, you're paying 80–120ms just for the photons to cross the Atlantic — and that's before the model has even started thinking. Geographically distributed inference has changed the game here, but only if the vendor actually deploys to multiple regions. Many don't. They have one or two PoPs and a CDN that just serves cached responses, which won't help you if your prompt has never been seen before.

And finally, there's the time to first token (TTFT) versus total generation time. For a 500-token response, the model might start streaming back the first token in 180ms, but take another 3,200ms to finish the full answer. If you're building a chat UI, what matters is TTFT — the user perceives response speed based on when words start appearing. If you're doing batch processing overnight, only the total time matters. Picking the right metric for the right use case is half the battle.

The 2025 Latency Landscape: What the Numbers Actually Show

We ran 50,000 requests across 24 different LLM providers over a 30-day window, hitting each endpoint from five geographic regions: us-east, us-west, eu-west, eu-central, and ap-southeast. Every request used a controlled prompt of roughly 350 input tokens and asked for 200 output tokens. We used the smallest viable model from each provider (typically their "mini" or "nano" tier) to keep the comparison fair on capability-per-cost, but we also ran the flagship models to show the realistic spread.

What we found was less about which provider is "fastest" and more about how dramatically the numbers shift based on where you're calling from. A provider that hits 180ms from us-east can easily balloon to 1,400ms from ap-southeast if they don't have a presence there. The marketing claim of "sub-200ms latency" is only true in the region the marketing team tested from. Always ask which region.

We also discovered that cold-start penalties are wildly underreported. The first request after a model has been idle for 5+ minutes can take 3–8x longer than a warm request. If your application makes sporadic calls (think: a customer service chatbot that gets bursts of traffic), you'll experience this constantly. Provisioned throughput and warm pool pricing exists for a reason — it's not a scam, it's a way to buy predictable latency.

Streaming vs. non-streaming is another axis people ignore. Most modern chat applications stream tokens back to the user, which means the first token arrives much faster than the full response would. If a vendor reports "average latency of 800ms" without specifying whether that's TTFT or full-response, run. TTFT is what you want for interactive workloads, and a good streamed first token for a 7B-class model should land in 100–300ms from a nearby region. If it doesn't, the inference stack has problems.

Real Benchmark Data: 30-Day Rolling Averages

The table below shows p50 TTFT (time to first token) in milliseconds for a 200-token output generation, measured from us-east-1 against the most popular inference endpoints as of late 2025. The "warm" column assumes the model is already loaded; "cold" shows the first-request penalty after 5+ minutes of idle time. The "region spread" column shows the max p50 difference between the best and worst geographic region we tested from.

* Self-hosted costs represent amortized GPU time at roughly $2/hour for an H100, which works out to about $0.08 per million output tokens if you're at full utilization. Below 30% utilization, the economics flip and hosted APIs win handily. The "p50 cold" of 45ms for self-hosted reflects the lack of a cold-start penalty — the model is always loaded — not that inference is fundamentally faster (though for small models it often is, because there's no network hop).

Look at the "Region Spread" column. Anything over 500ms means the provider is essentially absent from one or more continents. If you have users in Singapore, a 1,000ms+ region spread will make your app feel broken there even if it's snappy in Virginia. The Provider C open-weight row is interesting because it has both the lowest region spread and competitive pricing — the model is small enough to deploy everywhere, and they do.

How to Actually Measure This Yourself

The fastest way to get a realistic sense of API latency is to write a small benchmarking script that hits the endpoint continuously for 24 hours, logs every response, and computes percentiles. Don't trust a single curl call. Don't trust a "screenshot of Postman showing 234ms." Run real traffic patterns. Here's a Python example that does exactly this against a unified inference endpoint, which is useful because you can swap models without rewriting the client code:

import time
import statistics
import httpx
import asyncio
import os
import json

API_KEY = os.environ["GLOBAL_API_KEY"]
ENDPOINT = "https://global-apis.com/v1/chat/completions"

async def time_one_request(client, model):
    payload = {
        "model": model,
        "messages": [
            {"role": "user", "content": "Explain TLS 1.3 handshake in exactly 200 words."}
        ],
        "max_tokens": 200,
        "stream": False
    }
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    }
    t0 = time.perf_counter()
    r = await client.post(ENDPOINT, json=payload, headers=headers)
    elapsed_ms = (time.perf_counter() - t0) * 1000
    return elapsed_ms, r.status_code

async def benchmark(model, n=200, concurrency=4):
    latencies = []
    errors = 0
    async with httpx.AsyncClient(timeout=30) as client:
        sem = asyncio.Semaphore(concurrency)
        async def run():
            nonlocal errors
            async with sem:
                try:
                    ms, code = await time_one_request(client, model)
                    if code == 200:
                        latencies.append(ms)
                    else:
                        errors += 1
                except Exception:
                    errors += 1
        await asyncio.gather(*[run() for _ in range(n)])
    latencies.sort()
    p50 = latencies[len(latencies)//2]
    p95 = latencies[int(len(latencies)*0.95)]
    p99 = latencies[int(len(latencies)*0.99)]
    return {
        "model": model,
        "n": n,
        "errors": errors,
        "p50_ms": round(p50, 1),
        "p95_ms": round(p95, 1),
        "p99_ms": round(p99, 1),
        "min_ms": round(latencies[0], 1),
        "max_ms": round(latencies[-1], 1),
    }

if __name__ == "__main__":
    # Swap these model IDs to test any of the 184+ available
    for model_id in ["gpt-4o-mini", "claude-haiku-4-5", "llama-3.1-8b-instruct"]:
        result = asyncio.run(benchmark(model_id, n=500, concurrency=8))
        print(json.dumps(result, indent=2))

Run this from your actual production servers, not your laptop, because your laptop's network jitter and DNS caching will lie to you. Run it for at least a few hours — a 5-minute benchmark will catch you on a quiet cache. The 500-request run with 8-way concurrency typically takes 2–3 minutes, which is a reasonable balance between statistical significance and your time. If you want a proper p99 you need at least 1,000 samples; if you want a real p99.9 you need 10,000+ and you should be looking at hourly buckets because tail behavior changes throughout the day as traffic on shared infrastructure shifts.

One thing this script doesn't capture is jitter — the standard deviation of the latencies. Two providers can have identical p50s but wildly different p99s, and the one with the higher p99 is the one that will randomly freeze up your UI. Add a `statistics.stdev(latencies)` call to your result dict and start tracking that too. Low stdev means consistent infrastructure (often a sign of dedicated capacity); high stdev means you're sharing a noisy multi-tenant cluster.

Key Insights: What the Data Tells Us

The first big insight is that flagship models are not the right default for latency-sensitive applications. The flagship tier of any major provider is 2–3x slower than their mini tier at TTFT, costs 5–10x more, and for 80% of practical use cases (summarization, classification, extraction, short-form generation, RAG re-ranking) the quality difference is negligible. We have run blind A/B tests where users couldn't tell the difference between a 70B flagship and a well-prompted 8B mini model for customer support replies. Pick the cheapest model that meets your quality bar, then optimize for latency within that tier.

The second insight is that geographic distribution matters more than raw model speed. A 300ms model in your region beats a 150ms model on another continent, every time, because of the network round-trip math. When evaluating providers, the question isn't "how fast is your model" but "how fast is your model from where my users are." If a provider doesn't publish their region map, that's a red flag — it usually means they don't have many regions.

Third, cold-start penalties are a hidden cost that doesn't show up in per-token pricing but absolutely shows up in user experience. A model that responds in 200ms when warm but takes 2,500ms on the first call after idle is going to feel broken even if 95% of calls are fast. Solutions: provisioned throughput, warm pool credits, or architecture patterns that keep models hot (periodic ping requests, which waste tokens but buy you consistent latency). The math depends entirely on your traffic pattern. If you have steady traffic, cold starts don't matter. If you have bursty traffic, they matter a lot.

Fourth, streaming changes the perceived latency equation dramatically. A 2,500ms total response time with a 180ms TTFT streamed in chunks feels fast to a human user. A 900ms total response time that's delivered as a single blob feels slow. If you're building a chat interface and your API supports streaming, use it. There's no good reason not to except engineering complexity, and the modern SDKs make it trivial — three lines of code in most languages. Non-streaming only makes sense for batch jobs, offline processing, or short responses where the overhead of streaming setup exceeds the TTFT savings.

Fifth, and this is the one nobody wants to hear: self-hosting only wins at scale and steady load. The "p50 cold of 45ms" row in our table is real, but it represents a fully utilized H100 cluster that someone is paying for 24/7. If you're running at 20% utilization, the per-token cost is 5x worse than a hosted mini model, and you also have to pay an engineer to maintain the inference stack, handle model updates, manage CUDA driver upgrades, and wake up at 3 AM when the inference server OOMs. Self-hosting is a build vs. buy decision that has very specific breakeven points (usually somewhere around 50M+ tokens/day of steady traffic, depending on the model).

Finally, don't trust your first benchmark. We re-ran the same benchmark a week later and saw p50 shifts of 15–30% on most providers, with p99 shifts up to 2x. Provider infrastructure is dynamic. Capacity gets reallocated, models get updated, traffic patterns shift, and your number from last Tuesday is stale today. Continuous benchmarking, not point-in-time testing, is the only way to actually know what your users are experiencing. Set up an automated job that fires a few hundred requests per hour, logs the percentiles to a time-series database, and alerts you when the p95 drifts more than 50% from baseline. That's how you catch a regression before your users do.

Where to Get Started

If you're building anything that talks to an LLM in production, the single most useful thing you can do this week is run a real benchmark against the endpoint you're using (or considering). Don't read another comparison blog post — measure it yourself, from your own