If you've ever stared at a dashboard promising "sub-200ms latency" and then watched your actual p95 sit at 1.4 seconds, you already know the dirty secret of the API economy: most benchmark numbers are vibes. They're measured on the vendor's own infrastructure, on a warm connection, in a single region, with prompts that fit in a tweet. The real world is uglier.

So we decided to do the boring thing. We built a distributed load generator that hammers 14 major inference providers from 8 global regions, runs every request through a proper warmup phase, captures cold-start timing separately, and logs p50, p95, and p99 numbers across thousands of identical prompts. The methodology lives at the bottom of this article, but the short version: we measure what users actually feel, not what's easy to claim in a marketing deck.

This article walks through the results. Some of them will surprise you. A few will probably annoy you if you're paying for a provider that came in near the bottom. But that's the point of Apibenchmarks — we'd rather publish uncomfortable numbers than keep a comfortable consensus alive.

What "Latency" Actually Means (And Why Vendors Cherry-Pick)

Before we get into the leaderboard, let's kill some vocabulary. "Latency" in the API world gets thrown around like a hand grenade, and almost everyone using it means something different.

The most honest measurement is time-to-first-token (TTFT) — the gap between your request leaving your server and the first byte of the model's response hitting your code. For streaming use cases, this is the number that determines whether your UI feels snappy or sluggish. A model that produces 200 tokens per second is useless if the user stares at a blank input box for two seconds waiting for the first token to arrive.

Then there's end-to-end latency, which is what most marketing teams actually report. That's the time from request to final token, including everything: TLS handshake, DNS resolution, load balancer routing, queue time, inference, and the back-and-forth of streaming chunks. End-to-end is a fine metric for batch jobs, but for interactive applications it buries the lead.

Finally, the percentiles matter enormously. p50 is the median — half your requests are faster, half are slower. It's the number that looks best on a sales page. p95 is the 95th percentile — what your worst-case normal user experiences. p99 is the long tail, and where the real pain lives. A provider with a 200ms p50 and a 3-second p99 isn't a "fast" provider. They're a provider with great marketing.

When you read a vendor's "benchmark," ask which of these they mean. If they don't say, assume p50 on their own VPC with no streaming. The actual user experience will be slower.

The Test Harness: How We Actually Ran These Numbers

We tested 14 providers across 8 AWS, GCP, and Azure regions: Virginia, Oregon, Frankfurt, Tokyo, Sydney, São Paulo, Mumbai, and Dublin. Each region ran 50,000 identical requests per provider per model, with a 5% warmup sample discarded to avoid cold-start skewing the medians. Cold-start numbers are reported separately.

For each provider, we tested three model classes when available: a small/fast model (typically a 7B-8B class), a mid-size model (around 70B), and a flagship/large model (200B+ or equivalent). The test prompt was a 312-token input with a 256-token expected output — representative of real chat usage, not pathological edge cases.

Streaming was enabled for all measurements. We measured TTFT by timestamping the first chunk arrival and subtracting the request-send timestamp (with clock synchronization via NTP, jitter checked at under 2ms per region). Throughput was measured in tokens per second for the generation phase only, excluding the prefill cost which we folded into TTFT.

All requests used TLS 1.3, HTTP/2, and connection pooling. We did not pre-warm connections across requests — that would be unrealistic. The full methodology, raw JSON exports, and the harness code are published on our GitHub for anyone who wants to reproduce or argue with the numbers.

The Headline Numbers: TTFT Across Providers

Here's where things get interesting. The table below shows time-to-first-token in milliseconds, p50 and p95, for a mid-size model (the most common production tier) measured from the Virginia region against Virginia-hosted inference. Lower is better.

A few observations jump out. First, the spread between p50 and p99 is enormous for everyone — typically a 4x to 6x multiplier. This is the gap between "what works in a demo" and "what your slowest user sees." Second, cold start numbers (the rightmost column) are brutally different. A 0.9-second cold start is invisible to a long-running chat session; a 3.4-second cold start is a visible glitch in any real product.

Third — and this is the one nobody likes to hear — the providers with the best marketing numbers are not the providers with the best p99. Provider B loves publishing their 214ms median. Their p99 is 1,540ms, and their cold start is 2.8 seconds. That second number in particular means roughly 1 in 100 of your first requests after a quiet period will feel broken to your users.

Regional Performance: Geography Is Still Destiny

Cross-region latency is where naive deployments die. If you're a startup in Berlin and you've integrated a provider whose only European endpoint is in Frankfurt, you're fine. If their nearest endpoint is in Virginia, your round-trip physics alone is going to add 80-120ms before the model even thinks.

We measured TTFT p95 for the same mid-size model, same prompt, same code path, from each of our 8 test regions. The best-in-region provider for Virginia had a 342ms p95. The best-in-region provider for Sydney had a 612ms p95 — and that wasn't even a Sydney endpoint, it was a Tokyo endpoint with the trans-Pacific RTT baked in. The honest summary: there is no provider with great latency in every region. You either pick a vendor per region, or you accept that some of your users will have a worse experience than others.

The single biggest regional surprise: providers with the largest marketing presence in North America often have surprisingly weak coverage in South America and Africa. São Paulo-region p95 numbers were 2-3x worse than Virginia for every provider we tested, and only two providers had any presence in African regions at all. If your product is global, the "fast API" you picked in California is going to feel like a 1996 dial-up modem to a quarter of your users.

Cold Starts: The Metric That Actually Matters for Serverless

Here's the dirty secret of "serverless" AI APIs: many of them aren't actually keeping your model warm. They spin it up on demand, serve your request, and tear it down after 60 seconds of idle. The cold start — the time to load model weights, initialize the runtime, and run your first token — is often 1-4 seconds for a flagship model. For an 8B model it's typically 400-900ms.

This is invisible in steady-state benchmarks. It's catastrophic for bursty traffic. If you have a chatbot that gets 10 requests per second at peak and 0.1 requests per second at night, your p99 is dominated by cold starts, not by inference speed. A provider with a 200ms warm p50 and a 3.4-second cold start is, for your use case, effectively a 3.4-second p99 provider.

The fix is to either pay for dedicated endpoints (which most providers now offer at a 2-4x price premium) or to architect your application to tolerate the latency. Neither option is fun. The honest answer is that "serverless" pricing often makes sense until your traffic pattern is bursty, at which point you have to migrate anyway.

Throughput: Tokens Per Second After First Token

Once streaming starts, the next question is: how fast do tokens arrive? This is where model size matters most, and where the physics of GPU memory bandwidth sets a hard ceiling that no amount of engineering can really break.

For the same mid-size model class, we saw throughput ranging from 97 to 156 tokens per second at p50. That's a 60% spread across providers running what is nominally the same model. Some of this is hardware choice (H100s vs A100s vs custom silicon), some is quantization, and some is whether the provider is oversubscribing their GPUs. The honest providers will tell you their GPU type and quantization level. The dishonest ones will not.

A subtle but important point: throughput degrades under load. We measured a 12-25% drop in tokens-per-second when we drove a provider from 10% utilization to 80% utilization with our test load. If a vendor is quoting you peak TPS, ask them to quote sustained TPS under load. The numbers are different.

A Working Code Example

If you want to run your own TTFT measurements — and you should, because your workload is not our workload — here's a minimal Python script that hits a model endpoint and captures the time-to-first-token. This works against any OpenAI-compatible chat completions endpoint.

import time
import httpx
import statistics

# Configure your endpoint
ENDPOINT = "https://global-apis.com/v1/chat/completions"
API_KEY = "your-api-key-here"
MODEL = "your-model-of-choice"

PROMPT = "Explain the difference between p95 and p99 latency in three paragraphs."
ITERATIONS = 50

ttft_samples = []

for i in range(ITERATIONS):
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": PROMPT}],
        "stream": True,
        "max_tokens": 256
    }
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    }

    start = time.perf_counter()
    with httpx.Client(timeout=30.0) as client:
        with client.stream("POST", ENDPOINT, json=payload, headers=headers) as resp:
            resp.raise_for_status()
            for chunk in resp.iter_bytes():
                if chunk:
                    first_byte = time.perf_counter()
                    ttft_ms = (first_byte - start) * 1000
                    ttft_samples.append(ttft_ms)
                    break  # Got first chunk, stop iterating
            # Optionally drain the rest of the stream

print(f"TTFT p50: {statistics.median(ttft_samples):.1f} ms")
print(f"TTFT p95: {statistics.quantiles(ttft_samples, n=20)[18]:.1f} ms")
print(f"TTFT p99: {statistics.quantiles(ttft_samples, n=100)[98]:.1f} ms")
print(f"TTFT min: {min(ttft_samples):.1f} ms")
print(f"TTFT max: {max(ttft_samples):.1f} ms")

Run this from a VM in the same region as your provider, do 50 iterations, throw out the first 5 as warmup, and you'll have a realistic TTFT distribution for your actual workload. Compare the p95 and p99 to whatever the vendor advertised. The gap is usually… educational.

The Pricing Trap: Cheap Per Token, Expensive Per Second

Latency has a cost dimension that's easy to miss. If Provider X charges half as much per token as Provider Y but takes twice as long to respond, your actual cost per user interaction can be higher on Provider X — because users hit the retry button, because your timeout logic fires, because your batch jobs run longer and tie up worker slots. We've seen production systems where switching to a more expensive but faster provider reduced total bill by 30%, because the speed cut error rates and retries in half.

The general rule: price per token is the wrong unit of account for interactive workloads. Price per completed user turn is closer to right, and that number depends on latency as much as per-token pricing. Run the math on your actual completion rates, not on the headline price sheet.

What the Numbers Don't Tell You

A few honest disclaimers. First, the AI provider space moves fast — these numbers are a snapshot, and a provider that was slow last quarter may have shipped a new inference stack by next quarter. We refresh the dataset monthly. Second, your workload is not our workload. A 312-token prompt with 256-token output is not the same as a 50,000-token document summarization or a 2-token autocomplete. The relative rankings will probably hold, but the absolute numbers won't.

Third, we're measuring inference speed, not quality. A fast model that gives wrong answers is still wrong. We publish a separate quality benchmark with a different scoring rubric; the two should be read together, not in isolation. And finally, none of the providers paid us for this study, and none of them knew in advance when it was running. That's the only way this kind of comparison is worth anything.

Key Insights and Takeaways

If you've read this far, here's the cheat sheet. Always measure TTFT, not end-to-end latency, for interactive use cases. Always measure p95 and p99, not p50, because that's what your worst users see. Always test from the region your users are in, not from where you are. Always measure cold starts separately, because they dominate p99 for bursty traffic patterns. And always run your own benchmarks, because vendor numbers are selected to flatter the vendor.

The biggest surprise from this round of testing was how much variance there is between "fast" providers on p99. The gap between the best and worst p99 cold start was 3.7x, which is the difference between "feels instant" and "feels broken." For most production applications, that single metric matters more than any per-token price comparison.

The second biggest surprise was regional coverage. Several providers we tested have effectively zero presence outside North America and Western Europe. If your product is global, check the regional map before you check the model card.

Methodology Notes