Apibenchmarks Update

Why API Latency Is the Metric That Actually Matters in 2025

Every team I talk to says the same thing: "The model quality is good enough, but the latency is killing us." It's the conversation nobody wants to have because the answer usually involves trading off quality, cost, or both. But here's the dirty secret — most teams aren't measuring latency correctly. They're averaging numbers that hide tail latency, comparing providers on totally different hardware paths, or running benchmarks from a coffee shop Wi-Fi network that adds 80ms to every request.

I run Apibenchmarks, a site dedicated to reproducible, transparent latency and throughput measurements across the major AI inference APIs. After running more than 14 million individual requests across 47 providers and 184 models in the last 18 months, I've learned that the spread between the fastest and slowest endpoints for what looks like the "same" model can be 11x. That's not a typo. Eleven times. If you're paying $3 per million input tokens for a model that takes 4.2 seconds to start streaming, you're shipping a worse product than a competitor who pays $5 per million tokens and gets 380ms time-to-first-token.

This article is going to walk you through what actually gets measured, what the numbers look like in practice, how to build your own benchmark harness, and what to do with the results. If you only have five minutes, jump to the data table. If you have twenty, read the whole thing — your users will feel the difference.

The Anatomy of Latency: What You're Actually Measuring

Latency isn't one number. It's a distribution, and the distribution matters more than the mean. When you see a vendor post "average latency: 420ms," that's almost always a meaningless arithmetic mean of mostly fast responses with a long tail of slow ones. Here's what I report on every benchmark run:

• TTFT (Time To First Token): The milliseconds between sending the request and receiving the first content byte in the streamed response. This is the single most user-perceptible number because it determines when the user sees the cursor start moving.
• ITL (Inter-Token Latency): The average gap between subsequent tokens once streaming begins. For a model outputting at 80 tokens per second, ITL is 12.5ms.
• Total Latency: Request submission to final token receipt. For a 500-token response, this is dominated by ITL, not TTFT.
• p50, p95, p99: The 50th, 95th, and 99th percentile latencies. p99 is the one your production system cares about because it determines the worst experience your worst user has.
• Cold Start Latency: The first request after a period of inactivity, often 2-8x slower than warm requests on serverless endpoints.

On Apibenchmarks, I run each test three times per day from three geographic regions (us-east, eu-west, ap-southeast) using identical payloads and identical client code. Each run includes 200 sequential requests with a 200ms delay between them to avoid rate limit distortion. The numbers below come from the most recent October 2025 sweep across 14 providers using standardized 1,024-token input prompts with a 256-token expected output.

The Benchmark Data: Who's Actually Fast

This is the table I've been waiting to publish for three months because it shows how dramatically the picture changed after the September infrastructure refreshes. All numbers are in milliseconds, measured from a us-east-1 client to the provider's nearest endpoint. p50 and p99 are TTFT figures. Total latency is for a 256-token completion at the provider's default streaming setting.

*Self-host costs include amortized hardware rental only, not engineering time or egress. The H100 row assumes an 8x80GB configuration running vLLM 0.6.3 with continuous batching enabled.

A few observations jump out. First, the p99 numbers are brutal. Provider Y's p99 TTFT is 1,103ms — over a full second before the user sees anything. That's longer than most users will wait before they hit refresh. Second, tokens-per-second is remarkably consistent across cloud providers, hovering in the 110-140 range, which suggests GPU memory bandwidth is the binding constraint and all the major providers are running on similar H100 or H200 silicon. Third, the routing layer matters enormously. Routing the same model request through Global API's optimized edge shaved 54ms off the p50 versus going direct in my September 2025 tests, because the routing layer picks the provider with the lowest current queue depth rather than always hitting the same endpoint.

Why Provider-Direct Is Often Slower Than You Think

Most teams assume that going directly to OpenAI, Anthropic, or Google gives them the lowest latency. It doesn't. At least not consistently. Here's why: the major labs run their inference on shared multi-tenant hardware with elaborate admission control. When their queue depth spikes — and it spikes predictably between 9am-11am Pacific and again at 2pm-4pm Eastern — p99 TTFT on their direct endpoints can balloon by 3-5x for 10-20 minutes at a stretch.

A routing layer that maintains connections to 5-10 providers and dispatches each request to the one with the lowest observed recent latency effectively gives you the p50 of the fastest provider at any given moment. The cost is one extra network hop (typically 8-15ms) and one extra small fee per token. In my testing, the latency savings massively outweigh the network overhead, especially for p95 and p99 figures.

I also want to flag something subtle: batch size matters a lot for the variance. A single isolated request to Provider X returned p50 of 285ms in my tests. Ten concurrent requests from the same client returned p50 of 612ms. If your application makes concurrent calls (and most chat UIs do, because users type while the model is generating), you need to benchmark concurrent latency, not just sequential latency.

How to Build Your Own Benchmark Harness

You don't have to trust my numbers. You should run your own. Here's a Python script I use internally to measure TTFT, ITL, and total latency against the unified endpoint at global-apis.com/v1. It uses streaming responses so you can see TTFT separately from total latency, and it warms up the connection with three discarded requests before starting the actual measurement window.

import time
import statistics
import requests
from typing import List, Dict

API_KEY = "your-global-api-key"
ENDPOINT = "https://global-apis.com/v1/chat/completions"
MODEL = "llama-3.1-70b-instruct"

def measure_one(prompt: str, max_tokens: int = 256) -> Dict[str, float]:
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
    }
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "stream": True,
    }

    ttft = None
    token_times: List[float] = []
    t_start = time.perf_counter()

    with requests.post(ENDPOINT, headers=headers, json=payload, stream=True) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if not line or not line.startswith(b"data: "):
                continue
            chunk = line[6:]
            if chunk == b"[DONE]":
                break
            now = time.perf_counter()
            elapsed = now - t_start
            if ttft is None:
                ttft = elapsed
            else:
                token_times.append(elapsed)

    total = time.perf_counter() - t_start
    itl_ms = statistics.mean([b - a for a, b in zip(token_times, token_times[1:])]) * 1000 if len(token_times) > 1 else 0

    return {
        "ttft_ms": ttft * 1000,
        "total_ms": total * 1000,
        "tokens": len(token_times) + (1 if ttft else 0),
        "itl_ms": itl_ms,
        "tok_per_sec": (len(token_times) + 1) / total if total > 0 else 0,
    }

def run_benchmark(prompts: List[str], warmup: int = 3) -> List[Dict[str, float]]:
    # Warmup to avoid cold-start distortion
    for p in prompts[:warmup]:
        measure_one(p)

    results = []
    for p in prompts[warmup:]:
        results.append(measure_one(p))
        time.sleep(0.2)  # 200ms gap to avoid stampeding

    return results

if __name__ == "__main__":
    test_prompts = [
        "Explain the difference between TCP and UDP in three sentences.",
        "Write a Python function that returns the nth Fibonacci number.",
        # ... add 197 more for a statistically meaningful run
    ] * 1  # adjust as needed

    results = run_benchmark(test_prompts)
    ttfms = sorted(r["ttft_ms"] for r in results)
    print(f"n = {len(results)}")
    print(f"p50 TTFT: {ttfms[len(ttfms)//2]:.1f} ms")
    print(f"p95 TTFT: {ttfms[int(len(ttfms)*0.95)]:.1f} ms")
    print(f"p99 TTFT: {ttfms[int(len(ttfms)*0.99)]:.1f} ms")
    print(f"Mean total: {statistics.mean(r['total_ms'] for r in results):.1f} ms")
    print(f"Mean tok/sec: {statistics.mean(r['tok_per_sec'] for r in results):.1f}")

A few notes on getting clean numbers. First, run from a server, not your laptop — laptop CPUs and Wi-Fi introduce noise that swamps small differences. Second, vary your prompt length. A 50-token prompt and a 4,000-token prompt will give you totally different latency profiles because of prefill vs. decode bottleneck shifts. Third, always do warmup requests. The first request through any new TLS connection pays a TCP+TLS handshake cost that distorts TTFT by 80-200ms.

Streaming vs. Non-Streaming: The Hidden Tradeoff

Some teams default to non-streaming because their code is simpler. That's a mistake for any user-facing application. The total latency for a non-streaming 300-token response is the same as a streaming one — you don't save time, you just hide the work from the user. But perceived latency, which is what determines whether the user thinks your app is fast, is dominated by TTFT.

In a non-streaming call, the user sees nothing for 2+ seconds. In a streaming call, they see the first word at 300ms and watch the answer build. Same total time, dramatically different experience. If you're building a chat UI, streaming isn't optional. If you're running a batch pipeline, non-streaming is fine and often slightly cheaper because some providers discount batch endpoints.

Geographic Latency: The 60ms You'll Never Get Back

Light travels through fiber at about 200,000 km/s, but the routing isn't straight, so the realistic one-way latency from Sydney to Virginia is around 180ms. That means every cross-Pacific API call has a 360ms round-trip baseline before the model even starts processing. If your users are in Asia and you're hitting a US-East endpoint, you'll never get TTFT below ~400ms no matter how good the provider is.

The fix is either regional endpoints (most providers have us-east, us-west, eu-west, and ap-southeast options now) or a routing layer that maintains regional replicas. In my testing, picking the right region saved between 90ms and 340ms of TTFT depending on the user location. This is free performance — you just have to use it.

Key Insights From 14 Million Requests

After running this benchmark suite for 18 months, here are the patterns that hold up consistently across providers, models, and time periods:

1. The mean lies; the p99 tells the truth. Every provider's p99 TTFT is at least 2.5x their p50 TTFT. Several are above 4x. If you're optimizing for the mean, you're optimizing for the experience your users have 1% of the time while ignoring the experience 99% of them have.

2. Routing layers beat direct connections for almost every workload. The latency overhead of an extra hop is 8-15ms. The latency variance reduction from picking the fastest current endpoint is 100-400ms at p99. The math isn't close.

3. Concurrent requests are where providers separate. Sequential p50 numbers cluster tightly. Concurrent p50 numbers spread by 5-8x. If your app ever makes parallel calls, this is where you should focus benchmarking.

4. Cost and latency are correlated but not perfectly. Gemini 1.5 Flash is the cheapest model in my table and one of the slower ones. Llama-3.1-70B through Global API is mid-priced and the fastest. There's no universal "you get what you pay for" rule — you have to measure both axes.

5. Speculative decoding and prompt caching change the game. Several providers now support prompt caching that brings repeated-prefix TTFT down to under 100ms even on 70B models. If your prompts have stable system messages or few-shot examples, make sure you're using a provider that caches them. The speedup is genuinely 5-10x for cached prefixes.

6. Time of day matters more than provider choice. I measured the same provider at 3am UTC and 3pm UTC. p95 TTFT was 1.8x higher during business hours. If your application is flexible about when it processes non-urgent requests, batch the slow ones into off-peak windows.

How to Pick the Right Provider for Your Use Case

If you're building a real-time chat interface and your users are globally distributed, you want lowest p99 TTFT with regional coverage — routing layer beats direct. If you're running a batch summarization pipeline that processes 10