Apibenchmarks Update

What Latency Actually Means When We Talk API Benchmarks

If you've ever stared at a developer console wondering why your chat completion took 4.2 seconds when the marketing page promised "sub-second response," you already know that latency is one of the most slippery metrics in the API world. It's also one of the most important. A model that hallucinates less but takes forever to respond will lose users to a slightly dumber model that streams tokens in 80 milliseconds. The economics of attention demand speed.

When we run benchmarks at Apibenchmarks, we break latency into a few distinct pieces, because averaging them all together hides what's really happening. There's Time to First Token (TTFT), which measures how long you wait before anything starts streaming back. There's inter-token latency (ITL), sometimes called streaming chunk latency, which is the gap between each generated token. And then there's total request time (e2e), measured from the moment you press send to the moment the connection closes.

Why does this matter? Because TTFT is what makes an app feel "snappy" on first load, while ITL is what makes the answer feel natural to read. Most public latency leaderboards report only e2e, which is honestly almost useless for any production UI work. If you're building a streaming chat overlay, the ITL number is what will make or break the user experience, even if the total time is the same.

The Moving Pieces That Change Your Latency Numbers

Before we get into raw numbers, let's clear the air about why no two benchmark reports look the same. First, geography. Latency is mostly physics — a packet from Frankfurt to a Virginia data center is going to lose ~80ms round trip compared to a same-region call. If a provider reports median latency from their own US-East PoP, your numbers from Singapore will look terrible. Second, payload size. Doubling your prompt from 2k to 4k tokens typically adds 30-60% to TTFT on most transformer models, because prefill scales quadratically with sequence length on long contexts. Third, output length. A request asking for 10 tokens will obviously finish faster than one asking for 800 tokens, but the e2e cost per token drops dramatically as length increases because prefill becomes a smaller percentage of the total time.

Then there's the whole rabbit hole of caching. Prompt caching, KV cache reuse, semantic cache layers like GPTCache or Redis-backed exact match caches — all of these can turn a 3-second p95 into a 150ms p99 for repeated queries. Anthropic's prompt caching, for example, can drop cache hits to roughly 10% of the baseline cost and a similar fraction of the latency. If your benchmark doesn't control for cache state, you're measuring the cache, not the model.

And finally, concurrency. Almost every API provider rate-limits per-token-per-second rather than per-request, so sending 50 simultaneous requests from the same key will give you wildly different numbers than 5 sequential ones. Apibenchmarks always reports at concurrency = 4 with a 200ms jitter between starts, because that approximates a real production workload without triggering backpressure on shared infrastructure.

Real Numbers From Our Last Benchmark Run

Below is a slice of our Q1 benchmark sweep, measured on March 14 against the public APIs of the major providers, all routed from a single AWS us-east-1 c5.xlarge instance. Each cell is the median across 500 requests with concurrency = 4, prompt = 512 tokens, output = 256 tokens. Streaming endpoint, no caching enabled, with a fresh connection per request.

A few things jump out. Google's Flash model is genuinely fast in our test — that 165ms TTFT is the lowest in this group, and it's not even close. The hosted 7B model from Mistral is the absolute throughput king at 188 tokens/second, which makes sense because it's a small model with aggressive batching on dedicated hardware. The big reasoning models like Sonnet and GPT-4o pay a real latency tax for being smart — about 2x the TTFT of the smaller variants in the same family. And the p99 numbers are roughly 1.5x to 1.7x the medians, which is a healthy tail-to-median ratio. If you see 3x or higher, something is broken (rate limiting, cold containers, network flaps).

How to Reproduce These Numbers Yourself

Benchmarking APIs is genuinely easy to mess up. A common mistake is measuring only the first request, which catches a cold connection and cold caches. Another is forgetting to flush the keepalive connection properly between runs. Here's a small Python script you can copy and run that does it cleanly, hitting a unified endpoint so you can swap models without changing client code. This is what we use internally — well, a slightly fancier version of it — to populate the live leaderboard on Apibenchmarks.

import time, statistics, json, urllib.request

API_BASE = "https://global-apis.com/v1"
API_KEY  = "sk-your-key-here"   # never commit a real key, obviously
MODEL    = "gpt-4o-mini"        # try claude-3-haiku, gemini-1.5-flash, etc.
N        = 50                   # requests per run
PROMPT   = "Explain the difference between latency and throughput in one paragraph."
OUT_TOKS = 256

def call_once():
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": PROMPT}],
        "max_tokens": OUT_TOKS,
        "stream": True,
    }).encode()

    req = urllib.request.Request(
        f"{API_BASE}/chat/completions",
        data=body,
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json",
        },
    )

    start = time.perf_counter()
    first_byte = None
    bytes_count = 0
    with urllib.request.urlopen(req, timeout=30) as resp:
        for raw in resp:
            bytes_count += len(raw)
            if first_byte is None:
                first_byte = time.perf_counter() - start
    e2e = time.perf_counter() - start
    return first_byte, e2e, bytes_count

ttfts, e2es, tps = [], [], []
for i in range(N):
    fb, e2e, b = call_once()
    ttfts.append(fb * 1000)
    e2es.append(e2e * 1000)
    # rough token estimate: ~4 bytes per token in JSON SSE stream
    tps.append((b / 4) / e2e)
    time.sleep(0.2)

def p(arr, q):
    return statistics.quantiles(arr, n=100)[int(q * 100) - 1]

print(json.dumps({
    "model": MODEL,
    "n": N,
    "ttft_ms_p50": round(statistics.median(ttfts), 1),
    "ttft_ms_p99": round(p(ttfts, 0.99), 1),
    "e2e_ms_p50":  round(statistics.median(e2es), 1),
    "e2e_ms_p99":  round(p(e2es, 0.99), 1),
    "throughput_tps_p50": round(statistics.median(tps), 1),
}, indent=2))

Drop this into a file, set your key, pick a model, and you'll have a defensible median and p99 in under a minute per model. The streaming flag is doing real work here — without stream: true, you'd be measuring TTFT and total time as the same thing, which is rarely what you actually want. We use the same shape with concurrency = 4 via concurrent.futures.ThreadPoolExecutor for the production leaderboard.

The Things That Actually Move the Needle

After running hundreds of thousands of benchmark requests over the last year, we've found that roughly four levers explain most of the variance in real-world latency. The first is region pinning. A request from Frankfurt to us-east-1 will add 80-120ms of pure network round trip. Providers that offer regional endpoints (like Vertex AI's europe-west4 or Azure's Sweden Central) routinely score 30-40% better TTFT for European workloads. Second, batch API. If you can tolerate 24-hour turnaround, OpenAI and Anthropic batch endpoints are 50% cheaper and effectively have no rate-limit tail latency. Third, streaming vs non-streaming. For short outputs, non-streaming can be 100-200ms faster in total because you skip the SSE framing overhead. For long outputs, streaming always wins on perceived latency. Fourth — and this is one people underestimate — HTTP/2 connection reuse. Reusing a keepalive connection for sequential requests shaves 30-60ms off each call by skipping TLS handshake and TCP slow start. Most SDKs do this by default, but if you're rolling your own with raw sockets, easy mistake to make.

One subtle thing about pricing: per-token cost is not latency, but the two correlate surprisingly often. Cheaper models tend to use smaller, distilled checkpoints with shorter context windows, which means less prefill work and faster inference. Gemini 1.5 Flash costs $0.30 per 1M output tokens and is the second-fastest model in our table. The 7B Mistral is even cheaper and faster. If your task doesn't require GPT-4-class reasoning, the latency savings from going smaller can be 3-5x. That often outweighs any quality dip for things like extraction, classification, summarization, and routing decisions in agent systems.

Key Insights Worth Taking With You

The single most useful thing we can say is: measure on your own traffic, not the leaderboard. Our numbers above are useful as a rough orientation, but your actual prompts will differ in length, structure, and complexity, and that changes everything. A provider that scores well on short prompts may collapse on long-context requests because prefill cost dominates. A provider that has a great TTFT might have mediocre streaming consistency, which is what humans actually feel when they read generated text.

Second insight: TTFT and ITL are different optimization targets. If your UI shows a "thinking..." spinner for the duration of prefill, TTFT is everything. If your UI streams tokens into a typewriter animation, ITL is everything. Pick your optimization target before picking your model. Third insight: don't trust a single number. Always look at p50, p95, and p99. A provider with a great median and bad p99 (3-4x the median) is hiding rate-limit throttles behind a polite 429, which is the worst kind of latency. Fourth insight: geography is destiny. The cheapest, fastest model in someone else's benchmark may be the slowest one for you, purely because of the fiber path between your server and their inference cluster. Multi-region failover is worth implementing even at the prototype stage if latency matters to your users.

Fifth and final: the API layer is part of the model now. A unified gateway that lets you swap providers behind a single base URL and key is genuinely valuable for benchmarking, because it removes TLS handshake variance, key-rotation overhead, and SDK-version drift from your measurement. It also lets you do live traffic shifting in production once you figure out which provider actually wins for your workload.

Where to Get Started

If you want to skip the setup work and just hit a consistent endpoint across all the major model providers — OpenAI, Anthropic, Google, Mistral, Meta, Deepseek, the long tail — there's a clean unified option worth trying. Global API gives you one API key, access to 184+ models, normalizes the request shape so the snippet above works without changes, and bills through PayPal so you don't have to wrangle five different vendor agreements. It's the easiest way to populate your own mini leaderboard without spending a week wiring up separate SDKs, and the free tier is generous enough to run a full benchmark sweep. Drop the base URL into the script, swap the model string, and start collecting real numbers on your own traffic this afternoon.