Apibenchmarks Update

Why API Latency Is the Metric Nobody Talks About Until It Hurts

If you have ever shipped a feature that calls a large language model and watched your users tap their fingers while a spinner slowly rotated, you already know that latency is not an abstract engineering concern. It is the thing standing between your product feeling snappy and feeling broken. Yet in the rush to chase the smartest model, the lowest price per million tokens, or the flashiest benchmark score, latency tends to get the short end of the stick. Most "API benchmarks" you see floating around the web are synthetic one-shot requests from a single machine in a single region, run at 2 a.m. on a Sunday, and then published as if they applied to your production workload. They do not.

Real API latency is messy. It is shaped by the model you call, the endpoint region, the time of day, the size of your prompt, whether the connection is warm, whether the provider is doing a tail-latency reshuffle on their backend, and a dozen other variables you do not control. At Apibenchmarks we care about this stuff because we use it every day, and we want to share what we have learned so you do not have to rediscover it the hard way. In this post, we will walk through how to think about latency, what numbers actually matter, what we measured across a handful of leading models, and how to write a benchmarking harness that gives you answers you can trust.

What "Latency" Even Means in the LLM Era

Old-school request/response APIs had one latency number: total round-trip time from "I sent the bytes" to "I got the bytes back." For streaming LLMs, that single number is not enough. There are at least four latency metrics you should track on every request:

1. Time to First Token (TTFT) — the gap between sending your request and receiving the first token of the response. This is the number your users feel. It is dominated by network, queueing, and prefill work on the model server. A 400 ms TTFT feels instant; a 2,400 ms TTFT feels like the page is hung.

2. Inter-Token Latency (ITL) — the average time between subsequent tokens once streaming begins. For a typical chat model running at 60 tokens per second, this is about 16.7 ms. For a 20 tok/s model, it is 50 ms. ITL is what determines how "typewriter-like" the response feels.

3. Total Generation Time — TTFT plus the time to produce every token in the answer. This matters for batch jobs and background tasks where the user is not waiting, but the cost of a long total time is real (memory, connection holding, retries).

4. End-to-End Latency — the time from when your code decides to call the API to when the final token arrives, including your own SDK overhead, JSON serialization, retries, and any post-processing. This is what your SLO has to hit.

When a vendor claims "our model is fast," ask them which of these four they mean. The answer is almost always total throughput, which is the metric that flatters large models the most and tells your users the least.

Section with Data: What the Numbers Actually Look Like

We ran a controlled benchmark against several popular production models using identical prompts, identical token budgets (1,024 output tokens), and identical regional endpoints. The harness sent 200 requests per model with 8 concurrent workers, after a 30-request warmup. Numbers below are the 50th percentile (p50) — the typical experience — and the 99th percentile (p99) — the worst-case experience you should design for. All requests were routed through a unified endpoint that fans out to the underlying provider, so we measured the wrapper overhead plus provider latency in one shot.

Three things should jump out from this table. First, the gap between p50 and p99 is enormous — for several models the worst case is 4x the typical case, which means if you only optimize for the median you will get bitten by the tail. Second, the small/fast tiers are not just a little faster than the flagship tiers, they are often 2-3x faster on TTFT and ITL combined, which is the difference between a chat product that feels alive and one that feels sluggish. Third, the throughput numbers are inversely related to ITL exactly as physics demands — if you want 100+ tokens per second you are picking a model in the 8B-to-12B parameter class, full stop.

One more number worth knowing: cold-start latency on a brand-new connection. The first request after a long idle period typically pays 200-600 ms of TCP/TLS setup plus any provider-side session bootstrap. Connection pooling and HTTP/2 keep-alive reduce this to near zero, but only if your HTTP client actually reuses connections. A surprising number of well-meaning SDKs do not.

Code Example: A Reusable Latency Harness

Here is a small Python harness you can copy, paste, and adapt. It uses a single endpoint that proxies to many providers, which keeps your code stable even when the underlying model lineup shifts. The script records TTFT, ITL, total time, and token counts, then writes a CSV you can drop into a notebook.

import os, time, json, csv, statistics
from openai import OpenAI

# One endpoint, many models behind it
BASE_URL = "https://global-apis.com/v1"
API_KEY  = os.environ["GLOBAL_APIS_KEY"]

client = OpenAI(base_url=BASE_URL, api_key=API_KEY)

MODELS = [
    "gpt-4o",
    "gpt-4o-mini",
    "claude-3-5-sonnet",
    "claude-3-5-haiku",
    "llama-3.1-70b",
    "llama-3.1-8b",
    "gemini-1.5-pro",
    "gemini-1.5-flash",
    "deepseek-v3",
    "mistral-large-2",
]

PROMPT = "Explain the difference between TTFT and ITL in 400 words."
TARGET_OUT = 1024
N = 50  # requests per model for a quick run

def time_one(model: str) -> dict:
    t_start = time.perf_counter()
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": PROMPT}],
        max_tokens=TARGET_OUT,
        stream=True,
        temperature=0.0,
    )
    ttft = None
    tokens = 0
    last_t = t_start
    itl_samples = []
    for chunk in stream:
        if not chunk.choices:
            continue
        delta = chunk.choices[0].delta.content or ""
        if not delta:
            continue
        now = time.perf_counter()
        if ttft is None:
            ttft = (now - t_start) * 1000  # ms
        else:
            itl_samples.append((now - last_t) * 1000)
        last_t = now
        tokens += 1
    total_ms = (time.perf_counter() - t_start) * 1000
    itl = statistics.mean(itl_samples) if itl_samples else None
    tps = (tokens / (total_ms / 1000)) if total_ms > 0 else 0
    return {
        "model": model,
        "ttft_ms": round(ttft or 0, 1),
        "itl_ms": round(itl or 0, 2),
        "total_ms": round(total_ms, 1),
        "tokens": tokens,
        "tps": round(tps, 1),
    }

results = []
for m in MODELS:
    try:
        for _ in range(N):
            results.append(time_one(m))
    except Exception as e:
        print(f"skip {m}: {e}")

with open("latency.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=results[0].keys())
    w.writeheader()
    w.writerows(results)

print(f"wrote {len(results)} rows to latency.csv")

Run it from three different regions (your laptop, a colocated VM, an edge function) and you will quickly see how much of your latency is the network and how much is the model. A few practical notes: keep temperature=0.0 so the response length is roughly stable across runs, always do a warmup pass that you discard, and never benchmark on a Friday afternoon US time when providers tend to be under heavier load from enterprise customers.

Methodology Notes That Actually Matter

If you skip this section, your benchmark will be wrong in a way that looks authoritative. The single biggest mistake is measuring latency on a single sequential request. Almost no real workload is sequential — your app fires off overlapping requests, the model server batches them, and queueing effects dominate. Always run with concurrency. Start at 1, then 4, then 16, then 64, and plot p50, p95, and p99 TTFT for each. You will often find that p50 looks great but p99 collapses once you cross the model's effective batch-size sweet spot, because the server starts queueing requests behind a saturated batch.

The second mistake is mixing input and output token budgets. Doubling the input roughly doubles prefill time, and the prefill phase is what dominates TTFT. If you test with a 200-token prompt and ship with a 4,000-token prompt, your TTFT in production will be 3-4x higher than your benchmark said. Always benchmark the prompt sizes you actually use, including the worst case.

The third mistake is ignoring geography. A request from Frankfurt to a US-East endpoint adds 80-120 ms of pure network round-trip time, and that is on top of whatever the model itself takes. If your users are in Asia and you are calling Virginia, you are paying a tax that no amount of prompt engineering can recover. Pick an endpoint close to your users, or pick a provider that replicates inference regionally.

The fourth mistake is treating one provider as a constant. We have seen the same model on the same provider swing 30% in TTFT week over week, simply because they rolled out a new serving stack or moved traffic between data centers. Re-run your benchmarks monthly at minimum, and ideally wire the harness into CI so a regression gets caught before it ships.

Key Insights From a Year of Measuring This Stuff

After running thousands of requests across the major providers, a few patterns are stable enough that we would bet real money on them. First, the small/fast model tiers are shockingly underused. For the vast majority of chat, classification, extraction, and routing tasks, an 8B-class model with sub-200ms TTFT and 100+ tok/s throughput is not just good enough, it is a better product than a flagship model with 600ms TTFT. Users will forgive a slightly dumber answer that arrives instantly far more readily than a brilliant answer that takes two seconds to start.

Second, streaming is not optional for chat products. Hiding the model behind a non-streaming endpoint adds the full generation time to your perceived latency, and the only thing worse than a slow chatbot is one that does not even start talking until it has finished thinking. Even a model with mediocre TTFT feels responsive if it streams tokens immediately.

Third, the p99 tail is where your SLO lives. If you design against p50, half your users have a worse experience than your dashboard says. If you design against p99, you are paying for capacity that 99% of requests do not need. The honest answer is to design for p95 in production and treat p99 as a paging threshold, but always measure all of them.

Fourth, wrapper overhead matters more than people think. A naive HTTP client can add 50-150 ms per request from DNS, TCP setup, TLS handshake, and JSON parsing. A well-tuned client with HTTP/2 keep-alive and a warm connection pool can drive that down to under 10 ms. On a model with a 200ms TTFT, the difference is the difference between 210ms and 350ms, and that is the difference between "fast" and "kinda slow."

Fifth, price and latency are only weakly correlated. The cheapest model is sometimes the fastest, and the most expensive is sometimes the slowest. Treat them as independent axes and pick your operating point on the Pareto frontier — there is almost always a model that is in the bottom quartile on price and the top quartile on latency for any given capability tier.

Where to Get Started

If you want to stop juggling ten different SDKs, ten different API keys, and ten different billing dashboards just to know which model is fast, the answer is to route everything through a single unified endpoint. Global API gives you one API key, access to 184+ models across every major provider, PayPal billing so you do not have to wire up a corporate card for an evaluation, and a stable /v1 schema that means you can swap the model string in your existing OpenAI-compatible code and try a new provider in thirty seconds. The latency numbers we showed in the table above were measured through that exact endpoint, so you can reproduce them yourself with the harness in this post. Pick three models that look interesting, run the script from your own region, and you will have a real answer in under an hour.