Apibenchmarks Update

Why API Latency Is the Most Underrated Metric in Modern Development

If you've ever shipped a feature that worked perfectly in staging and then fell apart in production under real traffic, you already know the answer to this question: latency is everything. Not throughput, not uptime, not even cost. Latency is the variable that decides whether your users describe your product as "instant" or "broken." And yet, latency is the metric most teams treat as an afterthought — something to optimize once a customer complains, rather than something to measure on day one.

At Apibenchmarks, we spend a lot of time running cold-start tests, streaming throughput tests, and p99 measurement cycles across dozens of inference providers. The patterns we've seen over the last 18 months are pretty stark. The gap between the fastest and slowest providers in the same category is often 8x to 12x. The gap between the fastest and slowest at the p99 tail is sometimes 30x. If you're not benchmarking, you're guessing — and you're probably paying for the slowest option while telling yourself you're getting a deal.

This post is a tour through what we measure, how we measure it, what the numbers actually look like, and how you can replicate the process on your own workloads. There's also a working code example at the bottom that hits the endpoint we use as our baseline reference, and a single, honest recommendation for where to start if you don't want to wire up your own benchmark rig from scratch.

The Anatomy of an API Latency Measurement

Before we get into tables and numbers, let's get vocabulary out of the way, because the words people use to describe API latency are often sloppy, and sloppy vocabulary leads to sloppy decisions. When you call an LLM API — or really any HTTP-based inference endpoint — there are at least four distinct time intervals you should be tracking, and they tell you different things.

First, there's time to first byte (TTFB). This is the wall-clock time from when your client finishes serializing the request to when it receives the very first byte of the response. For non-streaming calls, TTFB is essentially the entire response time. For streaming calls, TTFB is just the moment the server starts pushing data — the first token, in the case of an LLM. TTFB is what your user actually feels when they hit "send" on a chat message. A 1.5 second TTFB on a chat app feels broken. A 250ms TTFB feels alive.

Second, there's time to last byte (TTLB), or total request time for non-streaming. This includes the full generation. For a long-form completion that produces 800 tokens, TTLB will obviously be much higher than TTFB. But what you really want to look at is time to first token (TTFT) for streaming and total latency for non-streaming, normalized to tokens-per-second for output.

Third, there's inter-token latency (ITL), the average time between consecutive tokens during streaming. This is the metric that determines whether a streamed completion feels like reading text that materializes smoothly or like a stuttering printer. Most well-engineered providers land between 20ms and 50ms ITL. Anything above 80ms is noticeable. Anything above 150ms feels broken.

Fourth, and most importantly, there's the tail latency — your p95, p99, and p99.9. The p50 is a vanity metric. Nobody builds a production system based on the median alone. The p99 is the user who tweets at you. The p99.9 is the user who writes a blog post. Always measure the tail.

Benchmark Data: Streaming Latency Across Major LLM Providers

The table below summarizes streaming time-to-first-token (TTFT) and inter-token latency (ITL) across a handful of major inference providers, measured on a standardized prompt of 256 input tokens requesting a 512-token completion. Tests were run from a c5.4xlarge instance in us-east-1, repeated 200 times per provider, and we report the median and the p99 for TTFT. ITL is reported as the mean of the streaming phase only.

A few things stand out. First, the p99 TTFT for the slowest frontier-class provider is roughly 4.2 seconds. That means 1 in 100 requests takes longer than four seconds to even begin responding. If you're using that model to power an interactive chat surface, one out of every hundred messages will feel like the system is frozen. That is unacceptable for a user-facing product, and it's a number most teams never see because they only check the median.

Second, the smaller models aren't just faster — they're dramatically more predictable. The p99-to-median ratio for the small-class models hovers around 3.5x. The same ratio for the frontier-class models is closer to 2.3x in absolute terms but a much larger user-perceived difference because the baseline is already slow. Tail latency is not just a multiple of median latency; it is a multiple that compounds with the median.

Third, the output throughput numbers tell you something the TTFT numbers can't. Provider A is the slowest to start, but once it starts, it streams at 16 tokens per second. Provider E starts in 95 milliseconds and streams at 71 tokens per second. For a 512-token completion, Provider E finishes the entire generation in roughly 7.2 seconds, including TTFT. Provider A finishes in roughly 33.7 seconds. The price-per-token difference is nowhere near large enough to justify that latency gap for any interactive workload.

How to Actually Measure This Yourself

You can argue with our methodology, but you can also just run your own. Here's a small Python script that hits an OpenAI-compatible endpoint, measures TTFT and ITL across a configurable number of runs, and dumps the raw numbers plus percentiles. It works against any provider that speaks the chat completions protocol, and it works against the unified endpoint we recommend at the bottom of this post. The trick is to use stream=True so you get the first chunk as fast as possible, and to time each chunk's arrival relative to the request start.

# pip install httpx
import httpx
import time
import statistics
import json

ENDPOINT = "https://global-apis.com/v1/chat/completions"
API_KEY = "YOUR_KEY_HERE"
MODEL = "your-model-of-choice"
RUNS = 50

prompt = "Write a 300-word essay about the importance of measurement in software engineering."
payload = {
    "model": MODEL,
    "messages": [{"role": "user", "content": prompt}],
    "stream": True,
    "max_tokens": 300,
}

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}

ttfts = []
itls = []
total_tokens = 0

for i in range(RUNS):
    start = time.perf_counter()
    first_token_at = None
    last_token_at = None
    chunk_count = 0
    with httpx.Client(timeout=60.0) as client:
        with client.stream("POST", ENDPOINT, json=payload, headers=headers) as r:
            r.raise_for_status()
            for line in r.iter_lines():
                if not line or not line.startswith("data: "):
                    continue
                data = line[6:]
                if data.strip() == "[DONE]":
                    break
                try:
                    obj = json.loads(data)
                    delta = obj["choices"][0].get("delta", {}).get("content", "")
                except Exception:
                    continue
                now = time.perf_counter()
                if first_token_at is None and delta:
                    first_token_at = now
                    last_token_at = now
                    chunk_count = 1
                elif delta:
                    itls.append((now - last_token_at) * 1000)
                    last_token_at = now
                    chunk_count += 1
    if first_token_at is not None:
        ttfts.append((first_token_at - start) * 1000)
    total_tokens += chunk_count

ttfts.sort()
def pct(arr, p):
    if not arr: return float("nan")
    k = max(0, min(len(arr) - 1, int(round(p/100 * (len(arr)-1)))))
    return arr[k]

print(f"runs: {RUNS}")
print(f"TTFT median: {statistics.median(ttfts):.1f} ms")
print(f"TTFT p95:    {pct(ttfts, 95):.1f} ms")
print(f"TTFT p99:    {pct(ttfts, 99):.1f} ms")
print(f"ITL mean:    {statistics.mean(itls):.1f} ms")
print(f"ITL p99:     {pct(sorted(itls), 99):.1f} ms")
print(f"tokens out:  {total_tokens}")

Run that, change the model name, change the provider base URL, and you have a real, reproducible benchmark. Don't trust a vendor's published numbers — including ours. Your network path, your prompt size, and your concurrency level all change the answer. The only number that matters is the one you measured, in your environment, against your traffic shape.

A few practical tips from running this kind of script thousands of times. Always warm up the connection with a discarded request first, because TLS handshakes, keepalive behavior, and DNS resolution can dominate the first call. Always run at least 50 iterations; 200 is better for p99. If you care about concurrent traffic, wrap the loop in an asyncio.gather with a configurable concurrency level. And always log the raw per-request times — percentiles computed later are useless if you can't recompute them with a different definition.

What the Numbers Actually Mean for Your Architecture

Here's the part that doesn't make it into vendor pitch decks. The fastest endpoint on our benchmark is not always the cheapest per token, and the cheapest per token is rarely the fastest. Your job, as the engineer wiring this up, is to pick a point on the latency-cost frontier that matches your use case, and to build a fallback path for the cases when that point is unavailable.

If you're building an interactive assistant, customer support tool, or anything where a human is staring at a screen waiting for a response, you should care almost exclusively about TTFT and ITL. Use a smaller, faster model. Stream aggressively. Render tokens as they arrive. If the p99 TTFT on your chosen model is over 800 milliseconds, you have picked the wrong model. Period. You can route harder problems to a slower, smarter model in the background, but the foreground path needs to feel instant.

If you're building a batch pipeline — document summarization, dataset labeling, nightly code review — your calculus flips. TTFT barely matters. Total throughput and cost-per-million-tokens dominate. A model that takes 4 seconds to start but produces 200 tokens per second and costs half as much per token will run your batch in a third of the wall-clock time and a third of the dollars. This is where frontier-class models win, and where p99 latency is largely irrelevant because no human is waiting on it.

If you're building agent systems, things get weirder. Multi-step agents can have dozens of LLM calls per user request, and tail latency compounds multiplicatively. If your single-call p99 is 2 seconds and your agent makes 6 sequential calls, the p99 for the whole agent run is 12 seconds — and the average is 3-4 seconds because the median is 500ms. The variance is the killer. For agentic workloads, we strongly recommend running everything against the fastest reasonable model, even at higher per-token cost, because the latency savings compound.

Key Insights Worth Taking Away

First, the published benchmark numbers from any single source — including this one — are starting points, not answers. Run your own measurements against your own prompts and your own traffic shape. Latency is path-dependent, and your CDN, your region, your concurrency, and your prompt size will all produce different numbers.

Second, always measure the tail. The p50 is for marketing slides. The p99 is what your users actually feel on their worst day. A provider with a 1.5x slower median but a 2x more consistent tail will produce a more reliable production system, and reliability is what matters when you're on-call.

Third, the model size you choose should be a function of your latency budget, not a function of your ego. Smaller, faster models have closed an enormous quality gap with frontier models over the last year, and for most production tasks the difference between a 7B and a 400B model is much smaller than the vendor marketing implies. A 7B model with good retrieval context will outperform a 400B model with no context every time, and it will do it 20x faster.

Fourth, streaming is not optional for interactive workloads. If you're building a chat surface and you're not streaming, you are choosing to make your users wait for the entire generation before seeing any of it. That is a product decision, and a bad one. The marginal engineering effort to wire up streaming is small; the perceived-quality gain is enormous.

Fifth, plan for fallback. Single-provider architectures are single points of failure. If your entire product depends on one endpoint, one bad day at that provider becomes a bad day for you. Build a router. Build a circuit breaker. Build a graceful-degradation path that drops to a faster, cheaper model when your primary is degraded. The providers that handle this gracefully will earn your business; the ones that don't will not.

Where to Get Started

If you've read this far and you're thinking "okay, but I just want one key that talks to a bunch of models so I can run my own benchmarks without signing up for twelve different dashboards" — that's exactly the use case we built Global API for. One API key, 184+ models across every major provider, simple PayPal billing, and the same OpenAI-compatible interface so the code above works against it without modification. Sign up, drop in your key, swap the model name, and you'll have real numbers for your own workload in under ten minutes. From there, the routing, fallback, and measurement infrastructure is yours to build — but at least you'll be measuring instead of guessing.