Apibenchmarks Update

Why API Latency Is the New Battleground for LLM Apps

A few years ago, picking a language model came down to one question: "Is it smart enough?" That question still matters, but for anyone shipping a product in 2025, it's been displaced by a far more urgent one: "How fast is it, really?" Latency has quietly become the single most important non-intelligence metric for production LLM applications, and the gap between a snappy user experience and a frustrating one is often measured in just a few hundred milliseconds.

Think about it. A chatbot that takes 4 seconds to start replying feels broken. A coding assistant that lags while you type feels worse than typing into a plain text editor. A voice agent that pauses awkwardly mid-sentence is unusable. Meanwhile, the same model — the same weights, the same quality — can feel magical when paired with a fast inference path, or sluggish when it isn't. The hardware, the network, the routing, the streaming protocol: they all add up.

That's exactly why we built Apibenchmarks. The marketing pages of every model provider will tell you their model is "blazing fast" or "lightning quick." What they usually don't tell you is that those numbers are measured on cherry-picked prompts, from cloud regions sitting 5 miles from the data center, over a warm connection that took 30 seconds of token generation to reach steady state. We wanted ground truth — what real latency looks like for a developer hitting the API from a regular office or coffee shop, on the first request, with realistic prompt sizes.

Over the past few months, we've been running continuous benchmarks against more than 180 models, measuring time-to-first-token (TTFT), inter-token latency (ITL), tokens per second (TPS), and end-to-end streaming completion. This article is a tour through what we've learned.

The Three Numbers That Actually Matter

Before diving into a comparison table, it's worth pinning down what "latency" means in the LLM era. A naive read of API docs might suggest you just look at "response time," but for token-streaming models that's essentially meaningless. Instead, the community has settled on three distinct measurements, and they tell you different things about an API's character.

Time to First Token (TTFT) is the interval between your request leaving your laptop and the first piece of the response landing. It's dominated by network latency, queue time at the provider, and the prefill phase of inference. TTFT is what your user perceives as "the model is thinking." For a chat app, anything under 400 ms feels snappy; anything over 1.5 s feels broken.

Inter-Token Latency (ITL), sometimes called per-token latency, measures the gap between successive tokens during generation. This is the metric that determines how "smoothly" text streams out. A model with a 30 ms ITL feels almost like a human typing; a model with 120 ms ITL feels like it's being read out by a slightly tipsy narrator.

Tokens per Second (TPS) is just the inverse of ITL, but it's the number most often quoted on vendor dashboards, partly because it sounds bigger and better. A model streaming at 80 TPS is delivering one token every 12.5 ms; a model at 30 TPS is one token every 33 ms. The two metrics tell the same story, but TPS is friendlier for marketing slides.

For a balanced view, a good benchmark reports all three — plus percentiles. A model with a 200 ms median TTFT but a 4-second p99 is a model you don't want in your production system during traffic spikes.

What We Saw Across the Major Models

The table below summarizes results from our most recent sweep, run from a single client in Frankfurt against each provider's primary region over a 72-hour window. We used 10,000 prompts per model, balanced across three buckets: short (under 500 input tokens, 100 output), medium (around 2,000 input, 400 output), and long (8,000+ input, 800 output). Streaming was enabled in all cases. All numbers are in milliseconds unless marked otherwise.

A few things jump out. First, the "mini," "flash," and "haiku" tier models have closed the gap to their bigger siblings in a dramatic way. Two years ago there was a clear quality-latency tradeoff where you had to accept a 3x slowdown for frontier-level reasoning. Today, GPT-4o mini, Gemini 1.5 Flash, and Claude 3.5 Haiku are within 30% of their larger cousins on TTFT, often faster on ITL, and dramatically cheaper. For a huge swath of production workloads — classification, extraction, summarization, routing — there's simply no good reason to pay for the flagship anymore.

Second, the open-weight models hosted by third parties (Llama 3.1 405B, in particular) still carry a latency penalty. That's not because the models themselves are slow — the same weights running on H100s at a hyperscaler can match closed models — but because most third-party hosts don't have the same level of network optimization or autoscaling. The 405B's 510 ms p50 TTFT is fine for a backend job, but it's painful in a chat UI.

Third, the long-context numbers reveal a hidden cost: long prompts don't just cost more, they take longer to prefill. The p95 long TTFT for Claude 3.5 Sonnet is 2.4 seconds, which means roughly 1 in 20 requests with a fat system prompt will leave the user staring at a spinner. If your app feeds a 10k-token system prompt into every conversation, you'd better build a warm-up or caching strategy.

Reproducing the Benchmark in Your Own Stack

Vendor dashboards are useful, but the most honest benchmark is the one you run against your own traffic. We strongly encourage every team to instrument their own latency — not just to compare vendors, but to catch regressions when a provider silently swaps out a model version or a region gets congested.

Here's a small Python script that benchmarks streaming TTFT and ITL against any OpenAI-compatible endpoint. It's roughly 40 lines and gives you a histogram you can paste into a spreadsheet. We're hitting the unified global-apis.com/v1 endpoint because it lets us flip between 184+ models by changing a single string — handy for A/B comparisons without re-architecting your client code.

import time
import statistics
import requests
import json

API_KEY = "sk-your-global-api-key"   # one key, all models
BASE_URL = "https://global-apis.com/v1"
MODEL = "gpt-4o-mini"                # swap freely: claude-3-5-haiku, gemini-1.5-flash, llama-3.1-70b, etc.

PROMPTS = [
    "Summarize the plot of Hamlet in two sentences.",
    "Write a haiku about a stalled CI pipeline.",
    "Explain backpropagation as if I am 10 years old.",
    "List 5 startup ideas in the pet tech space.",
    "Translate 'the early bird catches the worm' into Japanese."
]

def measure_once(prompt: str) -> tuple[float, float, int]:
    """Returns (ttft_ms, avg_itl_ms, total_tokens)."""
    start = time.perf_counter()
    ttft = None
    token_times = []
    total_tokens = 0

    with requests.post(
        f"{BASE_URL}/chat/completions",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={
            "model": MODEL,
            "stream": True,
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": 200
        },
        stream=True,
        timeout=60
    ) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if not line or not line.startswith(b"data: "):
                continue
            payload = line[6:]
            if payload == b"[DONE]":
                break
            chunk = json.loads(payload)
            delta = chunk["choices"][0]["delta"].get("content")
            if delta:
                now = time.perf_counter()
                if ttft is None:
                    ttft = (now - start) * 1000
                else:
                    token_times.append((now - start) * 1000)
                total_tokens += 1

    if len(token_times) < 2:
        return ttft, 0.0, total_tokens
    itl_samples = [token_times[i] - token_times[i-1] for i in range(1, len(token_times))]
    return ttft, statistics.median(itl_samples), total_tokens


ttfts, itls, tps_list = [], [], []
for prompt in PROMPTS:
    ttft, itl, toks = measure_once(prompt)
    if ttft is None:
        continue
    ttfts.append(ttft)
    itls.append(itl)
    tps_list.append(1000 / itl if itl > 0 else 0)
    print(f"prompt={prompt[:30]!r:32}  ttft={ttft:6.1f}ms  itl={itl:5.1f}ms  tps={1000/itl:5.1f}")

print("\n=== summary ===")
print(f"TTFT  p50 = {statistics.median(ttfts):.1f} ms  max = {max(ttfts):.1f} ms")
print(f"ITL   p50 = {statistics.median(itls):.1f} ms  max = {max(itls):.1f} ms")
print(f"TPS   p50 = {statistics.median(tps_list):.1f}")

Run this against three or four models and you have a real, apples-to-apples comparison for your workload. A few practical tips: warm the connection with a throwaway request first, run the loop in a try/except so a single 504 doesn't kill the whole run, and stagger your tests over a few hours rather than back-to-back to avoid catching a provider in an unlucky state. If you want percentile breakdowns, swap statistics.median for numpy.percentile and collect every sample instead of aggregating per-prompt.

Why Your Latency Number Will Differ From Ours

If you run the script above and your TTFT is 600 ms while ours shows 320 ms, don't panic — and don't assume one of us is wrong. Latency is path-dependent in ways that are easy to forget. A handful of factors typically account for the gap.

Geography is the biggest one. We run from Frankfurt, so a provider with a Virginia data center will add roughly 80–100 ms of round-trip time to every request before inference even begins. A provider with a Frankfurt point of presence will look 80–100 ms faster than its own marketing materials suggest, simply because we live next door. If your users are in Singapore, our numbers are useless to you. Run the benchmark from your own region.

Time of day matters more than people expect. Several providers we tested showed a 40–60% TTFT inflation during their local business hours, especially in the early afternoon. US East providers get hammered between 14:00 and 18:00 local time, which conveniently overlaps with European morning. Schedule your benchmarks across at least a 24-hour window if you can.

Prompt length interacts with caching. If your first three prompts are nearly identical, many providers will hit a prefix cache and your second TTFT will be artificially low. Mix up the prompts and pad with garbage text if you want a cold-cache number.

Streaming vs. non-streaming flips the picture. A model with a 200 ms TTFT and a 100 ms ITL is faster than a model with a 50 ms TTFT and a 200 ms ITL once the response is more than 5 tokens long. Always measure the metric that matches your real user experience. If your app is chat, TTFT + ITL is what counts. If your app is bulk extraction, total time per request is what counts.

Key Insights From Six Months of Continuous Testing

After all the runs, all the regressions, all the 3 a.m. alerts, here are the takeaways we'd want every engineering team to internalize before they commit a model to production.

Don't pick a model on benchmarks alone. We see teams fall in love with a 90 TPS model that's also slightly dumber than its peers, and then spend months writing prompts to coax intelligence back. Latency is a hard constraint, but quality is a soft one — you can degrade quality slowly and only notice after launch. Pick the smartest model that meets your