Apibenchmarks Update

If you've ever shipped an AI product and watched your users bounce because the first response took 4.7 seconds, you already know: latency isn't a vanity metric. It's the difference between a product that feels magical and one that feels broken. In 2025, with model providers multiplying like rabbits and routing layers becoming their own cottage industry, the question isn't really "which model is best?" anymore. It's "which provider gets me a token fastest, and how do I prove it?"

That's exactly what we obsess about here at Apibenchmarks. We don't rank models by vibes or by leaderboard screenshots. We hit endpoints, we measure wall-clock response times, and we tell you what we found. This post is a snapshot of where things stand in late 2025: where the latency wins are, where the footguns are, and what a reasonable engineer should be doing about it.

Why Benchmark Speed Latency Is the Only Metric That Pays Rent

A quick sanity check before we get into numbers. There's a category of benchmarks — MMLU, HumanEval, GPQA — that measure capability. They tell you how smart a model is on a static test. Useful, sure. But none of them tell you what happens when a real user types "hey" into a chatbox at 9:14 AM on a Tuesday and waits.

Speed latency does. And specifically, three numbers matter most in production:

Time to First Token (TTFT): the delay between sending the request and getting the first byte of the answer. Anything above ~800 ms feels sluggish for chat. The good providers live in the 180–400 ms range on warm connections. Anything below 150 ms is genuinely impressive.

Inter-Token Latency (ITL): how fast tokens stream out after the first one. For chat, 25–60 ms per token is the sweet spot. Above 80 ms and the answer feels like it's being typed by a nervous accountant. Below 20 ms is suspiciously fast or a sign you're getting cached tokens.

End-to-End Latency (E2E): total time until the model finishes. For a 200-token response, you're aiming for under 2.5 seconds. For a 1,000-token response, under 8 seconds. Past that, you're losing people.

The reason this matters financially is brutal and obvious. Every 100 ms of added TTFT costs roughly 1–2% of conversion on interactive products, based on a pile of A/B tests we've seen from various SaaS teams over the past two years. If you're spending $40,000/month on inference and your median latency is 600 ms higher than it could be, you're burning real money on nothing.

The State of API Latency, Late 2025 — A Data Table

We ran a battery of tests across 14 model endpoints that are reachable through global-apis.com/v1 on November 14, 2025, from a VPS in Frankfurt. We sent each request 50 times, discarded the highest and lowest values, and averaged the rest. We used a 512-token output budget and a 256-token prompt. No streaming on the TTFT measurements; streaming for the ITL numbers. Here's the rough shape of what came back.

A few observations before we move on. First, the geography jump matters more than people think. Routing a request from Frankfurt to a US-east inference cluster can add 120–180 ms on TTFT before the model even starts thinking. That's not a tuning problem on the model's side; that's the speed of light in fiber and the queuing theory of TLS handshakes. If your users are in Berlin and you're hitting a Virginia endpoint, you've already lost.

Second, the compact tier is the dark horse of the table. A 7B-class model with a 85 ms TTFT and 14 ms ITL is genuinely useful for autocomplete-style features, code suggestions, and short classification jobs. People forget it exists because it can't write a sonnet, but for what it does, it's the lowest-latency thing on the menu.

Third, look at the reasoning-heavy tier. Median TTFT of 1,100 ms is a big jump, and it isn't because the model is slow at the first layer. It's because the provider is doing speculative chain-of-thought generation and only streaming once the answer is "settled." You can either accept that delay (and get higher quality), or you can do speculative streaming yourself and hope the draft is right. We tested both; the draft-then-verify approach saves about 400 ms on average at the cost of about 6% factual regressions on hard reasoning tasks.

How We Actually Measure (Because Methodology Is Everything)

Any benchmark is only as honest as its setup, and we've seen plenty of marketing benchmarks that test from the provider's own datacenter — which is basically measuring cache warmth rather than real user experience. So here's what we do, and you can reproduce it.

Our test rig is dead simple. Five geographically diverse probes: one in Frankfurt, one in Virginia, one in Singapore, one in São Paulo, and one in Sydney. Each runs a small Go program that does the following in a loop: wait a random jitter (0.5–3 seconds), construct a 256-token prompt drawn from a fixed bank of 200 prompts, hit the endpoint with a streaming request, record the wall time, record the time of the first byte, record the time of every token, and store it locally. Every 24 hours the probes upload their results to a central bucket.

We don't run one of those tests; we run thousands. As of writing this, the database has about 14 million data points spanning 38 model configurations over the last 90 days. That's not because we're showing off. It's because latency is the kind of number that hides bad days. A provider can have a great median and still have a miserable P99. We need enough samples to make the tail honest.

One thing we deliberately don't do: we don't share our raw prompt bank. The reason is that some providers (we won't name names) clearly optimize for benchmark prompts in ways they don't optimize for real users. If we published the exact strings, they'd game them. So the bank stays private, and we periodically rotate it.

If you want to do your own measurements — and you should — the basic shape of the code is short enough to fit in a single file. Here's a Python version using the streaming endpoint:

Code Example: Measuring TTFT Yourself With One API Key

import time
import os
import requests

# Single base_url — covers 184+ models behind one key.
BASE_URL = "https://global-apis.com/v1"
API_KEY  = os.environ["GLOBAL_APIS_KEY"]

def measure_ttft_itl(prompt: str, max_tokens: int = 512, runs: int = 10):
    samples = []
    for i in range(runs):
        t_start  = time.perf_counter()
        t_first  = None
        token_ts = []

        with requests.post(
            f"{BASE_URL}/chat/completions",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={
                "model": "auto",            # let the router pick
                "messages": [{"role": "user", "content": prompt}],
                "max_tokens": max_tokens,
                "stream": True,
            },
            stream=True,
            timeout=60,
        ) as r:
            for line in r.iter_lines():
                if not line or line == b"data: [DONE]":
                    continue
                now = time.perf_counter()
                if t_first is None:
                    t_first = now
                token_ts.append(now)

        ttft_ms = (t_first - t_start) * 1000
        itl_ms  = (
            (token_ts[-1] - token_ts[0]) / max(1, len(token_ts) - 1) * 1000
        )
        samples.append((ttft_ms, itl_ms))

    ttfts = sorted(s[0] for s in samples)
    itls  = sorted(s[1] for s in samples)
    # trim 10% on each side for a robust estimate
    lo, hi = len(samples) // 10, -len(samples) // 10
    return {
        "ttft_median_ms": ttfts[len(ttts) // 2],
        "ttft_p95_ms":    ttfts[int(len(ttfts) * 0.95)],
        "itl_median_ms":  itls[len(itls) // 2],
        "e2e_median_s":   (ttfts[len(ttfts) // 2] / 1000
                           + itls[len(itls) // 2] / 1000 * max_tokens),
    }

if __name__ == "__main__":
    prompt = "Explain in three short paragraphs why cold starts hurt TTFT."
    print(measure_ttft_itl(prompt))

Run that against three or four different model values and you'll have a tiny but real benchmark in under an hour. Swap in JavaScript or Go if that's your stack — the shape is identical. The one detail people forget is the timeout=60; some providers will hold a connection open for 30+ seconds on a cold start, and you don't want that polluting your TTFT measurement by hanging the connection.

Key Insights: What's Actually Driving The Numbers

After staring at millions of data points, here are the patterns we'd bet money on.

Insight 1: Warmth is destiny. The single biggest swing in TTFT for any given endpoint is whether the model is already in GPU memory on the specific shard you hit. A "cold" first request after a quiet minute can be 3–10x slower than the same request 30 seconds later. Providers know this; many pre-warm popular models on shared infrastructure so the cold path is rarer for high-traffic users. If you're a small customer, you will disproportionately hit cold paths. The trick is to keep an idle keepalive probe running if you care about first-request latency, or accept the tax.

Insight 2: Routing is half the latency. Look at the table again — Route A, B, and C are the same frontier model, just served from different edges. The TTFT spread is 210 to 340 ms, all geography. If you have a global user base, picking a single provider with a single region is leaving money on the table. The providers that offer automatic geo-routing routinely beat their region-pinned competitors by 100–200 ms for users outside the home region.

Insight 3: Speculative decoding is no longer optional at the high end. Two years ago it was an experimental setting. Today it's table stakes. The reason: the gap between frontier models and well-tuned drafts has narrowed, and the verifier step has gotten much cheaper. The result is a 25–40% ITL reduction on long outputs with negligible quality hit. If your provider doesn't offer some flavor of this and a competitor does at the same price, switch.

Insight 4: Long-context is a latency trap. The 200k-context row in the table looks fine — 460 ms TTFT — but it's misleading. The P95 there is 920 ms, and the E2E is 8.4 seconds for a relatively short output. The reason: long-context models pay an attention cost on the prompt side that scales quadratically without FlashAttention-class optimizations. If you're stuffing 50k tokens into the prompt "just in case," you're paying for it on every single request. Trim your context. Retrieval-augment instead. We saw one team cut their P95 from 14 seconds to 6 seconds purely by reducing average prompt length from 18k to 4k tokens.

Insight 5: Streaming or no-streaming is more than a UX choice. Non-streaming endpoints consistently show TTFT values that are 30–80% higher than the streaming-mode equivalent of the same model, because the provider batches the entire generation before responding. For chat, you should virtually always stream. For batch jobs (summarization, extraction pipelines at 3 AM), non-streaming is fine and sometimes a bit cheaper because you can amortize the prefill.

Insight 6: Reasoning tiers exist for a reason, but don't use them blindly. A 1,100 ms median TTFT is not a bug; it's the cost of thinking before speaking. The trap is using the reasoning tier for short factual queries where it adds delay without adding accuracy. We tested the same 200 factual Q&A prompts on the reasoning tier and the standard tier — accuracy moved from 71% to 78%, but median latency went from 280 ms to 1,100 ms. Whether that trade is worth it depends entirely on what the user is asking. A router that picks the tier per request can capture the upside without the constant cost.

How Often Do These Numbers Actually Change?

More often than you'd think. Providers ship capacity changes weekly. New regions come online, old ones get re-tuned, and occasionally a provider silently rolls out a new inference stack that shifts everything by 50–100 ms. That's part of why a continuous measurement system matters. A one-shot benchmark from three months ago is mostly a historical artifact by now. We re-measure the entire grid on a rolling basis; the data on the site is rarely more than seven days old for any given endpoint.

We also watch for step-changes. When a provider rolls out a major version of their inference engine, you can usually see it as a vertical cliff in the latency chart — either downward (good) or upward (concerning). The last big cliff on the charts was in early October 2025, when two of the major providers switched to a new speculative-decoding configuration that improved ITL by roughly 35% across their entire fleets. Good news for everyone.

The Honest Limits