Apibenchmarks Update

Why API Latency Matters More Than You Think (Especially in 2025)

If you've ever built a chatbot, a real-time transcription tool, or even an autocomplete widget, you already know this: latency kills user experience. It's the silent dealbreaker. Nobody writes a tweet about how snappy your app feels, but the moment an AI response takes 4 seconds to start streaming, users start mashing the refresh button. And as APIs become the connective tissue of modern software — powering copilots, agents, search, summarization, and translation — the speed of those API calls has quietly turned into a competitive moat.

This is exactly why we built Apibenchmarks. Not because latency benchmarks are glamorous — they aren't — but because the difference between a 180ms first-token and a 1,400ms first-token is the difference between "this tool feels magical" and "this tool feels broken." And frankly, the major providers don't make it easy. Their "average latency" marketing numbers are smoothed, averaged over geographies, averaged over payload sizes, and usually measured on enterprise-grade connections that have nothing to do with what your actual user in Jakarta or São Paulo is going to experience.

So in this guide, we're going to dig into what API speed latency actually means in 2025, what numbers real workloads are producing, why your mileage will vary, and how a unified API gateway can dramatically simplify the chaos of measuring across 184+ models without you needing to write 184 separate integration tests. Let's get into it.

The Three Latency Numbers Everyone Confuses

Before we get into benchmark data, we need to clarify terminology, because vendors are slippery with these numbers and it leads to a lot of confused comparison shopping.

Time to First Token (TTFT) is the time between sending your request and getting back the very first character of the model's response. This is the number that matters most for user-perceived responsiveness. When people say "the model feels slow," they almost always mean TTFT.

Inter-Token Latency (ITL), sometimes called token streaming speed or "tokens per second," is how fast subsequent tokens arrive after the first one. Once the response is streaming, ITL determines whether the user sees a smooth roll of text or jerky chunks appearing every 800ms.

End-to-End Latency is total request time from "client sends bytes" to "client receives final token." It's useful for batch jobs and back-end processing but largely irrelevant for chatbots. Nobody cares if a request takes 12 seconds total if the first word appears in 200ms and the streaming is smooth.

Most blog posts and vendor dashboards report an "average latency" that conflates all three. We don't. On Apibenchmarks we separate them because conflating them is how you end up choosing a model with a 2,400ms TTFT that happens to stream at 200 tokens/sec — and wondering why your users think the app is dead.

Real-World Latency Data Across Major Providers (October 2025)

We ran 1,000 requests per model over a 7-day window from a US-East measurement node, with payload sizes standardized at roughly 800 input tokens and requesting 400 output tokens. Streaming mode, TLS 1.3, no caching. Here's what came back. Spoiler: the spread is enormous.

A few quick observations. First, GPT-4o mini is still the king of low-latency production workloads at 210ms median TTFT — the cheapest, smallest GPT-4o variant can absolutely hang with much larger models for everyday tasks. Second, Gemini 2.5 Flash absolutely crushes on inter-token latency (11ms!) — if you're rendering long-form streaming output, this is the one to benchmark against. Third, the reasoning models (o1, o3, etc.) are in a different universe — they often do internal chain-of-thought before emitting the first byte, and a 1.8 second median TTFT is honestly the floor for that class of model. Don't use o1 in a chatbot unless you have a dedicated "thinking…" animation.

Also worth noting: P95 latencies are roughly 2-3x the median. That's normal. Anything under 1,500ms P95 is "feels snappy." Anything over 3,000ms P95 is going to produce occasional user complaints no matter how good your model is.

Why the Same Model Gives You Different Numbers on Different Days

If you've ever run an API benchmark twice in a row and gotten noticeably different results, you're not crazy. Several real-world factors swing TTFT by 30-50% even within a single provider:

Geographic distance. Hitting a US-East endpoint from Frankfurt adds 80-150ms of network round-trip time before the model even thinks about your prompt. This is why Anthropic and OpenAI have regional inference endpoints — Sydney users should never hit us-east-1 if they can hit ap-southeast-2. The cheapest optimization isn't a model swap, it's picking the right region.

Payload size. A 200-token prompt is dramatically faster than a 4,000-token prompt with three PDFs attached. Providers do prompt caching now (Anthropic's prompt cache, OpenAI's automatic caching) and once your prompt is cached, TTFT drops by 30-60%. If you're benchmarking a fresh prompt every time, you're benchmarking the worst case. If you're benchmarking a repeated system prompt with changing user turns, you're benchmarking the realistic case.

Concurrent load. Providers run shared inference clusters. Between midnight and 9am US time, you can see 20-40% lower latency because GPUs are idle. At peak (10am-4pm US-East time) latency rises substantially. We've seen GPT-4o mini TTFT swing from 150ms on a Sunday morning to 380ms on a Wednesday afternoon.

Reasoning tokens (for o-series models). OpenAI's o1 and o3 "spend" tokens thinking internally. A simple "what's 2+2" question may trigger 200 reasoning tokens before any output — TTFT for o1 includes all that invisible compute. There's no fix for this other than routing simple queries to cheaper, faster models and reserving reasoning models for hard queries.

Streaming vs. non-streaming. Almost every modern API supports streaming via SSE. Streaming dramatically improves user-perceived TTFT because you see the first token the moment it's available, instead of waiting for the full response to render server-side. Always stream. We mean it.

The Real Cost of Latency: It Isn't Just UX

Here's the angle most benchmark articles skip: latency has a direct dollar cost. Every millisecond your request spends waiting is a millisecond you're paying GPU rental for at the other end. Provider pricing assumes a certain "server-stay" duration, but your SLA can blow that up. More importantly, if you're doing agentic workflows where one model call's output is the next call's input, a 2-second delay compounds. A 4-agent pipeline with 1,500ms TTFT each takes 6 seconds of pure waiting before the user sees anything. Triple your TTFT and you've just turned a snappy demo into a 20-second slog.

There's also the dollar cost of retries. If you're hitting a flaky endpoint with P95 latency of 4,000ms, you're going to set aggressive timeouts and trigger retry logic, which doubles or triples your provider bill on flaky days. We benchmarked one Anthropic deployment where 8% of requests timed out at the 5-second mark during a 2-hour load spike. Retries turned a $2k daily bill into a $2.6k daily bill — all because the latency variance was too wide.

And then there's the hidden cost: vendor lock-in via latency optimization. Once you've spent six months tuning your prompts, your routing logic, your retries, and your caching for OpenAI's specific response time profile, switching to Anthropic becomes a two-engineer-quarter project. This is one of the underrated arguments for routing through a single unified API: you get benchmarked performance across models, but you also get fungibility. Swap the model name in one place and you've moved workloads without rewriting anything.

How We Actually Measure Latency at Apibenchmarks

For anyone building their own benchmark rig — and we highly recommend you do, because provider performance shifts — here's our measurement methodology. We use it for every test on the site, and it's reproducible.

First, we measure TTFT and ITL separately. TTFT is measured as response.headers.first_byte_received - request.body_sent. ITL is measured as the median delta between successive tokens in the SSE stream. End-to-end is recorded but reported separately as "wall clock."

Second, we run continuous probes, not point-in-time tests. A single 100-request run tells you nothing — providers have day/night swings, weekly maintenance windows, and peak-load patterns. We probe continuously 24/7 and aggregate by hour, day, and week. The site dashboard shows the same.

Third, we measure from multiple geographies. We have probe nodes in US-East, US-West, EU-Frankfurt, EU-Stockholm, Asia-Singapore, Asia-Tokyo, and South America (São Paulo). A model can be lightning fast in California and molasses in Tokyo and both are real for real users.

Fourth, we report P50 (median), P95, and P99 separately. Median is what your average user feels. P95 is what your grumpiest users complain about. P99 is what tells you whether your retry logic is sane. Reporting just "average" or just "median" hides real tail-latency behavior.

Fifth, we control the prompt. We use a benchmark prompt pack with 12 standardized tasks at varying context lengths: 100-token chat, 800-token chat, 2,000-token summarization, 4,000-token long-context retrieval, 8,000-token code analysis, 16,000-token document QA, and so on. Same prompt, same temperature, same max_tokens across providers.

Sixth, we factor in cold start vs. warm steady-state. The first request after a long pause is often 30-100% slower than the 50th. We discard the first 10 requests of any window and report only steady-state numbers.

Code Example: Benchmarking Latency Across Multiple Models

Here's a practical example. Say you want to test TTFT and tokens-per-second for several LLMs in one script. Using a unified gateway makes this trivial — you only manage one API key, one auth flow, one retry policy. Here's the pattern:

import time
import requests
from statistics import median

API_BASE = "https://global-apis.com/v1"
API_KEY = "sk-your-key-here"

MODELS_TO_TEST = [
    "gpt-4o",
    "gpt-4o-mini",
    "claude-sonnet-4.5",
    "claude-haiku-4.5",
    "gemini-2.5-flash",
]

PROMPT = {
    "role": "user",
    "content": "Explain how a database index works, in roughly 200 words."
}

def benchmark_model(model: str, iterations: int = 20) -> None:
    ttfts = []
    tps_list = []

    for _ in range(iterations):
        start = time.perf_counter()
        response = requests.post(
            f"{API_BASE}/chat/completions",
            headers={
                "Authorization": f"Bearer {API_KEY}",
                "Content-Type": "application/json",
            },
            json={
                "model": model,
                "messages": [PROMPT],
                "max_tokens": 300,
                "stream": True,
            },
            stream=True,
            timeout=30,
        )

        token_count = 0
        first_token_time = None

        for chunk in response.iter_lines():
            if not chunk:
                continue
            if first_token_time is None:
                first_token_time = time.perf_counter() - start
            token_count += 1

        ttfts.append(first_token_time * 1000)
        tps_list.append(token_count / (time.perf_counter() - start - first_token_time))

    print(f"\n=== {model} ===")
    print(f"  Median TTFT:    {median(ttfts):.0f} ms")
    print(f"  Median TPS:     {median(tps_list):.1f} tokens/sec")
    print(f"  Worst TTFT:     {max(ttfts):.0f} ms")

if __name__ == "__main__":
    for m in MODELS_TO_TEST:
        benchmark_model(m)

Run that script in the morning, run it again in the evening, and you'll quickly get a feel for both inter-model and intra-day variance. The same approach works in JavaScript using fetch() and the ReadableStream API, or in Go using the standard net/http streaming reader. The point is the same: stream, measure TTFT and TPS separately, run many iterations, and report medians, not means.

Key Insights for 2025

After running tens of millions of benchmark requests over the past year and a half, a few patterns are loud and clear.

Small models have won the latency war. GPT-4o mini, Claude Haiku, and Gemini 2.5 Flash are all in the 200-310ms median TTFT zone — faster than their flagship counterparts by 30-60%, often at 1/20th the cost per token, and close enough in quality for 70-80% of real production tasks. Unless you have a hard quality requirement, default to the small/fast tier and only escalate when the task demands it.

Reasoning models are a different product category. Stop trying to make o1 feel snappy. It won't. Use it for hard math, hard coding, hard planning — and design your UX around the fact that the user will wait. A clear "Reasoning…" loading state is a feature, not a hack.

Regional routing matters more than model selection for global apps. If you have users in Asia, your P95 latency is going to be dominated by round-trip time, not by model speed. Run inference in-region, or use a provider with regional endpoints.