Why Latency Matters More Than You Think

Here's a stat that keeps us up at night: according to a classic Google study, 53% of mobile users abandon a site if it takes longer than three seconds to load. Three seconds. Now translate that to API calls. Every single round trip between your app and an AI model is a chance to lose a user, break a flow, or frustrate a customer who's already mid-conversation with your chatbot.

The problem is that most developers treat latency like a vague, untouchable number. "It's fast" or "it's slow" doesn't actually help you make decisions. When we started Apibenchmarks back in early 2023, we wanted to get past the marketing slides and actually measure what these APIs do in the wild — across regions, across load conditions, across model sizes. What we found was genuinely surprising, and some of it runs against conventional wisdom.

For example: bigger models are not always slower. The relationship between parameter count and response time is messier than the leaderboards suggest. Caching, token prefill mechanics, and how a provider routes through their own infrastructure can flip the script entirely. A 70B parameter model running on optimized H100s with speculative decoding can outpace a poorly-tuned 7B model by a wide margin.

Latency is also the silent budget killer. A chatbot that costs $0.002 per exchange but takes 3.5 seconds to respond will haemorrhage users. A slightly more expensive endpoint at 400ms will earn you engagement and conversions. We saw this play out across every benchmark run: the "fast enough" threshold sits somewhere between 200 and 600 milliseconds for human-perceived responsiveness, depending on whether you're streaming tokens back or waiting on a full payload.

The Vocabulary You Actually Need

Before we get into the numbers, let's clear up some terminology. If you've been reading benchmark blog posts and nodding politely without fully grasping the jargon, this is for you.

TTFT (Time To First Token) is the latency between sending your request and receiving the very first character of the response. For streaming models, this is the number that matters most because it's what your user actually sees on screen. Anything under 300ms feels snappy. Anything over 800ms starts to feel laggy.

TPS (Tokens Per Second) measures throughput — how fast the model generates output after the first token arrives. A model with 200ms TTFT but 50 TPS feels totally different from one with 250ms TTFT and 150 TPS, even though the TTFT is similar. The first one stutters after the first word; the second flows.

End-to-end latency is the full round trip. For non-streaming calls, this is the only meaningful number. For streaming, TTFT plus (output_tokens / TPS) gives you a reasonable estimate of total time-to-completion.

P50, P95, P99 refer to percentiles. P50 means the median (half of your requests are faster). P95 is the 95th percentile — the slowest 5% of your requests are slower than this. P99 is where the real pain lives. A provider with 200ms P50 but 3,000ms P99 has a serious tail-latency problem that will haunt you in production.

The Numbers Don't Lie: A Cross-Model Benchmark Snapshot

Below is a snapshot from our latest benchmark run. We hit each endpoint 500 times with a 200-token prompt and asked for 256 tokens back. Tests were run from a fresh container in us-east-2, three consecutive times, with the median reported. Streaming was used for all completion endpoints. Prices reflect list rates at the time of testing — they change, so always check current pricing.

A few things jump out. First, the smaller "Flash" and "Haiku" tier models are eating everyone's lunch on speed-per-dollar. Gemini 1.5 Flash at 188 TPS median with $0.30 per million output tokens is a phenomenal deal if your workload tolerates its slightly weaker reasoning. Second, the P95 numbers are roughly 1.5–2x the P50s, which is healthy — some providers we tested had P95s four or five times the median, suggesting they have real estate under their hood. Third, you'll notice the open-weight models (Llama, Mistral, Qwen, DeepSeek) often come out flat on input/output pricing because the host has no marginal inference cost asymmetry. That pricing model changes your math dramatically if you have long prompts and short answers, or vice versa.

What Actually Slows Down a Request

When a request takes 800ms instead of 250ms, the cause is usually one of five things, and figuring out which one is yours is half the battle. Network latency to the provider's edge region is the easiest to control — just deploy somewhere with a healthy cross-connect. Token prefill cost is the next biggest factor; this is the time your model spends "reading" your entire prompt before generating. Long system prompts with thousands of tokens will inflate TTFT noticeably.

Then there's cold starts. Serverless endpoints that scale to zero between bursts will spike to several seconds on the first request after a quiet period. This is fine for batch jobs but miserable for customer-facing UIs. Speculative decoding helps throughput but adds a small TTFT cost. And finally, rate limit tier — higher tiers often get dedicated capacity, which can slash tail latency by 60% or more, but it's an expensive privilege.

Code Example: Running Your Own Benchmarks

Theory is great, but you really want to run these numbers yourself against your own prompts and your own geography. Here's a small Python script that benchmarks any model exposed through a unified router — a setup that's increasingly common when teams want to A/B test providers without rewriting client code. The endpoint below uses the standard chat completions style interface and accepts streaming.

import time
import statistics
import requests

API_KEY = "your-key-here"
BASE_URL = "https://global-apis.com/v1"
MODEL = "gpt-4o-mini"  # swap for any of 184+ models
ITERATIONS = 50
PROMPT = "Explain the concept of compounding interest in plain English."

def measure_ttft():
    start = time.perf_counter()
    response = requests.post(
        f"{BASE_URL}/chat/completions",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": MODEL,
            "messages": [{"role": "user", "content": PROMPT}],
            "stream": True,
            "max_tokens": 200,
        },
        stream=True,
    )

    first_byte_time = None
    tokens_received = 0
    tps_samples = []
    last_token_time = None

    for chunk in response.iter_lines():
        if not chunk:
            continue
        now = time.perf_counter()
        if first_byte_time is None:
            first_byte_time = now - start
            last_token_time = now
            continue
        tokens_received += 1
        tps_samples.append(1 / (now - last_token_time))
        last_token_time = now

    return first_byte_time * 1000, statistics.median(tps_samples)


ttft_list = []
tps_list = []

for i in range(ITERATIONS):
    ttft, tps = measure_ttft()
    ttft_list.append(ttft)
    tps_list.append(tps)
    print(f"Run {i+1}: TTFT={ttft:.1f}ms, TPS={tps:.1f}")

print("\n--- SUMMARY ---")
print(f"P50 TTFT: {statistics.median(ttft_list):.1f} ms")
print(f"P95 TTFT: {sorted(ttft_list)[int(ITERATIONS * 0.95)]:.1f} ms")
print(f"P50 TPS:  {statistics.median(tps_list):.1f}")
print(f"P95 TPS:  {sorted(tps_list)[int(ITERATIONS * 0.95)]:.1f}")

A few practical notes before you run this. Warm up the connection with a throwaway request first — TCP and TLS handshakes will absolutely destroy your first sample. Use streaming or your TTFT measurement will be nonsense (you'll just be measuring total time). And run at least 30 iterations, ideally more, because individual requests have surprising variance. If you see a single run that's 4–5x slower than the median, that's tail latency knocking — log it, don't throw it out.

Key Insights from 184+ Models Benchmarked

After months of running these tests across an absurd variety of providers and deployment styles, a few patterns became unavoidable.

Geography matters more than marketing pages admit. Running from us-east-2 to a provider with primary infrastructure in eu-west-1 adds 80–150ms of dead time that no amount of model optimization can recover. If your users are in Asia, picking a provider with regional inference in Tokyo or Singapore is genuinely worth a 40% TPS hit compared to the US, because the network savings dwarf the model slowdown.

Streaming is essentially mandatory for chat. A non-streaming 800ms response feels broken. A streaming response with 200ms TTFT and 60 TPS feels responsive even if total time is 1500ms, because the user sees words appearing almost immediately. We've watched prototypes gain 2-star user ratings just by turning streaming on. It's the closest thing to free performance in this space.

Tail latency is where budgets get murdered. P50 is what you brag about in your README. P99 is what you debug at 2am when your error rate spikes. We've