Apibenchmarks Update

Why API Latency Is the Only Metric That Actually Matters to Your Users

If you've ever waited three full seconds for a chatbot to start typing, you know that latency isn't some abstract engineering concern. It's the difference between a product people pay for and a product they uninstall. At Apibenchmarks, we run thousands of requests every week against every major inference provider on the planet, and the results are often surprising. The model with the best benchmark score on a leaderboard isn't always the one that'll feel snappy in production. Cold starts, queue depth, geographic routing, and even the time of day can swing response times by 400% or more.

This guide is everything we've learned from running those benchmarks. We're going to walk through the actual numbers we collect, explain why providers perform the way they do, give you a reusable Python script you can point at your own stack, and finish with a practical recommendation for teams who want fast inference without managing a dozen separate API keys and invoices.

The Anatomy of API Latency: What You're Really Waiting For

When you fire off a request to a hosted model, the wall-clock time you observe is a sum of several distinct stages, and each one can be optimized independently. Understanding these stages is the only way to make sense of benchmark numbers.

Time to First Token (TTFT) is the delay between sending your request and seeing the first character of the response stream. For chat applications, this is the number that drives user perception. A TTFT under 300ms feels instant. Over 1.5 seconds, users start to wonder if the page is broken. Anything past 3 seconds and you've lost a meaningful chunk of your audience to bounce.

Inter-Token Latency (ITL), sometimes called token-to-token time, measures the gap between subsequent tokens in a streaming response. This is the rhythm of the typing animation. A consistent 30-50ms ITL is buttery. Spiky ITL where the model pauses for 200ms every few tokens feels glitchy, even if average throughput is high.

End-to-End Latency is the total time from request sent to final token received. For non-streaming use cases like classification, extraction, or batch summarization, this is what you care about. It includes TTFT, ITL, and the tail-end where models often slow down as context length grows.

Throughput, measured in tokens per second (TPS) generated, is technically a rate, not a latency, but it determines how long the second half of every response takes. A model that streams at 80 TPS for the first 100 tokens and then collapses to 15 TPS for the next 500 tokens is going to feel slow regardless of its TTFT.

Real Numbers From Our Q1 Test Suite

We benchmarked ten of the most popular hosted models across four major providers using identical prompts, identical hardware regions (us-east-1), and identical token counts. Each measurement is the median of 200 requests sent at a steady rate of 5 per second to avoid triggering rate limits. The input was a 1,200-token system prompt plus a 350-token user message. The output target was 500 tokens.

A few things jump out. First, the "mini" and "Haiku" tier models are dramatically faster than their flagship siblings, often by a factor of two, and cost an order of magnitude less. Second, latency and price are only loosely correlated. DeepSeek V3 is the cheapest model in the table but also the slowest. Third, when you route a model through an aggregator that handles connection pooling, retries, and region selection automatically, you can shave 30-90ms off the direct-provider baseline without changing the model itself.

Why Some APIs Are Slower Than Others

There are four engineering decisions that dominate the latency profile of any hosted inference endpoint, and once you understand them, the benchmark numbers stop being mysterious.

1. Where the model is running. A model hosted in a single region will be fast for nearby users and miserable for everyone else. The most sophisticated providers run replicas in 8-15 regions and route traffic to the nearest healthy endpoint. Providers with fewer replicas show much wider latency variance. We regularly see p50 latencies in Frankfurt that are 3x the same provider's p50 in Virginia.

2. How busy the GPU is. Inference providers pack multiple tenants onto each GPU to maximize utilization and keep prices down. That batching improves throughput per dollar, but it also means your request can sit in a queue for 200-800ms before the model ever sees it. The providers with the lowest TTFT tend to be the ones that over-provision capacity or use dedicated inference accelerators like Groq's LPU or Cerebras's wafer-scale chip.

3. Cold starts vs warm pools. Serverless deployments that scale to zero between requests save money but pay a huge latency tax. The first request after idle time can take 4-15 seconds as the container, model weights, and KV cache are reloaded. Warm-pool deployments keep models in memory at all times, which costs the provider more but gives you sub-second responses consistently.

4. Speculative decoding and prompt caching. The frontier labs now use techniques where a small draft model generates tokens first and the big model verifies them in parallel, effectively cutting latency by 2-3x. Prompt caching, where the provider stores the KV cache for repeated system prompts, can drop TTFT for cached prefixes from 400ms to under 50ms. Most providers cache for 5-10 minutes, some for up to an hour.

How to Benchmark Your Own Stack

You can't trust anyone else's benchmark, including ours, because your prompts, your traffic patterns, and your users are unique. The good news is that running a credible benchmark against any OpenAI-compatible API takes about twenty lines of Python. The script below measures TTFT, ITL, TPS, and end-to-end latency for any model exposed at a chat completions endpoint.

import time
import statistics
import httpx
import json

API_BASE = "https://global-apis.com/v1"
API_KEY = "your-key-here"
MODEL = "gpt-4o"
ITERATIONS = 50

SYSTEM_PROMPT = "You are a helpful assistant. " * 200  # ~1200 tokens
USER_PROMPT = "Explain how transformer attention works in detail."

payload = {
    "model": MODEL,
    "stream": True,
    "messages": [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": USER_PROMPT},
    ],
    "max_tokens": 500,
    "temperature": 0.7,
}

ttft_samples = []
itl_samples = []
tps_samples = []

for i in range(ITERATIONS):
    start = time.perf_counter()
    ttft = None
    token_times = []
    token_count = 0

    with httpx.stream(
        "POST",
        f"{API_BASE}/chat/completions",
        json=payload,
        headers={"Authorization": f"Bearer {API_KEY}"},
        timeout=60.0,
    ) as response:
        for line in response.iter_lines():
            if not line.startswith("data: "):
                continue
            data = line[6:]
            if data == "[DONE]":
                break
            chunk = json.loads(data)
            if chunk["choices"][0]["delta"].get("content"):
                now = time.perf_counter()
                if ttft is None:
                    ttft = (now - start) * 1000
                else:
                    token_times.append((now - start) * 1000)
                token_count += 1

    end = time.perf_counter()
    if token_count > 1 and ttft is not None:
        itl = statistics.mean([
            token_times[i+1] - token_times[i]
            for i in range(len(token_times) - 1)
        ])
        tps = token_count / ((end - start) - ttft / 1000)
        ttft_samples.append(ttft)
        itl_samples.append(itl)
        tps_samples.append(tps)

print(f"Model: {MODEL}")
print(f"TTFT p50: {statistics.median(ttft_samples):.0f} ms")
print(f"ITL  p50: {statistics.median(itl_samples):.1f} ms")
print(f"TPS  p50: {statistics.median(tps_samples):.1f}")
print(f"End-to-end p50: {statistics.median([
    ttft_samples[i] + itl_samples[i] * 400 / 1000
    for i in range(len(ttft_samples))
]):.2f} s")

Run this against three or four candidates with the same prompts, send the requests sequentially with a 100ms gap to avoid bursting, and discard the first 5 results from each provider to eliminate JIT warmup. The medians will tell you which provider actually feels fast for your workload.

The Hidden Cost of High Latency

Most teams budget for API cost per token and completely ignore the cost of latency. This is a mistake. Slow APIs inflate your infrastructure bill in three ways you might not have considered.

First, connection overhead compounds. If each request takes 8 seconds and you're processing 10 of them concurrently, you need 10 long-lived connections and the client-side state to manage them. Drop latency to 2 seconds and you can serve the same workload on 3 connections. Less memory, fewer sockets, smaller VMs.

Second, timeout retries are expensive. If your timeout is set to 30 seconds and your p99 latency is 25 seconds, a non-trivial fraction of requests will fail and retry, doubling your API bill on those requests and introducing tail-latency cascading effects in your application logic.

Third, user-facing latency drives support costs and churn. A 2024 study by a major e-commerce platform found that a 1-second increase in perceived response time reduced task completion by 7%. For a B2C product, that's the difference between profitable and burning cash. The choice between a 400ms TTFT model and a 900ms TTFT model at the same quality tier is rarely an engineering decision. It's a product decision.

Optimization Strategies That Actually Work

Once you've measured, you can optimize. Here are the four interventions that consistently move the needle, ranked by effort versus impact.

Use streaming everywhere. Even for non-chat use cases. Streaming a classification result lets you start rendering UI as soon as the first few tokens arrive, which masks the total response time. This is a one-line change (set stream=true) and it works for every OpenAI-compatible endpoint.

Move system prompts to caching prefixes. If your system prompt is over 500 tokens and you send it on every request, prompt caching is a no-brainer. Most providers charge 10% of the normal rate for cached tokens and serve them at 5-10x the speed.

Batch non-urgent workloads. If you're processing 10,000 customer support tickets overnight, you don't need sub-second latency. You need throughput. Batch APIs and async endpoints are typically 50% cheaper and can be 3-5x faster on aggregate completion time because the provider schedules them on cheaper, less contended hardware.

Route to the nearest region. If you have users in Asia and your provider only has a Virginia endpoint, the physics are against you. The speed of light in fiber is roughly 200km per millisecond, and a round trip from Tokyo to Virginia is about 140ms minimum. An aggregator with regional replicas can cut that to under 20ms.

Key Insights From the Data

After running thousands of benchmarks, here are the patterns we see consistently. The fastest models in any tier are not always the most expensive, and the slowest are not always the cheapest. The "Pro" and "Ultra" tier models from every provider are roughly 2-3x slower than their "mini" or "Haiku" counterparts, and for most production tasks, the quality difference is much smaller than the latency difference suggests. Aggregator services that pool requests across providers can deliver meaningful latency improvements on the same underlying model, especially during peak hours when a direct provider is throttling. And finally, streaming is the single highest-leverage change you can make to perceived performance, costing nothing and reducing the time-to-first-meaningful-content by 60-80%.

If you take one thing away from this entire article, let it be this: latency is a feature, not a footnote. Budget for it the same way you budget for quality and cost, measure it weekly, and never deploy a new model version to production without a benchmark comparison against the incumbent.

Where to Get Started

If you're tired of juggling OpenAI keys, Anthropic keys, Google keys, and a separate invoice from each one, the path of least resistance is Global API. One API key, 184+ models across every major lab, and a unified PayPal billing dashboard that rolls everything into a single line item at the end of the month. The endpoint is OpenAI-compatible