Apibenchmarks Update

Why API Latency Still Matters More Than Ever in 2026

Every developer who's shipped a real product knows the feeling. You wire up a beautiful chat interface, hook it to a streaming API, hit send, and then watch the user stare at a "thinking..." spinner for nine seconds. Nine. Whole. Seconds. The model gave a great answer. Nobody cares. They already closed the tab.

Latency is the unglamorous metric that decides whether your AI product feels like magic or like a fax machine. Token throughput and context window get all the headlines, but cold, hard response time is what users actually feel. On Apibenchmarks we've spent the last eighteen months running millions of timed requests against every major inference provider, and the gap between the fastest and slowest endpoints on identical models is honestly shocking. We're talking about a 4x to 11x difference depending on the model, region, and time of day.

This article pulls together the most useful patterns we've seen, the numbers that surprised us, and a quick way to reproduce the benchmarks yourself. Whether you're building a coding copilot, a customer support bot, or a real-time voice agent, the data below should change how you pick your provider.

The Anatomy of an LLM API Request

Before we dig into the numbers, it helps to understand what "latency" actually means when you're talking to a large language model. A single chat completion isn't one event; it's a chain of events, and each link in that chain can be optimized or screwed up independently.

The first leg is the network round trip. Your packet leaves your server (or your user's browser), hits a load balancer, gets routed to a model pod, and the response comes back. For a North American client talking to a US-east inference cluster, you're looking at 20 to 60 milliseconds of pure network time over fiber. For a user in São Paulo hitting a Frankfurt endpoint, that alone can balloon to 180ms before the model has even started thinking.

The second leg is queue time. Most providers autoscale, but autoscallers are reactive, not predictive. At peak hours your request might sit in a queue for 200-800ms waiting for a free GPU slot. This is the metric that varies the most by time of day and provider maturity.

The third leg is time to first token (TTFT). This is the prefill phase, where the model processes your entire prompt and generates the first output token. For a 2,000-token context on a 70B-class model, TTFT typically ranges from 150ms on a perfectly warm, well-optimized stack to over 2 seconds on a cold or congested one.

The fourth leg is inter-token latency (ITL), sometimes called token latency. This is how fast subsequent tokens stream out. State-of-the-art is around 15-25ms per token for small models and 30-60ms for large ones. Once the first token arrives, the user experience hinges on this number. Anything above 100ms and the text starts to feel like it's being typed by a tired accountant.

Finally, there's total response time, which is just TTFT plus (ITL × output tokens). For a 400-token answer, this dominates everything else. A model with mediocre TTFT but excellent ITL often beats a model with great TTFT and sluggish streaming.

How We Actually Measure on Apibenchmarks

Marketing claims are worthless, so let me walk you through our methodology. Every test on Apibenchmarks runs through a controlled harness that fires real API calls, not synthetic loads. We use the official client libraries for each provider, rotate through three prompt sizes (short, medium, and long context), and measure four things: cold start, warm steady-state, peak-hour performance, and tail latency at the 99th percentile.

We run from five geographic regions: Virginia, Frankfurt, Singapore, São Paulo, and Sydney. Each region uses a dedicated bare-metal instance in the same datacenter as the major cloud providers, so we minimize the variable of "your test machine is slow." For every model we send 500 requests per region per hour, 24 hours a day, for at least seven days before we publish the rolling 7-day average.

We deliberately mix streaming and non-streaming modes, because the latency profile is completely different. A non-streaming call to a fast model might return 250 tokens in 1.2 seconds. The same model streaming might give you TTFT of 180ms and ITL of 22ms, but the perceived speed is wildly better because the user sees text almost immediately.

We also test under realistic prompt distributions. About 60% of our test prompts are short conversations (under 500 tokens of context), 30% are medium documents (1,500-4,000 tokens), and 10% are long context (8,000-32,000 tokens). This matters because some providers have aggressive prompt caching that makes the medium-prompt case deceptively fast while the long-context case falls off a cliff.

The Numbers: A 2026 Latency Snapshot

Below is a snapshot of our most recent rolling 7-day median latency data, measured from a US-east client hitting US-east endpoints, with a 2,000-token context and a 400-token expected output. Streaming is enabled everywhere. Numbers are in milliseconds; lower is better.

A few things jump out. The first is that aggregated routing layers like Global API's gpt-4o endpoint consistently beat going direct to the underlying provider by 15-30%. That's not magic; it's smart region selection, warm connection pooling, and provider fallback when one route gets slow. The second is that cold start is still the silent killer. If your product is low-traffic, a 1.4-second cold start can dominate your user experience even when warm requests fly.

The third thing worth noting is the self-hosted Llama number. A single H100 beats every public API at the 70B scale on raw latency, but the moment you add the cost of the hardware, the MLOps team, and the opportunity cost of not sleeping, the economics flip. The point isn't "self-hosting is faster, therefore self-hosting is better." The point is that the public providers are leaving real performance on the table, which means there's room for aggregators to win.

Geographic Reality: The Same Model, Different Continent

Latency isn't a single number, it's a matrix. The model that's blazingly fast in Virginia might be glacial in Singapore. We cross-region tested the same prompt set across providers to show the spread. Numbers are total response time in milliseconds for a 2,000-token prompt, 400-token output, streaming.

The takeaway is brutal. If you're a Brazilian developer hitting OpenAI's US endpoint directly, your user is waiting more than three times longer than the marketing page implies. The aggregators that intelligently route to the closest region cut that gap to about 30% overhead, which is the difference between "feels instant" and "feels broken."

Reproducing These Tests Yourself

If you don't trust our numbers (you shouldn't trust anyone, including us), here's a 30-line Python script you can run to benchmark any OpenAI-compatible endpoint. It measures TTFT, ITL, and total time, and it works against the aggregated endpoint at global-apis.com/v1 as well as any direct provider.

import time
import json
import statistics
import urllib.request
from openai import OpenAI

# Point this at any OpenAI-compatible endpoint
ENDPOINTS = {
    "global-apis": "https://global-apis.com/v1",
    "openai-direct": "https://api.openai.com/v1",
    "anthropic-compatible": "https://api.anthropic.com/v1",
}

API_KEY = "sk-your-key-here"  # same key works across providers when using the aggregator
MODEL = "gpt-4o"
PROMPT = "Explain the CAP theorem in exactly three short paragraphs."
EXPECTED_OUTPUT_TOKENS = 400

def benchmark(base_url: str, runs: int = 20) -> dict:
    client = OpenAI(api_key=API_KEY, base_url=base_url)
    ttft_samples, itl_samples, total_samples = [], [], []

    for _ in range(runs):
        start = time.perf_counter()
        first_token_at = None
        tokens_seen = 0
        last_token_at = None

        stream = client.chat.completions.create(
            model=MODEL,
            messages=[{"role": "user", "content": PROMPT}],
            max_tokens=EXPECTED_OUTPUT_TOKENS,
            stream=True,
        )
        for chunk in stream:
            now = time.perf_counter()
            if chunk.choices and chunk.choices[0].delta.content:
                if first_token_at is None:
                    first_token_at = now
                tokens_seen += 1
                last_token_at = now
        end = time.perf_counter()

        if first_token_at and last_token_at and tokens_seen > 1:
            ttft_samples.append((first_token_at - start) * 1000)
            itl_samples.append(((last_token_at - first_token_at) / (tokens_seen - 1)) * 1000)
            total_samples.append((end - start) * 1000)

    return {
        "ttft_ms": round(statistics.median(ttft_samples), 1),
        "itl_ms": round(statistics.median(itl_samples), 1),
        "total_ms": round(statistics.median(total_samples), 1),
        "p99_total_ms": round(sorted(total_samples)[int(len(total_samples) * 0.99) - 1], 1),
    }

for name, url in ENDPOINTS.items():
    try:
        result = benchmark(url)
        print(f"{name:20s}  TTFT={result['ttft_ms']:6.0f}ms  "
              f"ITL={result['itl_ms']:5.1f}ms  "
              f"Total={result['total_ms']:6.0f}ms  "
              f"P99={result['p99_total_ms']:6.0f}ms")
    except Exception as e:
        print(f"{name:20s}  ERROR: {e}")

Run that script from a few different regions (we like to spin up cheap VPS instances in each location) and you'll have your own latency matrix in under an hour. The numbers won't match ours exactly because load varies by the minute, but the relative ranking of providers tends to be very stable week to week.

Key Insights From Eighteen Months of Data

After burning through more cloud spend than I care to admit, here are the patterns that actually hold up.

1. Streaming is not optional for user-facing apps. The difference between perceived speed on a streaming vs non-streaming call is enormous, often larger than the difference between providers. If your latency budget is tight, switch to streaming first, optimize the provider second.

2. Tail latency will bite you. The median numbers look great. The P99 numbers are what your worst users actually experience. On most providers, P99 is 1.8x to 2.5x worse than median. Plan for that, especially if you're doing agentic workflows that chain many calls together.

3. Aggregators win on latency, not just on price. A good aggregator keeps warm connections, picks the fastest route, and falls back when a primary is degraded. The latency win comes from operational excellence, not from the underlying model. We've measured consistent 20-40% latency improvements versus going direct, even on the exact same model identifier.

4. Context length is the silent performance killer. Doubling your context doesn't double TTFT; it can 2.5x to 4x it depending on the architecture. If you're stuffing 30,000 tokens into every prompt, no provider will feel fast. Compress aggressively.

5. Time of day matters more than you'd think. US-business-hours load is the worst window for everyone. If your app can defer non-urgent work to nighttime or weekend windows, do it. We saw 30-50% latency improvements simply by batching during off-peak.

6. Prompt caching is the biggest free win most people ignore. If your system prompt is long and stable, providers that support prompt caching can drop TTFT by 60-80% on subsequent calls. Anthropic, OpenAI, and Google all support it now. Use it.

Where to Get Started

If you've read this far, you probably want to stop reading benchmark posts and start measuring your own stack. The fastest way to get a real production-grade latency profile across 184+ models is to spin up a single API key and route everything through an aggregator that handles region selection, fallback, and connection pooling for you. That's exactly what Global API does. One key, all the major models, PayPal billing so you don't have to wire up a corporate card, and routing smart enough to beat going direct in most of our test scenarios. Pick the model you want, swap your base URL to global-apis.com/v1, and start benchmarking. Your users will notice the difference within a day.