Apibenchmarks Update

Latency Is the Only Metric Your Users Actually Feel

If you've ever stared at a chat bubble that says "Generating..." for three full seconds while a customer waits on your checkout page, you already know why we built Apibenchmarks. Token counts are nice. Context windows are nice. But if the model takes forever to spit out the first word, none of that matters. Users don't read your marketing copy. They read the loading spinner.

We spent the last 90 days running a continuous latency suite against every major model API we could get our hands on, from the usual suspects (OpenAI, Anthropic, Google, Mistral) to the new crop of open-weight providers that route through Groq, Together, Fireworks, and a dozen other inference clouds. We measured first-token latency, inter-token latency, end-to-end latency, and tokens-per-second under realistic prompt loads. The results were surprising in places, predictable in others, and in a few cases, genuinely shocking.

This is what we found, how we measured it, and how you can reproduce our exact setup in under half an hour. Whether you're a solo developer shipping a weekend side project or a platform team picking a default model for an enterprise SaaS, these numbers should change how you think about "fast."

What "Speed" Actually Means When You Talk to an LLM

Before we get into the leaderboard, let's get the vocabulary right, because the industry is sloppy about this. There are really four distinct numbers that matter, and they measure very different things.

Time To First Token (TTFT) is how long you wait from hitting "send" to seeing the first character of the response appear. This is the one your users actually notice. If TTFT is over one second, the experience starts feeling sluggish. Under 300ms, it feels conversational. This number is dominated by prompt processing time, network round-trip, and queue depth at the provider.

Inter-Token Latency (ITL) is the gap between tokens once generation starts. Measured in milliseconds per token, it's the inverse of "tokens per second" for the streaming output phase. A model doing 100 tok/s has an ITL of 10ms. This is dominated by the speed of the inference engine and how the provider batches requests.

End-to-End Latency (E2E) is the total wall-clock time from request to fully completed response. For a 500-token answer, E2E ≈ TTFT + (500 × ITL). People often confuse this with throughput, but they're not the same. A model can have a low TTFT but a high E2E if it's slow per token.

Tokens Per Second (TPS) is what most benchmarks love to brag about, but it's only half the picture. A provider advertising 200 tok/s on a marketing page usually means peak throughput on a short prompt with no other tenants on the cluster. Real-world TPS on a Tuesday afternoon in your timezone, with a 4,000-token context window, is often half that.

At Apibenchmarks we report all four. The single number that correlates most strongly with "does this feel fast?" is TTFT plus the first 50 tokens of streaming output, which we call Time To Useful Response (TTUR). That's the metric we care about most.

Real Numbers From Our Latency Suite

The table below shows median TTFT, median ITL, end-to-end latency for a 300-token completion, and the input price per million tokens. Every row was sampled at least 500 times over a 14-day window from a US-East client, using the same prompt template (a 1,200-token system message plus a 200-token user question) to keep things comparable. Prices reflect publicly listed rates as of January 2026 and can change.

A few things jump out immediately. Google's Gemini 2.0 Flash is the speed king of the big-name commercial models, with a TTFT that feels closer to a database query than to an LLM call. Groq's Llama 3.3 70B endpoint is even faster on TTFT despite being an open-weight model running on custom LPU silicon, and it has the lowest ITL of anything we tested. On the slow end, reasoning models like o1 are roughly five to ten times slower than their non-reasoning siblings, and the bigger open-weight 405B-class models routed through generic GPU clouds are slower still, because you're paying for flexibility with raw tokens-per-second.

Pricing is the other story. DeepSeek V3 is a wild outlier at $0.27 input / $1.10 output while still being a frontier-tier model in terms of capability. Combined with its slower ITL, it occupies a unique spot in the matrix: cheap per token, expensive per second. For batch jobs where you're willing to wait, it's unbeatable on cost. For an interactive chat UI, you can do better.

Why Some APIs Feel Snappy and Others Feel Like Dial-Up

Latency isn't really a property of the model. It's a property of the system serving the model. A "fast" model running on a saturated cluster with 200 concurrent tenants will feel slow. A "slow" model running on dedicated H100s with speculative decoding will feel instant. Here are the four levers that explain 90% of the variance we saw.

Hardware. Groq's LPU, Google's TPUs, and AWS Trainium all deliver meaningfully better ITL than commodity A100 deployments. The difference between an A100 cluster and an H100 cluster for the same model is typically 30-50% lower ITL and a noticeable TTFT improvement thanks to better KV cache handling.

Batching and scheduling. Providers batch incoming requests to maximize GPU utilization. If you're the only request on a node, your ITL is essentially the model's raw decode speed. If you're batched with 32 other requests, your effective ITL multiplies by the batch factor. The best providers expose a "priority" or "fast lane" tier for an upcharge; the worst silently degrade you at peak hours.

Cold starts and routing. First request to a fine-tuned model, first request after a deployment change, or first request to a new region often pays a 2-10 second penalty while the system loads weights, warms caches, or spins up a pod. We saw this hit unpredictably on smaller providers who run on-demand autoscaling rather than warm pools.

Geography. Network round-trip from Sydney to a US-East inference cluster adds ~180ms of pure physics. If your users are global and you care about TTFT, you either need a provider with regional endpoints (OpenAI, Anthropic, Google, Mistral all have them) or you need to accept that your Australian users will wait. We tested the same model from US-East and from Frankfurt and consistently saw 80-150ms added TTFT for the cross-Atlantic hop.

How to Reproduce Our Benchmark in 30 Lines

You don't need to take our word for it. The script below hits the same endpoint we used to build the table above, streams the response so you can measure TTFT and ITL separately, and prints a summary. It uses the unified OpenAI-compatible endpoint at global-apis.com/v1, which lets us test models from a dozen providers with a single API key and the same request format.

import os, time, statistics, requests
from typing import List

API_KEY = os.environ["GLOBAL_APIS_KEY"]
BASE    = "https://global-apis.com/v1"
MODEL   = "gpt-4o-mini"  # swap for any of 184+ models

PROMPT = "Write a 300-word overview of vector databases for a CTO."

def benchmark(n_runs: int = 25) -> None:
    ttfts: List[float] = []
    itls:  List[float] = []
    e2es:  List[float] = []

    for _ in range(n_runs):
        t0 = time.perf_counter()
        first_token_at = None
        token_times: List[float] = []

        with requests.post(
            f"{BASE}/chat/completions",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={
                "model": MODEL,
                "messages": [{"role": "user", "content": PROMPT}],
                "stream": True,
                "max_tokens": 400,
            },
            stream=True,
            timeout=60,
        ) as r:
            r.raise_for_status()
            for line in r.iter_lines():
                if not line or not line.startswith(b"data: "):
                    continue
                if line == b"data: [DONE]":
                    break
                now = time.perf_counter()
                if first_token_at is None:
                    first_token_at = now
                    ttfts.append((now - t0) * 1000)
                else:
                    itls.append((now - token_times[-1]) * 1000)
                token_times.append(now)

        e2es.append((time.perf_counter() - t0) * 1000)

    def ms(xs): return f"{statistics.median(xs):.0f} ms"
    print(f"Model:          {MODEL}")
    print(f"Runs:           {n_runs}")
    print(f"Median TTFT:    {ms(ttfts)}")
    print(f"Median ITL:     {ms(itls)}")
    print(f"Median E2E:     {ms(e2es)}")
    print(f"P95 TTFT:       {ms(sorted(ttfts)[int(len(ttfts)*0.95)])}")

if __name__ == "__main__":
    benchmark()

Run this with your GLOBAL_APIS_KEY exported, and within a couple of minutes you'll have your own latency numbers on the exact model you're considering. Swap the model string for any of the others in the table (claude-3-5-sonnet, gemini-2.0-flash, llama-3.3-70b, deepseek-chat, mistral-large-latest, etc.) and rerun. The numbers won't match ours exactly because of the time-of-day, region, and routing differences, but the ranking will be stable.

Key Insights From 90 Days of Data

After running hundreds of thousands of requests, here's what genuinely surprised us and what simply confirmed our priors.

Reasoning models are a latency tax you choose to pay. o1, o3-mini, and the various "thinking" variants from DeepSeek and Qwen all add 1-3 seconds of prefill reasoning time before the first token appears. For coding tasks and math, that's a fantastic tradeoff. For a chatbot greeting, it's catastrophic. Match the model class to the interaction class.

The "fast lane" tier matters more than the model choice. Several providers (Anthropic, Google, Mistral) offer a priority routing tier for 2-3x the price. In our tests this dropped TTFT by 40-60% during US business hours. If you're building a real product with paying users, factor this in.