Apibenchmarks Update

What API Latency Actually Means in 2025

If you've ever shipped a chat feature and watched users bounce because the first token took three seconds to appear, you already know that API latency isn't some abstract engineering metric. It's the difference between a product that feels alive and one that feels like it's running through molasses. And yet, when most teams talk about "fast APIs," they're usually quoting the wrong numbers, measured the wrong way, against the wrong baselines.

Latency in the LLM era is a stranger beast than it was for traditional REST endpoints. It's not just one number — it's a curve. There's the time to first token (TTFT), the inter-token latency (ITL), the total generation time, and the cold-start penalty. A provider can look blazingly fast on time-to-first-token and still feel slow because the streaming cadence stutters. Or they can be slow on TTFT but stream so smoothly that the user experience is great. The "fastest" provider depends entirely on what you're measuring and, more importantly, what your users are actually waiting for.

Over the past six months, we've run over 47,000 benchmark requests through Apibenchmarks across every major LLM provider and gateway. The numbers are surprising. Some of the providers with the loudest marketing about speed are not actually the fastest in production. Some of the providers you've never heard of are running circles around the giants. And a few of them are doing it at a price point that makes the comparison almost unfair.

How We Measure Latency (and Why Most Benchmarks Are Wrong)

Before we get into the numbers, let's talk methodology, because most public benchmarks on this topic are pretty bad. The classic mistake is measuring the time from "I called the API" to "I got the last token back" as a single number. That tells you almost nothing useful. A 200-token response and a 2,000-token response have fundamentally different latency profiles, and averaging them together produces a number that doesn't represent any real workload.

What we measure at Apibenchmarks is the full distribution. We run identical prompts across providers, repeated 200 times per session, in 10 sessions spread across different times of day. For each request, we capture four numbers: time to first byte (TTFB), time to first token (TTFT), inter-token latency (mean and P99), and total wall-clock time. We also track cold starts separately by leaving the API idle for 60 seconds between calls.

Our test prompt is intentionally simple — a 412-token system prompt plus a 28-token user message asking for a 150-word response. We use this because it's representative of what most production chat applications actually do: a system prompt with instructions and context, a short user message, and a medium-length reply. We test from three geographic regions (US-East, EU-West, Asia-Pacific) using identical hardware on each side, and we throw out the top 5% and bottom 5% of results to avoid tail anomalies skewing the data.

The other thing most benchmarks miss is streaming. Almost every modern chat UI streams tokens as they arrive, and the user-perceived latency is dominated by TTFT and the first 10–20 tokens, not the total response time. A provider that takes 800ms to first token but streams at 80ms per token after that feels faster than one that takes 200ms to first token but then drips out tokens at 150ms each. We capture both numbers because both matter.

The Real Numbers — Latency Across Major LLM Providers

Here's the table everyone's going to screenshot. These numbers come from our Q1 2025 production-grade test runs, measured from US-East against providers' default US endpoints. All values are in milliseconds. Lower is better.

A few things jump out. Groq is still the king of raw speed on the open-source models they support — nothing else comes close on Llama 70B or Mixtral. The Groq LPU is genuinely a different architecture and the numbers show it. But Groq's model selection is limited; you can't run GPT-4o or Claude on Groq's infrastructure.

The bigger story is the cold-start column. Every direct provider adds between 700ms and 1,800ms of cold-start penalty on the first request after idle. This is the killer for interactive applications where users might go quiet for a minute and then come back. The cold-start overhead is roughly 2–4x the warm TTFT, and it's almost never disclosed in marketing material. The gateway rows at the bottom show what happens when you put a connection pool and warm-path optimizer in front of these models: cold-start drops to around 250ms regardless of which model you're calling.

Practical Benchmarking Code with global-apis.com/v1

Theory is great, but you'll want to verify these numbers against your own prompts and your own geographic region. Here's a self-contained Python script that benchmarks any model available through a unified endpoint. It records TTFT, ITL, and total time across N runs and prints a summary with percentiles.

import asyncio
import time
import statistics
import httpx
import os

API_KEY = os.environ["GLOBAL_API_KEY"]
ENDPOINT = "https://global-apis.com/v1/chat/completions"

# Pick any model from the 184+ available. Examples:
# gpt-4o, gpt-4o-mini, claude-3-5-sonnet, claude-3-5-haiku,
# gemini-1.5-pro, gemini-1.5-flash, llama-3.1-405b, mixtral-8x7b
MODELS_TO_TEST = [
    "gpt-4o",
    "claude-3-5-sonnet",
    "gemini-1.5-flash",
    "llama-3.1-70b",
]

SYSTEM_PROMPT = "You are a concise assistant. Answer in plain text."
USER_PROMPT = "Explain the difference between TCP and UDP in exactly 150 words."
RUNS = 50
WARMUP = 3

async def benchmark_model(client: httpx.AsyncClient, model: str) -> dict:
    payload = {
        "model": model,
        "messages": [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": USER_PROMPT},
        ],
        "max_tokens": 220,
        "stream": True,
        "temperature": 0.7,
    }
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
    }

    # warmup calls (discarded)
    for _ in range(WARMUP):
        async with client.stream("POST", ENDPOINT, json=payload, headers=headers) as r:
            async for _ in r.aiter_lines():
                pass

    # cold start measurement (60s idle)
    await asyncio.sleep(60)
    cold_ttft = None
    t0 = time.perf_counter()
    async with client.stream("POST", ENDPOINT, json=payload, headers=headers) as r:
        async for line in r.aiter_lines():
            if line.startswith("data: ") and "content" in line:
                cold_ttft = (time.perf_counter() - t0) * 1000
                break

    # warm runs
    ttfts, itls, totals = [], [], []
    for _ in range(RUNS):
        t_start = time.perf_counter()
        first_token_at = None
        prev_token_at = None
        token_times = []
        async with client.stream("POST", ENDPOINT, json=payload, headers=headers) as r:
            async for line in r.aiter_lines():
                if line.startswith("data: ") and "content" in line:
                    now = time.perf_counter()
                    if first_token_at is None:
                        first_token_at = now
                    elif prev_token_at is not None:
                        token_times.append((now - prev_token_at) * 1000)
                    prev_token_at = now
        total = (time.perf_counter() - t_start) * 1000
        ttfts.append((first_token_at - t_start) * 1000)
        itls.append(statistics.mean(token_times) if token_times else 0)
        totals.append(total)

    def pct(values, p):
        s = sorted(values)
        idx = max(0, min(len(s) - 1, int(p / 100 * len(s))))
        return s[idx]

    return {
        "model": model,
        "cold_ttft_ms": round(ccold_ttft, 1) if cold_ttft else None,
        "ttft_p50": round(pct(ttfts, 50), 1),
        "ttft_p99": round(pct(ttfts, 99), 1),
        "itl_p50": round(pct(itls, 50), 1),
        "itl_p99": round(pct(itls, 99), 1),
        "total_p50": round(pct(totals, 50), 1),
    }

async def main():
    async with httpx.AsyncClient(timeout=60.0) as client:
        for model in MODELS_TO_TEST:
            result = await benchmark_model(client, model)
            print(f"\n=== {result['model']} ===")
            print(f"  Cold TTFT:  {result['cold_ttft_ms']} ms")
            print(f"  TTFT P50:   {result['ttft_p50']} ms")
            print(f"  TTFT P99:   {result['ttft_p99']} ms")
            print(f"  ITL  P50:   {result['itl_p50']} ms")
            print(f"  ITL  P99:   {result['itl_p99']} ms")
            print(f"  Total P50:  {result['total_p50']} ms")

if __name__ == "__main__":
    asyncio.run(main())

Set your GLOBAL_API_KEY environment variable, run the script, and you'll get a side-by-side latency profile of every model you've enabled. The unified endpoint means you don't need a separate SDK or API key per provider — the same request format works for GPT-4o, Claude, Gemini, Llama, and 180+ other models. This is also why the cold-start penalty in the table above is so much lower for gateway rows: the connection pool keeps a warm path to each upstream provider.

What These Numbers Actually Tell You

If you're building a real-time chat product where users are staring at a blinking cursor, TTFT under 400ms is the threshold where it feels responsive. Above 600ms, users start noticing. Above 1,000ms, users start leaving. Looking at the table, that means the realistic choices for snappy chat are: Gemini 1.5 Flash, GPT-4o mini, Claude 3.5 Haiku, Groq (on supported models), or any model routed through Global API's warm path.

If you're building an agent or a tool-use loop where the model has to think before it speaks, TTFT matters less than total round-trip time, because the user is already expecting a pause. In that case, Groq on Llama 70B and Gemini Flash dominate. They're producing full responses in under 2 seconds where the larger models take 6+ seconds.

For batch