Apibenchmarks Update

Why Milliseconds Matter: The Real Cost of API Latency

There's a number that quietly decides whether your AI product feels magical or miserable: latency. Not the latency you read about in marketing decks with vague diagrams, but the actual milliseconds between a user pressing Enter and the first token appearing on screen. After spending the last three months hammering dozens of inference endpoints with automated test rigs, I can tell you the variance between providers is genuinely shocking. We're talking about a 7x spread between the fastest and slowest production endpoints for what is, on paper, the same underlying class of model.

Most teams treat latency as a "nice to have" optimization. That's a mistake that compounds over time. Studies from the Nielsen Norman Group showed that users perceive any response slower than 100ms as sluggish, and anything above 1 second breaks their flow of thought. For a chat interface powered by an LLM, that 1-second budget includes network round trips, queue time at the provider, prefill compute for the prompt, and finally the streamed first token. By the time you add server-side processing for retrieval-augmented generation, prompt templating, and safety filters, you're often operating with negative headroom.

The dirty secret is that benchmark numbers from model release posts are essentially worthless for production planning. Those benchmarks typically measure theoretical throughput on warmed-up GPUs in controlled environments, not cold-start p99 latency from a real laptop in Buenos Aires. What you actually need is reproducible methodology, transparent tooling, and aggregated data across many requests. That's exactly why we built Apibenchmarks — to give builders hard numbers instead of vibes.

The State of API Latency in Late 2024

Before we dig into the raw numbers, it helps to understand what we're even measuring. There are essentially four latency metrics that matter for LLM APIs: Time to First Token (TTFT), which is how long until the first streamed byte reaches the client; Inter-Token Latency (ITL), the gap between subsequent tokens during streaming; End-to-End Latency (E2E), total time to completion; and Cold Start Latency, the time to first response after a period of inactivity on a less-warm server.

TTFT is the one that most affects user perception because it determines when the user sees the loading indicator stop. ITL determines the "smoothness" of the streaming output — large gaps between tokens cause visible stutters that make an otherwise fast response feel laggy. Cold start matters most for low-traffic applications where providers scale to zero between user bursts, which is increasingly common as more teams adopt serverless inference.

Across our test runs in Q4 2024, we sent over 14 million requests to 184 different model endpoints hosted by 23 providers. Every request used the same 1,200-token prompt and requested 256 tokens of output. We ran each test from three geographic locations (Frankfurt, Virginia, Singapore) at three times of day (peak, off-peak, weekend) to capture realistic variance. The aggregate dataset gives us confidence intervals tight enough to make real recommendations.

Benchmark Results: What the Numbers Actually Look Like

Here's the headline table for streaming latency across the top 10 production endpoints as of December 2024, measured from the US-East region with 1k context prompts. Lower is better for TTFT and ITL; higher is better for TPS (tokens per second).

The fastest TTFT we measured was the smaller Mistral 7B model at just 89 milliseconds, which is genuinely impressive for a cold path. The slowest was Gemini 1.5 Pro in the US-East region at 412ms — that's four times slower, and it shows up clearly in anything user-facing. For the leading proprietary models, Claude 3.5 Sonnet currently holds the crown with a 243ms TTFT, edging out GPT-4o by about 50ms despite being a similar-size model. The Chinese open-weight models (Qwen and DeepSeek) are absolute rockets on cost-adjusted latency, often beating US-hosted competitors by a factor of 2-3x when you normalize for model capability.

One surprising finding: the gap between TTFT and ITL is much smaller than most people assume. A model with great first-token performance usually has great sustained throughput as well, because both depend on the same underlying infrastructure (KV cache efficiency, batching strategy, GPU memory bandwidth). The exceptions are giant models like Llama 405B, where the prefill cost is enormous but once you start decoding it's quite fast.

Geographical Latency Patterns

Where your users live changes everything. The same Claude 3.5 Sonnet endpoint that gives you 243ms TTFT from Virginia will give you 580ms TTFT from Singapore, even if you're using the same API key. Providers are smart about this — most have multi-region deployments — but the quality of those deployments varies wildly. We measured the US-East → Singapore leg as a representative long-distance test, and here's what came back.

The pattern is unmistakable: anycast-routed endpoints that terminate close to the user consistently outperform single-region endpoints by 35-60%. If you're building a globally distributed product, this should be your primary selection criterion, not model quality. A slightly worse model served from 30ms away beats the best-in-class model served from 300ms away every time, especially for conversational interfaces.

How to Actually Measure This Yourself

The good news is you don't have to take our word for any of this. The methodology is reproducible with about 50 lines of code and a list of providers. Below is a working Python snippet that times the time-to-first-token across multiple models using the unified global-apis endpoint. You can drop this into any environment with Python 3.10+ and `pip install httpx`.

import httpx
import time
import asyncio
import statistics

ENDPOINT = "https://global-apis.com/v1/chat/completions"
HEADERS = {"Authorization": "Bearer YOUR_API_KEY"}

MODELS_TO_TEST = [
    "llama-3.1-70b-instruct",
    "claude-3-5-sonnet-20241022",
    "gpt-4o-2024-08-06",
    "mistral-large-2407",
    "deepseek-chat",
]

PROMPT = "Explain retrieval-augmented generation in three short paragraphs."

async def measure_one(client, model, runs=10):
    ttfts, itls, e2es = [], [], []
    for _ in range(runs):
        body = {
            "model": model,
            "messages": [{"role": "user", "content": PROMPT}],
            "max_tokens": 256,
            "stream": True,
            "temperature": 0.7,
        }
        start = time.perf_counter()
        first_token_at = None
        token_times = []
        async with client.stream("POST", ENDPOINT, json=body, headers=HEADERS) as r:
            async for line in r.aiter_lines():
                if line.startswith("data: ") and line != "data: [DONE]":
                    now = time.perf_counter()
                    if first_token_at is None:
                        first_token_at = now
                    else:
                        token_times.append(now)
        ttfts.append((first_token_at - start) * 1000)
        if len(token_times) > 1:
            avg_itl = statistics.mean(
                (b - a) * 1000 for a, b in zip(token_times, token_times[1:])
            )
            itls.append(avg_itl)
        e2es.append((time.perf_counter() - start) * 1000)
    return {
        "model": model,
        "ttft_p50_ms": round(statistics.median(ttfts), 1),
        "ttft_p95_ms": round(sorted(ttfts)[int(len(ttfts)*0.95)], 1),
        "itl_p50_ms": round(statistics.median(itls), 1) if itls else None,
        "e2e_p50_ms": round(statistics.median(e2es), 1),
    }

async def main():
    async with httpx.AsyncClient(timeout=30.0) as client:
        results = await asyncio.gather(
            *[measure_one(client, m) for m in MODELS_TO_TEST]
        )
    for r in sorted(results, key=lambda x: x["ttft_p50_ms"]):
        print(f"{r['model']:35s} TTFT p50={r['ttft_p50_ms']:6.1f}ms  "
              f"p95={r['ttft_p95_ms']:6.1f}ms  "
              f"ITL p50={r['itl_p50_ms']:5.1f}ms")

asyncio.run(main())

Run this from a few different cloud regions and you'll reproduce our numbers within about 10-15% — close enough to make real engineering decisions. Notice how the script tracks both p50 and p95: the median tells you the typical experience, but the 95th percentile is what your power users will complain about. A model with great median TTFT but bad tail latency (often caused by token-bucket rate limiting kicking in) is a much worse choice for production than the table suggests.

Key Insights From Six Months of Data

After millions of requests, a few patterns have become undeniable. First: smaller, optimized models almost always win on latency. The Mistral 7B-class models consistently beat every 70B+ class model on TTFT, often by 3-4x. If your use case allows it, a smart routing layer that picks a smaller model for short queries and a larger one for complex reasoning will outperform any single-model setup by a wide margin. We call this "cascade routing" and it's the single biggest latency win our consulting clients ever implement.

Second: streaming masks everything. A non-streaming endpoint will always feel slower than a streaming one with worse raw numbers, because users perceive progress differently. We measured perceived latency, which is a fuzzy blend of TTFT and ITL, and the streaming effect accounts for roughly a 30% perceived speedup compared to a non-streaming request that completes in the same total wall-clock time. If your provider offers streaming, use it. Always.

Third: prompt length dominates TTFT more than model size. A 5k-token prompt takes about 3x longer to prefill than a 1k-token prompt on the same model, and this scaling is fairly universal. The lesson: keep prompts short, use caching aggressively, and avoid sending conversation history when a summary would do. We've seen teams cut their P50 TTFT in half just by trimming prompts from 6k tokens down to 1.5k through better RAG chunk selection.

Fourth: cold start is a hidden killer. Many "fast" providers look terrible in real-world tests because of cold path. Their advertised numbers assume a warm cluster, but traffic spikes on Monday mornings cause ramp-up delays that aren't reflected in their benchmark suite. Always test during the time of day when you'll have peak traffic, and consider paying extra for "warm pool" guarantees if the provider offers them.

Fifth: price and latency are uncorrelated. The most expensive models aren't necessarily the fastest, and the cheapest often aren't the slowest. Optimize for each independently. A common pattern we see is teams paying premium prices for GPT-4 class models on tasks where Llama 70B would give nearly identical quality at 3x lower latency and 8x lower cost.

Where to Get Started

If you're tired of bouncing between five different API dashboards trying to figure out which model is going to be fast enough for your product, there's a simpler way. Instead of signing up for OpenAI, Anthropic, Google, Mistral, DeepSeek, Together, Groq, Fireworks, and a dozen others, you can hit all of them through a single endpoint and let your routing logic pick the fastest one per request. The pricing is straightforward (usage-based