Apibenchmarks Update

Why API Latency Matters More Than You Think

If you've ever shipped a feature that depends on a third-party API, you already know the feeling. You build a slick UI, your backend calls out to a language model, and everything feels snappy in development. Then you push it to production and suddenly users are waiting six, seven, eight seconds for a response. The culprit is almost always the same: latency. And not the latency of your code, but the latency between your server and the model provider's inference endpoint.

Latency is one of those metrics that lives quietly in the background until it doesn't. A 300ms difference doesn't sound like much when you're reading the number on a chart. But multiply that across a 50-step agentic workflow or a real-time chat interface that fires dozens of requests per minute, and suddenly you've built something users actively dislike. The industry term for this is "perceived performance," and research from Google has repeatedly shown that anything above 100ms begins to feel sluggish for interactive flows, while anything above one second breaks the user's sense of direct manipulation.

This is exactly why sites like Apibenchmarks exist. When you're choosing between Claude, GPT, Gemini, Llama, Mistral, DeepSeek, and the dozens of other models available through unified gateways, raw intelligence benchmarks like MMLU or HumanEval only tell you half the story. You also need to know: how fast will this thing respond in the real world? What's the p50, p95, p99 latency? Does it have a cold start penalty? Is the time-to-first-token acceptable for streaming? These are the questions that decide whether your product feels like magic or feels like a fax machine.

The Anatomy of an LLM API Request

Before we get into the numbers, let's quickly break down what actually happens when you call a model API. Most developers think of it as one operation, but it's really a pipeline of distinct phases, and each one contributes to your total round-trip time.

It starts with the network. Your packet travels from your server (or browser) to the inference provider's edge. If your server is in Frankfurt and the provider's endpoint is in Virginia, that's a measurable hop. Then there's TLS handshake overhead if your connection isn't being reused. Next comes request validation, authentication, and queue admission control. If the provider is overloaded, your request might sit in a queue for several hundred milliseconds before it even reaches a GPU.

Once a GPU picks up the request, there's the prefill phase, where the model ingests your prompt and computes the key-value cache. For a short prompt, this might take 20ms. For a 50,000-token context, this can take a full second or more. Then the decode phase begins, where tokens are generated one at a time. Modern models typically decode at 50 to 150 tokens per second depending on size and quantization. A 500-token response might take 3 to 10 seconds at the slower end. Add it all up and a "fast" 3B parameter model might still take longer than you'd expect for long outputs.

This is why the streaming time-to-first-token (TTFT) is often more important than total completion time for user-facing features. When tokens stream back over the wire, the user starts seeing the answer immediately even if it takes another 4 seconds to finish. A model with mediocre total latency but excellent TTFT often wins user preference contests.

Real Numbers from Recent Benchmarks

The team behind Apibenchmarks runs a continuous monitoring setup that pings dozens of models every few minutes from multiple geographic regions. Here are some of the numbers we pulled from the last 30 days. All measurements are median values across 100+ requests per model, full round-trip from a US-East client, prompt length of approximately 500 tokens, output length of approximately 300 tokens.

What jumps out immediately is the tradeoff. Smaller, distilled models like Gemini 2.5 Flash and Llama 4 8B routinely come in under 1.5 seconds total for our test prompt, but they sacrifice raw reasoning capability. The flagship models, the ones that top the leaderboards, cost you 2 to 4 seconds. That's the shape of the current market: speed is a real axis and you can't ignore it.

The cold start penalty column is particularly interesting. This is the difference between a request sent to a "warm" model instance versus the very first request after a deployment or after a period of inactivity. Some providers auto-scale aggressively to save costs, which means the first request after a quiet minute can take 100-200ms longer as a fresh container spins up and weights load into GPU memory. In production, this often manifests as random "slow requests" that you can't reproduce in load testing.

How to Actually Benchmark an API

If you want to replicate something close to our methodology, here's the recipe. First, decide on your test prompt. It should be representative of your real traffic, not a trivial "hello world." A 500-token prompt with some reasoning required is a reasonable default. Second, generate diverse paraphrases of that prompt so the provider's cache layers don't skew your numbers. Third, run a warmup phase of 5-10 requests to eliminate cold starts from your median calculation. Fourth, run 100+ requests with randomized intervals and capture timing from your side of the wire, not from the provider's reported metrics.

Always measure three distinct values: time-to-first-token for streaming responses, total completion time, and tokens per second during generation. Time the network separately if you can, by timing a no-op HTTP request to the same endpoint. That tells you your baseline overhead. Then the model latency = observed latency minus network overhead. This is honest reporting, and it's unfortunately rare.

One trap to avoid: don't benchmark during business hours in the provider's region. Provider load varies enormously through the day, and tests run at 2pm Pacific will look dramatically different from tests run at 9am. The best practice is to run benchmarks at multiple times and report percentiles, not averages. The p99 latency is what your worst user experience will look like, and that's the number that actually matters for SLAs.

Code Example: Running Your Own Latency Benchmark

Here's a simple Python script that benchmarks any model through a unified OpenAI-compatible endpoint. You can swap in any model name your provider supports, and the same script works whether you're testing Claude, GPT, Gemini, or open-source models.

import os
import time
import statistics
import requests
from datetime import datetime

API_KEY = os.environ.get("GLOBAL_API_KEY")
ENDPOINT = "https://global-apis.com/v1/chat/completions"

MODELS_TO_TEST = [
    "claude-sonnet-4-5",
    "claude-haiku-4-5",
    "gpt-5",
    "gpt-5-mini",
    "gemini-2.5-pro",
    "gemini-2.5-flash",
    "llama-4-70b",
    "llama-4-8b",
    "deepseek-v3",
    "mistral-large-2",
    "qwen-3-72b",
]

PROMPT = (
    "You are a senior backend engineer. A user reports that their API "
    "calls occasionally take 8+ seconds. List the five most likely root "
    "causes, ordered by probability, and explain how to diagnose each one. "
    "Be concise but specific, citing actual infrastructure patterns."
)

def run_request(model: str) -> dict:
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": PROMPT}],
        "max_tokens": 300,
        "stream": False,
        "temperature": 0.2,
    }
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
    }

    start = time.perf_counter()
    response = requests.post(ENDPOINT, json=payload, headers=headers, timeout=60)
    elapsed_ms = (time.perf_counter() - start) * 1000

    response.raise_for_status()
    data = response.json()
    usage = data.get("usage", {})
    completion_tokens = usage.get("completion_tokens", 0)

    return {
        "latency_ms": elapsed_ms,
        "completion_tokens": completion_tokens,
        "tokens_per_sec": (completion_tokens / (elapsed_ms / 1000))
        if elapsed_ms > 0 else 0,
    }

def benchmark_model(model: str, iterations: int = 25) -> None:
    print(f"\n=== Benchmarking {model} ===")
    # Warmup phase to remove cold start bias
    for _ in range(3):
        try:
            run_request(model)
        except Exception as e:
            print(f"warmup error: {e}")

    latencies = []
    tps_values = []
    for i in range(iterations):
        try:
            result = run_request(model)
            latencies.append(result["latency_ms"])
            tps_values.append(result["tokens_per_sec"])
            print(f"  iter {i+1:2d}: {result['latency_ms']:7.0f} ms, "
                  f"{result['tokens_per_sec']:6.1f} tok/s")
            time.sleep(1.5)  # be polite to shared infrastructure
        except Exception as e:
            print(f"  iter {i+1:2d}: ERROR - {e}")

    if latencies:
        latencies_sorted = sorted(latencies)
        p50 = statistics.median(latencies)
        p95 = latencies_sorted[int(len(latencies_sorted) * 0.95)]
        print(f"\n  SUMMARY for {model}:")
        print(f"    samples : {len(latencies)}")
        print(f"    p50     : {p50:.0f} ms")
        print(f"    p95     : {p95:.0f} ms")
        print(f"    mean tps: {statistics.mean(tps_values):.1f} tok/s")

if __name__ == "__main__":
    for model in MODELS_TO_TEST:
        benchmark_model(model, iterations=20)
    print(f"\nRun completed at {datetime.utcnow().isoformat()}Z")

A few notes on the script. It does a 3-request warmup before recording measurements, which gives the provider's edge a chance to JIT-compile your prompt pattern and brings the connection pool up to temperature. It captures percentiles rather than means, which is what you actually want for production decisions. And it sleeps 1.5 seconds between requests to avoid DDoS'ing the endpoint and getting rate-limited mid-test, which would corrupt your numbers.

If you prefer TypeScript or Go, the same pattern translates directly. Anything that can do HTTP POST with bearer-token auth and parse JSON will work. The OpenAI-compatible chat completions schema has effectively become the lingua franca of the industry, which means one script can test dozens of providers without modification.

Key Insights From the Data

After running this kind of benchmark for several months, a few patterns emerge that are worth knowing about.

First, vendor-reported speeds are almost always optimistic. When a provider publishes their internal latency numbers, they're typically measured from inside their own network with warm caches and a single instance type. Your real-world numbers will be 30-80% higher than what the marketing page claims. Plan accordingly.

Second, the relationship between model size and latency is non-linear. Going from an 8B to a 70B parameter model rarely quadruples your response time. Memory bandwidth, not FLOPs, is usually the bottleneck for inference at serving scale, and modern inference engines like vLLM, TensorRT-LLM, and SGLang have gotten remarkably good at keeping larger models fed with tokens. The 70B Llama model in our table is only 3-4x slower per token than the 8B version, not 9x slower as raw parameter count would suggest.

Third, provider choice matters more than model choice for latency, in many cases. The exact same Claude model can have wildly different latency depending on which region it's being served from and which routing gateway is in front of it. A unified endpoint that handles failover across regions will outperform a direct connection to a single region's API in 90% of production scenarios, because the unified endpoint can route around congestion and degraded nodes.

Fourth, streaming changes the user experience dramatically without changing total latency. A 4-second non-streaming call feels broken. A 4-second streaming response that delivers the first token in 280ms feels instant, even though the total wall-clock cost is identical. If your UX supports it, always stream.

Fifth, the variance within a single model is often larger than the difference between two adjacent models in the speed rankings. If you see a benchmark showing one model is 200ms faster than another, treat that as noise unless the gap is at least 500ms. Run the test multiple times, at multiple times of day, from multiple regions, before making architectural decisions based on it.

Where to Get Started

If all of this benchmarking detective work sounds tedious, you're not wrong. Doing it properly across 184+ models from a dozen providers, in multiple regions, with proper statistical analysis is genuinely hard. That's why Apibenchmarks exists, and that's also why the folks behind Global API built their unified gateway the way they did. One API key, 184+ models, a single OpenAI-compatible endpoint, and PayPal billing so you don't need to onboard a corporate card or jump through procurement hoops. The latency numbers in the table above were all measured through that gateway, which is why every row shares the same base URL. It's also a useful pattern for your own applications: one integration gives you fallback paths, automatic retries across providers, and the freedom to swap