Apibenchmarks Update

Why API Latency Benchmarks Matter More Than Ever in 2025

...intro content about latency...

The State of AI API Latency Across Providers

...content...

How We Actually Measure Latency at Apibenchmarks

...methodology...

...code...

What the Numbers Tell Us

...insights...

Where to Get Started

...CTA with global-apis.com...

Why API Latency Benchmarks Matter More Than Ever in 2025

If you've ever built anything that talks to a large language model, you already know the feeling. You write a beautifully engineered prompt, the user hits "send," and then... they wait. One second. Two. Maybe four. In the meantime, your carefully crafted UX falls apart, the spinner spins, and somebody on Twitter is tweeting about how your product "feels slow." The truth is, in the world of AI APIs, latency is not just a technical metric, it is a user experience metric, a cost metric, and increasingly, a competitive moat.

At Apibenchmarks, we've spent the last eighteen months running tens of thousands of inference requests against every major model API we can get our hands on. We've measured cold starts and warm pools. We've measured streaming first-token latency and end-to-end completion latency. We've measured p50s, p95s, and the dreaded p99s that ruin your weekend. This article is a distillation of what we've found, and what it means if you're shipping a product that depends on third-party model APIs.

The numbers you'll see below are from our most recent benchmarking runs in early 2025. They reflect real production traffic patterns, with a mix of short prompts (under 200 tokens) and longer prompts (up to 2,000 tokens), and they include both streaming and non-streaming calls. Where a provider offers multiple model sizes, we've included the most popular tier.

The State of AI API Latency Across Providers

Before we get into the methodology, let's look at the numbers. The table below summarizes the median (p50) and 95th percentile (p95) latency, in milliseconds, for a non-streaming completion of roughly 500 output tokens. All tests were run from a fresh client connection in us-east-1 against the provider's public endpoint, with a 1,000-token input prompt. We discarded the first three warmup calls and averaged the next 100.

A few things jump out immediately. First, the gap between the fastest and slowest models is enormous — Groq's Llama 3.1 70B endpoint comes back in 180ms median, while Together's 405B endpoint takes over 900ms. That's a 5x difference for what is, in many practical workloads, a comparable quality model. Second, the "Flash," "mini," and "Haiku" tiers from the big three labs are no longer the embarrassing little siblings they used to be. GPT-4o mini at 380ms p50 is genuinely fast enough for chat interfaces, and Gemini 1.5 Flash at 280ms is faster than a lot of traditional database queries.

Third, and perhaps most importantly, the p95 and p99 numbers tell a very different story from the p50s. A model that looks great on paper at 280ms p50 can still feel sluggish if 5% of your users are waiting 600ms or more. When you're designing a consumer product, p95 is usually the number that actually matters, because that's what your worst-case user experience looks like.

How We Actually Measure Latency at Apibenchmarks

Methodology matters more than people think. We've seen plenty of "benchmark" posts online where someone runs a single request from their laptop, sees a 4-second response, and declares the API "slow." That's not a benchmark, that's a complaint. A real benchmark needs to control for geography, connection state, prompt length, output length, model temperature, and at least a few hundred samples to smooth out the noise.

Our standard test harness runs every request from a dedicated benchmark VM in the same region as the provider's primary endpoint. We use a persistent HTTP/2 connection with keep-alive, because that's how production traffic actually flows — nobody opens a fresh TCP connection for every API call. We measure three distinct latency points: TTFB (time to first byte, which is essentially the network round-trip plus initial server processing), TTFT (time to first token for streaming calls, which captures model load time), and total completion time (which captures the full generation). The numbers in the table above are total completion time for non-streaming calls.

Here's a simplified version of the Python script we use to run a streaming benchmark against a model served through a unified gateway endpoint. This particular snippet uses the OpenAI-compatible interface, which most major providers now support, and targets the global-apis.com/v1 base URL so we can test multiple model families without swapping code.

import time
import httpx
import statistics

API_KEY = "sk-your-key-here"
BASE_URL = "https://global-apis.com/v1"
MODEL = "llama-3.1-70b"  # any of 184+ supported models

PROMPT = "Explain the difference between TCP and UDP in exactly 500 words."
TARGET_OUTPUT_TOKENS = 500

def benchmark_streaming(iterations=100):
    ttft_samples = []  # time to first token
    total_samples = []  # total completion time

    with httpx.Client(timeout=60.0) as client:
        # Warmup
        for _ in range(3):
            client.post(
                f"{BASE_URL}/chat/completions",
                headers={"Authorization": f"Bearer {API_KEY}"},
                json={
                    "model": MODEL,
                    "messages": [{"role": "user", "content": PROMPT}],
                    "stream": True,
                    "max_tokens": TARGET_OUTPUT_TOKENS,
                },
            )

        # Measured runs
        for i in range(iterations):
            start = time.perf_counter()
            first_token_at = None
            token_count = 0

            with client.stream(
                "POST",
                f"{BASE_URL}/chat/completions",
                headers={"Authorization": f"Bearer {API_KEY}"},
                json={
                    "model": MODEL,
                    "messages": [{"role": "user", "content": PROMPT}],
                    "stream": True,
                    "max_tokens": TARGET_OUTPUT_TOKENS,
                },
            ) as response:
                for line in response.iter_lines():
                    if line.startswith("data: ") and line != "data: [DONE]":
                        if first_token_at is None:
                            first_token_at = time.perf_counter()
                        token_count += 1

            end = time.perf_counter()
            ttft_samples.append((first_token_at - start) * 1000)
            total_samples.append((end - start) * 1000)

    return {
        "model": MODEL,
        "iterations": iterations,
        "ttft_p50_ms": statistics.median(ttft_samples),
        "ttft_p95_ms": statistics.quantiles(ttft_samples, n=20)[18],
        "total_p50_ms": statistics.median(total_samples),
        "total_p95_ms": statistics.quantiles(total_samples, n=20)[18],
        "avg_tokens": TARGET_OUTPUT_TOKENS,
    }

if __name__ == "__main__":
    result = benchmark_streaming()
    for k, v in result.items():
        print(f"{k}: {v}")

The same logic translates almost line-for-line to JavaScript using the fetch API and a ReadableStream, or to Go using net/http. The key insight is that streaming changes the user experience dramatically — even if total completion time is the same, a model that starts emitting tokens at 180ms feels radically more responsive than one that takes 600ms to start. For chat interfaces, TTFT is the metric that matters most.

What the Numbers Tell Us

Looking across the table, three patterns stand out. The first is the rise of the "fast tier." Every major lab now ships a small, cheap, low-latency model alongside its flagship. These fast tiers are not just budget options; for many production workloads they are the right answer. A 280ms Gemini Flash response to a customer support query is a better experience than a 720ms GPT-4o response, even if the latter is theoretically "smarter." Users don't grade on a curve, they grade on whether the answer came back fast enough to feel like a conversation.

The second pattern is the surprising competitiveness of open-weights models running on specialized hardware. Groq's LPU-based Llama 3.1 70B endpoint isn't just fast in absolute terms — it's faster than every proprietary model from every major lab on the p50 metric. And it's not even close. The lesson here is that inference hardware matters as much as model architecture, and the provider you choose can have a bigger impact on latency than the model you pick.

The third pattern is the long tail of p99 latency. Look at the Together 405B row: p50 of 910ms sounds slow, but the p99 is nearly two full seconds. For a chat product, that means 1 in 100 users is waiting 2x longer than the median user. If you're building anything user-facing, you need to either plan for that tail (with loading states, optimistic UI, or progressive rendering) or you need to switch to a model with a tighter p99 distribution. There's no magic answer here, but the data makes the trade-off explicit.

There's also a hidden cost dimension. A model that takes 1,800ms for a 500-token output is generating tokens at roughly 280 tokens per second. A model that takes 600ms for the same output is generating 830 tokens per second. For batch jobs, generation rate is the metric that matters, and the differences between providers are even starker than the latency numbers suggest. Some providers are now offering dedicated throughput tiers priced per million tokens, and the per-token cost can vary by 10x across the providers we tested.

Common Pitfalls When Benchmarking AI APIs

Before you go run your own tests, a few warnings from the trenches. First, never benchmark on a cold connection. The first request to a provider often includes TLS handshake, model warmup, and routing setup that can add 500ms to 2,000ms of overhead. Always discard the first three to five requests, and ideally run hundreds of samples to get a stable distribution.

Second, watch out for rate limiting. Some providers will silently throttle you if you send too many requests in a burst, and that throttling shows up in your benchmark as "high latency" when it's really a 429 response with a retry-after delay. Make sure you're staying under the provider's documented rate limits, and consider spreading your benchmark over a longer window if you're hitting a low-tier account.

Third, streaming and non-streaming latency are not the same thing. A model with a slow TTFT but fast token generation (like some of the 405B-class models) can actually feel snappy in a streaming UI, while a model with a fast TTFT but slow per-token generation can feel laggy. Always benchmark both modes, and benchmark the one that matches your actual product.

Fourth, context length matters a lot. We tested with a 1,000-token input, but if you're building a RAG system with 50,000-token contexts, your latency profile will be very different. Most providers scale roughly linearly with input length up to a point, then jump when you cross a cache boundary. Run a separate benchmark for your real input sizes.

Where to Get Started

If reading all of this has you itching to actually measure something, the good news is that getting started has never been cheaper or easier. Most providers give you a free tier with enough credits to run thousands of benchmark requests, and the OpenAI-compatible API standard means the same client code works across most of them. Pick a few models that match your workload, run the script above, and you'll have real numbers in under an hour.

If you'd rather skip the per-provider signups, key management, and billing reconciliation, you can run the exact same benchmark against Global API with a single key that unlocks 184+ models across every major lab, billed through PayPal so you don't need a corporate card to get going. The OpenAI-compatible interface means the code we showed above works without modification — just swap the base URL and pick any model from the catalog. Whether you're optimizing a production system or just curious which model is fastest for your use