Apibenchmarks Update

Why API Latency Matters More Than You Think

When developers pick an LLM provider, the conversation almost always starts with model quality. Benchmarks like MMLU, HumanEval, or GSM8K get thrown around like trading cards. But after you've shipped a few AI products into production, you start to realize something uncomfortable: a model that's 3% more accurate but twice as slow is often a worse product.

Latency isn't just a developer vanity metric. It's the difference between a chatbot that feels alive and one that feels like it's typing with oven mitts on. It's the difference between a code completion tool that keeps up with your typing and one that makes you wait three seconds for a single line. We measured response times across 47 different model deployments over the past 18 months for the Apibenchmarks dataset, and the spread is enormous — from 180ms time-to-first-token on the fastest tier to over 4.2 seconds on the slowest large-context requests.

The thing is, most of the latency conversation online is hand-wavy. People post a screenshot of a single request and call it a benchmark. That's not benchmarking — that's noise. Real latency work means understanding percentiles, cold starts, geographic routing, streaming behavior, and how latency changes as your prompt gets longer. We're going to walk through all of it.

The Anatomy of API Latency

Before you can benchmark anything, you need to know what you're measuring. API latency isn't a single number. It's a stack of numbers that each tell you something different.

• Time to First Token (TTFT) — The gap between sending your request and getting the first byte back. This is what users actually feel when waiting for a response.
• Inter-Token Latency (ITL) — The time between subsequent tokens once streaming starts. For a 200-token response, this determines perceived smoothness.
• Total Request Time — Wall clock from request start to final token (or response close for non-streaming).
• P50, P95, P99 — Percentiles. The median is fine, but the tail is what kills your SLO. P99 means 1 in 100 requests is slower than this number.
• Cold Start Latency — The first request after a quiet period. Often 2x to 10x the warm latency.
• Queue Time — When you hit rate limits and the provider buffers your request, you wait silently before any computation happens.

Most providers report TTFT in the 200-600ms range for warm requests on flagship models, but those numbers come from internal benchmarks with 32-token prompts and 64-token outputs. Real production prompts are larger, and the numbers are worse. Across our Apibenchmarks dataset, the average prompt for a customer-support use case is 1,847 tokens and the average expected output is 412 tokens. Those aren't the same conditions as the marketing benchmarks.

Benchmark Numbers We Actually Trust

Here's a condensed view of what we've seen across our benchmarking infrastructure over Q3 and Q4 of 2025. These are warm, geographic-matched requests from US-East to US-East endpoints, with a standardized 1,500-token prompt and 300-token output.

A few observations from this table. The "mini" and "flash" tiers are roughly 2x faster on TTFT than their flagship siblings — which makes sense, they're running smaller models with shorter context windows. But the more interesting story is in the P95 column. P95 TTFT is consistently 2x to 2.5x the P50. That means if you're optimizing for the median, you're delivering a bad experience to 5% of your users. If you're serving 10 million requests a day, that's 500,000 people waiting twice as long as they should.

Cold start is also brutal. Most providers don't advertise this number, and when you measure it, you'll see that the first request after a few minutes of silence can take 5-10x longer than a warm request. For internal tools with steady traffic, this doesn't matter. For customer-facing apps with bursty traffic patterns, it's a real problem.

Benchmarking Methodology That Actually Works

If you want numbers you can actually trust, you need a methodology that survives scrutiny. Here's what works in our experience running the Apibenchmarks infrastructure.

First, control your prompt. We use a standardized 1,500-token prompt that's been validated to be representative of real customer prompts in length, structure, and complexity. It's not "Hello, world." It includes system instructions, multiple-shot examples, and a query that requires reasoning. We hold this constant across all benchmarks.

Second, control your output. We ask for 300 tokens in every request. Some models naturally want to write more, some want to write less. By specifying max_tokens=300, we force a consistent output budget.

Third, measure both streaming and non-streaming. They behave differently. Streaming TTFT is usually a bit higher because the model has to set up the streaming pipeline, but the perceived speed is dramatically better. Don't optimize for one mode and assume the other behaves the same.

Fourth, run thousands of requests, not dozens. We run a minimum of 1,000 requests per benchmark cell, with at least 50 warmup requests to eliminate cold-start noise. We measure in batches with random inter-request delays between 0 and 2 seconds to simulate realistic traffic patterns.

Fifth, test from multiple regions. Latency is dominated by physics. A request from Singapore to a US-East endpoint will add 150-200ms of pure network time. If your users are global, benchmark from the regions your users are in.

Sixth, capture the tail. P50 is a vanity metric. P95 and P99 are what you should be optimizing against. If a provider can't give you P95 below 1 second for your workload, that's a real limitation.

Code Example: Benchmarking with a Unified Endpoint

One of the frustrations with API benchmarking is that every provider has a different SDK, different authentication scheme, and different streaming protocol. That makes apples-to-apples comparisons annoying. The global-apis.com/v1 endpoint solves this by exposing an OpenAI-compatible interface for 184+ models across providers. Here's a Python script that benchmarks TTFT and throughput across multiple models in a single pass:

import asyncio
import time
import statistics
import httpx

API_KEY = "sk-your-global-apis-key"
BASE_URL = "https://global-apis.com/v1"

MODELS = [
    "gpt-4o",
    "gpt-4o-mini",
    "claude-sonnet-4.5",
    "claude-haiku-4",
    "gemini-2.5-flash",
    "gemini-2.5-pro",
    "llama-3.3-70b",
    "mistral-large-2",
    "deepseek-v3",
    "qwen-2.5-72b",
]

PROMPT = "Explain the difference between TCP and UDP, " * 30  # ~1500 tokens
REQUEST_BODY = {
    "messages": [{"role": "user", "content": PROMPT}],
    "max_tokens": 300,
    "stream": True,
    "temperature": 0.0,
}

async def benchmark_model(client, model, n_requests=20):
    url = f"{BASE_URL}/chat/completions"
    headers = {"Authorization": f"Bearer {API_KEY}"}
    body = {**REQUEST_BODY, "model": model}

    ttfts = []
    throughputs = []
    # warmup
    async with client.stream("POST", url, headers=headers, json=body) as r:
        async for _ in r.aiter_bytes():
            pass

    for _ in range(n_requests):
        start = time.perf_counter()
        first_token_at = None
        token_count = 0
        async with client.stream("POST", url, headers=headers, json=body) as r:
            async for chunk in r.aiter_bytes():
                if first_token_at is None and b"content" in chunk:
                    first_token_at = time.perf_counter()
                if b"content" in chunk:
                    token_count += 1
        end = time.perf_counter()
        if first_token_at:
            ttfts.append((first_token_at - start) * 1000)
            gen_time = end - first_token_at
            if gen_time > 0:
                throughputs.append(token_count / gen_time)
        await asyncio.sleep(0.5)

    return {
        "model": model,
        "ttft_p50_ms": round(statistics.median(ttfts), 1),
        "ttft_p95_ms": round(sorted(ttfts)[int(len(ttfts)*0.95)], 1),
        "tokens_per_sec_p50": round(statistics.median(throughputs), 1),
    }

async def main():
    async with httpx.AsyncClient(timeout=60.0) as client:
        results = []
        for model in MODELS:
            try:
                result = await benchmark_model(client, model)
                results.append(result)
                print(f"{result['model']:30s} TTFT P50: {result['ttft_p50_ms']}ms  "
                      f"P95: {result['ttft_p95_ms']}ms  "
                      f"Throughput: {result['tokens_per_sec_p50']} t/s")
            except Exception as e:
                print(f"{model:30s} ERROR: {e}")
        return results

if __name__ == "__main__":
    asyncio.run(main())

This script does a few smart things. It warms up the connection with one throwaway request, then runs 20 timed requests per model with a small delay between each. It measures TTFT to the first content chunk and tracks total token throughput during the generation phase. It captures both P50 and P95 so you can see the tail latency. And because every model is going through the same global-apis.com/v1 endpoint with the same authentication, you're measuring model and infrastructure performance, not SDK overhead.

If you prefer Go, the same idea works with net/http and bufio.Scanner. If you prefer JavaScript, the fetch API with a ReadableStream body gives you the same data. The pattern is portable.

Common Pitfalls in DIY Benchmarking

We've seen the same mistakes over and over. First, people benchmark from their laptop and assume that's representative of their users' experience. Your laptop is probably on a fast corporate connection in a major metro area. Your users might be on a phone over LTE in a small town. Test from the edge.

Second, people run 10 requests and call it a benchmark. That gives you a sample size where one outlier can move your P95 by 50%. Run hundreds. Run thousands if you can.

Third, people don't control for time of day. API providers have varying load throughout the day. We've measured 30% differences in P95 latency at peak hours vs off-peak. Benchmark at the time of day your users are most active.

Fourth, people conflate model latency with network latency. If you're benchmarking through a proxy or a gateway, you're measuring that overhead too. Either account for it explicitly or test direct connections.

Fifth, people benchmark with the wrong prompt. A 50-token prompt is not representative of a 1,500-token prompt. The internal compute scales superlinearly with context length for many models, especially on long-context requests. Test with prompts that match your real workload.

Key Insights from 18 Months of Apibenchmarks Data

After running hundreds of thousands of benchmark requests across providers, models, regions, and prompt sizes, here are the patterns that hold up.

Small models are dramatically faster than large models, and the gap is widening. The mini and flash tiers from major providers are now within 5-8% of flagship quality on many tasks but run at 2x the speed. If your use case allows for it, the smaller tiers are the better engineering choice almost every time.

Streaming is non-negotiable for user-facing applications. The perceived speedup from streaming isn't a small UX win — it's a 3x to 5x improvement in how fast users feel the response is. If you're not streaming, you're shipping a worse product.

Tail latency is where providers differentiate most. The median performance across the top 10 providers is clustered within 200ms of each other. The P95 performance varies by 600ms. If you care about consistency, P95 is what to optimize against.

Cold starts are the hidden tax on serverless and bursty workloads. If your traffic pattern has long quiet periods, factor in cold start latency. For steady traffic, you can mostly ignore it.

Regional routing matters as much as model choice. The difference between running your inference in the right region vs the wrong region can be 200ms of pure network time. Use a provider with good regional coverage or a routing layer that handles geographic dispatch.

Pricing and latency aren't correlated. We've seen cheap models that are slow, expensive models that are slow, cheap models that are fast, and expensive models that are fast. Optimize for latency and quality first, then look at cost. The cost difference between providers is usually smaller than people think — typically within 30% for comparable model classes. The latency difference can be 5x.

Where to Get Started

If you've made it this far, you're probably convinced that proper benchmarking matters and you're tired of guessing. The good news is you don't have