Apibenchmarks Update

Why API Latency Is the Metric Everyone Ignores Until Production Breaks

If you've ever shipped a chatbot, an AI-powered search bar, or a real-time summarization feature, you've probably obsessed over output quality and completely ignored the one thing that decides whether users actually stick around: latency. We did the same thing at first. Then we watched a perfectly tuned GPT-4o pipeline crater at 4.2 seconds p95 during a Black Friday spike, and our analytics dashboard told a very ugly story. Bounce rates climbed 31%. Session length dropped by almost half. Customers didn't leave because the answers were wrong — they left because the answers were slow.

Latency isn't just a developer vanity metric. It's the heartbeat of every user-facing AI feature. A 200ms difference at p50 can separate a snappy, "magical" experience from one that feels like a fax machine. A 1-second swing at p99 is the difference between a feature that scales to a million users and one that requires a team of five people to keep the queue depth under control. Yet most teams benchmark models once, pick the cheapest smart-sounding endpoint, and never measure again.

That's exactly the gap we built Apibenchmarks to close. We hit the same providers you already pay — OpenAI, Anthropic, Google, Mistral, DeepSeek, Cohere, Groq, Together, Fireworks, AWS Bedrock — through the same routing layer real production traffic uses, and we publish fresh p50, p95, and p99 numbers every six hours. Below is what we've learned after running more than 14 million timed requests across 184 models, and how you can reproduce the numbers yourself with a single API key.

The Hidden Cost of a "Cheap" Endpoint

Let's talk dollars, because that's where latency quietly bleeds budgets. Suppose you're routing 10 million tokens a day through an endpoint that costs $0.15 per million input tokens. Switching to a "cheaper" $0.075 endpoint looks like a 50% saving — until you factor in that the slower model makes your application spend 800ms extra waiting per request. Across 10 million tokens at an average 220 tokens per request, that's roughly 45,000 requests. Multiply that by 0.8 seconds and you're burning 36,000 seconds of wall-clock time, which translates to roughly 10 extra hours of compute spent idle per day on your worker fleet. If you're running 200 CPU-bound workers at $0.04/hour, that's $9.60 a day just from the wait. None of that shows up on the model provider's invoice.

Then there's the concurrency tax. Slow endpoints force you to keep more connections open, more pods warm, more gateway threads alive. We've seen teams quadruple their HPA minimum replicas just to absorb tail latency, which doubles or triples their cloud bill. A 2024 Datadog report found that the median p95 latency for top-tier LLM APIs ranged from 780ms to 4.1 seconds depending on provider and region — a 5x spread for nominally similar workloads. That's not a rounding error. That's architecture-defining.

Worse, latency variance matters as much as the median. A provider with p50 of 600ms and p99 of 1.1s is dramatically easier to engineer around than one with p50 of 500ms and p99 of 6.4s. We saw one popular inference provider ship a "fast" tier that did 380ms at p50 but routinely spiked to 8 seconds during North American business hours. Their marketing page still shows the 380ms number. Their customers' status pages do not.

Live Latency Snapshot — 184 Models, Refreshed Every 6 Hours

This is the table our team checks every morning before standup. It reflects the rolling 24-hour p50/p95 latency in milliseconds, measured from a cold request after a 30-second idle window, sent from us-east-1 to each provider's nearest advertised endpoint. Tokens in / tokens out are kept constant per model tier. Numbers were captured over the seven days preceding publication.

Two things jump out. First, the dedicated inference specialists (Groq, Fireworks) are genuinely faster than the general hyperscalers for open-weight models — sometimes by 2x or more. Second, the "reasoning" tier (o1, deepseek-reasoner) is a different beast entirely. Pushing 5+ seconds at p99 isn't a bug, it's the price of chain-of-thought. If your product can't tolerate that, you need to architect around it with streaming, prefetch, or background generation rather than expecting the model to suddenly get faster.

One more pattern: flash/mini tiers on the hyperscalers are now genuinely competitive with the inference specialists on pure speed. gemini-1.5-flash at 340ms p50 is the fastest production-grade model in the table, and it's 7x cheaper than Groq's flagship. If your workload fits within its context and quality envelope, the latency story is almost impossible to beat.

How We Actually Benchmark (So You Can Reproduce It)

Vendor benchmarks are fiction. Every provider publishes latency numbers measured under ideal conditions: warm pools, single-tenant inference, regional co-location, and a workload that conveniently matches their sweet spot. We don't trust any of it. Our measurement protocol is deliberately adversarial.

Every test runs from a c5.xlarge in us-east-1 hitting the provider's public endpoint over a single TLS connection. We send a fresh request every 45 seconds — long enough for any keep-alive to expire, short enough to avoid being throttled as abuse. Each request uses the OpenAI-compatible chat completions schema with a fixed 500-token system+user prompt and asks for 200 output tokens. We capture time-to-first-byte (TTFB) and full-body latency separately, because for streaming applications TTFB is the only number that matters, and for batch jobs full-body is what hits your SLO.

We run 200 requests per model per run, three runs per day, rotating the time-of-day across morning, afternoon, and overnight to surface peak-hour degradation. We discard the first 10 requests of every run as warmup. We log region, response size, and HTTP status to catch silent retries. The full dataset — over 14 million rows — is open to anyone with an account, and the script below will reproduce a single measurement against any of the 184 models we cover in well under a minute.

Reproduce a Latency Benchmark in 30 Lines of Python

This is the exact script we use internally for ad-hoc spot checks when a provider announces a new model. It hits the unified routing layer at https://global-apis.com/v1, which means you don't need a separate OpenAI key, Anthropic key, Google key, and so on. One key, every provider, PayPal billing if you want it. Swap "openai/gpt-4o-mini" for "anthropic/claude-3-haiku" or "groq/llama-3.1-70b" and the rest of the script stays identical.

import os, time, statistics, json
import urllib.request, urllib.error

ENDPOINT = "https://global-apis.com/v1/chat/completions"
API_KEY  = os.environ["GLOBAL_APIS_KEY"]        # one key, all providers
MODEL    = "openai/gpt-4o-mini"                  # try "groq/llama-3.1-70b" too

payload = {
    "model": MODEL,
    "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user",   "content": "Explain API latency in one paragraph."}
    ],
    "max_tokens": 200,
    "stream": False
}

def call_once():
    req = urllib.request.Request(
        ENDPOINT,
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type":  "application/json"
        }
    )
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=30) as r:
        body = json.loads(r.read())
    return (time.perf_counter() - t0) * 1000, body

# warmup
for _ in range(5):
    try: call_once()
    except Exception as e: print("warmup err:", e); time.sleep(1)

samples, errors = [], 0
for i in range(50):
    try:
        ms, _ = call_once()
        samples.append(ms)
    except urllib.error.HTTPError as e:
        errors += 1
        print(f"req {i} HTTP {e.code}")
    time.sleep(0.3)

samples.sort()
def pct(p): return samples[min(len(samples)-1, int(len(samples)*p))]
print(f"model={MODEL}  n={len(samples)}  errors={errors}")
print(f"p50={pct(0.50):.0f}ms  p95={pct(0.95):.0f}ms  p99={pct(0.99):.0f}ms")
print(f"mean={statistics.mean(samples):.0f}ms  stdev={statistics.pstdev(samples):.0f}ms")

Run it twice, once in the morning and once during US business hours, and you'll see exactly what we see: a meaningful gap between off-peak and peak p99. That gap is what your production SLO needs to budget for. A Node.js equivalent using fetch and a Go version using net/http are both in the Apibenchmarks GitHub repo if Python isn't your thing.

What the Numbers Don't Tell You

Latency tables are seductive because they reduce a chaotic system to a few integers. Don't fall for it. Here are the four things we've learned the hard way that no table can capture.

Streaming changes everything. If you're shipping tokens as they arrive, TTFB matters far more than full-body latency. Gemini-1.5-flash has a 340ms p50 full-body, but its TTFB is around 90ms — your user sees the first word almost instantly. Compare that to o1-mini, where TTFB can exceed 1.8 seconds because the model won't emit anything until the chain-of-thought resolves. Same "latency," radically different UX.

Cold starts are a separate distribution. Every provider has a first-request penalty after idle. We've measured 4–9x slowdowns on cold calls versus warm calls for some hosted endpoints. If you're a SaaS product with bursts of traffic, your p99 is dominated by cold starts, not by steady-state performance.

Region is destiny. Calling a US-hosted model from Singapore will roughly double your latency versus calling it from Virginia. Some providers let you pin to a region; most don't. We publish per-region numbers and the spread is brutal — up to 600ms of pure geography between the fastest and slowest regions for the same model.

Rate limits throttle more than they admit. If your application suddenly starts hitting HTTP 429s during a traffic spike, your "latency" effectively becomes infinity. Several providers enforce token-per-minute limits that only surface once you're past a certain scale. We've started tracking effective throughput per dollar, because a $0.075 model that lets you do 50 RPM is often worse than a $0.15 model that lets you do 5,000 RPM.

Key Insights From 14 Million Requests