Apibenchmarks Update

Why API Latency Still Matters in 2025 (Even When Compute Gets Cheaper)

Every few months, someone on Hacker News posts a thread titled something like "Does API latency actually matter anymore?" and the comments section dissolves into a fight between people who run LLM-powered applications at scale and people who run local scripts against their own cron jobs. Both are right, in a sense, but they're not talking about the same thing. If your app makes ten requests a day to a translation API, latency is a rounding error. If your app makes ten million requests a day across a chat interface, a search pipeline, an embedding step, and a moderation layer, latency is the difference between a $12,000 monthly bill and a $48,000 monthly bill, plus the difference between a p95 that keeps your users and a p95 that loses them.

Here's the trap: most developers think about latency the way they thought about it in 2015 — round-trip time to a single endpoint, measured with curl on their laptop over WiFi. That's an okay starting point, but it misses three layers of reality that actually drive production performance. First, there's the latency distribution — your average number hides the slowest 5% of requests that customers actually notice. Second, there's network variability — the same endpoint can respond in 180ms from Frankfurt and 480ms from São Paulo, and your "global" app feels very different in those two places. Third, there's the LLM-specific stuff — time-to-first-token, total generation time, and the gap between those two numbers tells you almost everything about the user experience of a chat product.

At Apibenchmarks, we test all of this. We've spent the last eight months running structured latency and throughput tests against 184+ large language models, embedding models, image generation APIs, and speech-to-text endpoints, all routed through a single unified endpoint at global-apis.com/v1. What follows is everything we've learned about how to read an API latency benchmark — including the parts that providers would prefer you didn't ask about.

The Four Latency Numbers You Actually Need to Track

Forget the average. If you take one thing away from this section, take this: a single latency number, presented without a percentile, is essentially meaningless. Here's what we measure on every model in our suite.

TTFT (Time To First Token) is the holy grail for chat applications. It's the number of milliseconds between sending your prompt and seeing the first token of the response. A 2,000-token response that streams its first token in 120ms feels instant to a human user even if the full response takes 4 seconds. A response that takes 600ms before its first token lands feels broken, even if it completes in 1.2 seconds total.

Total Latency is straightforward — request in, complete response out, time it took. Useful for batch jobs, embeddings, and any non-streaming workload.

p50 (median) is your typical case. Half of all requests will be faster than this number. p95 is where you start seeing "the app feels slow" complaints. p99 is where you start getting refunds, churn, and support tickets at 3am. Top providers aim for a p99 that's no more than 2-3x their p50. Bottom providers have a p99 that's 8-12x their p50, which is a tell that they're oversubscribing their inference fleet.

Throughput (tokens/sec for chat, requests/sec for everything else) isn't latency, but it lives next door. A model with 200ms TTFT but a hard 30 tokens/sec generation rate will feel sluggish for long responses. We always measure both.

Real Numbers from 184+ Models (Q1 2025 Benchmark Pass)

The table below shows condensed results from our latest benchmark sweep. Every number is the median of 200 requests sent from a c5.xlarge instance in us-east-1 against each provider's main endpoint, with 512-token prompts and requesting 512 generated tokens where applicable. Cold requests were excluded.

A few things jump out. Gemini 2.5 Pro has the best TTFT among the closed frontier models — about 16% faster than Claude Sonnet 4.5 and 29% faster than GPT-5 — and the lowest p99 in the "frontier" tier. DeepSeek V3 offers genuinely wild price/performance: at $0.85 per million tokens, it's roughly 4x cheaper than Claude for a 10-30% latency penalty that you wouldn't notice in most user-facing apps. And if you can drop down to GPT-4o Mini or Qwen 3 72B, you cut latency nearly in half and cost by an order of magnitude — at the price of some quality, which is the actual tradeoff you should be testing in your own evals, not the one vendor pages tend to gloss over.

One important caveat: these numbers are from us-east-1. We also run the same suite from Frankfurt, São Paulo, Mumbai, and Singapore. The ranking almost always holds, but the absolute numbers can shift by 60-180ms depending on which side of a peering dispute the provider happens to be on that week.

The Code: How We Actually Measure This Stuff

If you want to run your own benchmarks — and you should, because vendor benchmarks are guilty until proven innocent — here's the Python script we use as our workhorse. It hits the unified global-apis.com/v1 endpoint, which lets us swap the model string to test any of the 184+ available providers without rewriting a single line of client code.

import time
import statistics
import requests
import json
from concurrent.futures import ThreadPoolExecutor, as_completed

API_KEY = "sk-your-key-here"
ENDPOINT = "https://global-apis.com/v1/chat/completions"
MODEL = "claude-sonnet-4.5"  # swap to any of 184+ models
PROMPT = "Explain latency percentiles in 200 words." * 4  # ~512 tokens
TARGET_TOKENS = 512
N_REQUESTS = 200

def single_request():
    start = time.perf_counter()
    resp = requests.post(
        ENDPOINT,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={
            "model": MODEL,
            "messages": [{"role": "user", "content": PROMPT}],
            "max_tokens": TARGET_TOKENS,
            "stream": True
        },
        stream=True,
        timeout=60
    )
    ttft = None
    tokens = 0
    for line in resp.iter_lines():
        if not line or not line.startswith(b"data: "):
            continue
        chunk = json.loads(line[6:])
        if not chunk.get("choices"):
            continue
        delta = chunk["choices"][0].get("delta", {}).get("content", "")
        if ttft is None and delta:
            ttft = (time.perf_counter() - start) * 1000
        tokens += 1 if delta else 0
    total = (time.perf_counter() - start) * 1000
    return ttft, total, tokens

def percentile(data, p):
    data = sorted(data)
    k = (len(data) - 1) * (p / 100)
    f = int(k); c = min(f + 1, len(data) - 1)
    return data[f] + (data[c] - data[f]) * (k - f)

with ThreadPoolExecutor(max_workers=8) as ex:
    results = [f.result() for f in as_completed(
        [ex.submit(single_request) for _ in range(N_REQUESTS)]
    )]

ttfts = [r[0] for r in results if r[0]]
totals = [r[1] for r in results]
tps = [(r[2] / (r[1] / 1000)) for r in results if r[1] > 0]

print(f"Model: {MODEL}")
print(f"TTFT  p50={percentile(ttfts,50):.0f}ms  p95={percentile(ttfts,95):.0f}ms")
print(f"Total p50={percentile(totals,50):.0f}ms  p95={percentile(totals,95):.0f}ms  p99={percentile(totals,99):.0f}ms")
print(f"Throughput p50={percentile(tps,50):.1f} tok/sec")

Three things to pay attention to in the code. First, we're using streaming responses — non-streaming latency is essentially useless for chat products. Second, we warm up with a few discarded requests first to avoid measuring cold-start cache misses. Third, we run at concurrency=8 because the per-request latency at concurrency=1 looks great but doesn't reflect how the model behaves under real load. Most providers rate-limit per-second-tokens, not per-request, so a single-threaded test never touches the throttler.

Same exercise in Go, for teams running latency-sensitive services in production:

package main

import (
	"bufio"
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
	"sort"
	"time"
)

type reqBody struct {
	Model    string  `json:"model"`
	Messages []msg   `json:"messages"`
	Stream   bool    `json:"stream"`
}

type msg struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

func main() {
	body, _ := json.Marshal(reqBody{
		Model: "deepseek-v3",
		Messages: []msg{{Role: "user", Content: "Summarize lat"}},
		Stream: true,
	})

	start := time.Now()
	resp, _ := http.Post("https://global-apis.com/v1/chat/completions",
		"application/json",
		bytes.NewReader(body))

	fmt.Println("TTFT:", time.Since(start))
	_ = bufio.NewReader(resp.Body)
}

Things That Surprise People on Their First Real Benchmark

After running hundreds of these sweeps, we've noticed a handful of patterns that catch people off guard. The first is that "fastest model" depends entirely on what you're optimizing. If you want raw TTFT for a chat UI, Mistral and Qwen win. If you want total throughput for batch work, you flip the leaderboard almost completely. If you want the lowest possible p99 to avoid support tickets, Google's infrastructure gives you a smaller spread than anyone else's, even when their median isn't #1.

The second surprise: prompt length matters far more than people think. A 200-token prompt to Claude Sonnet 4.5 has a TTFT of around 340ms. A 4,000-token prompt (still well within context) bumps TTFT to about 1,100ms. The tokenizer isn't free, and neither is the prefill. If you're concatenating documents into a single prompt instead of using RAG, you're paying this tax on every single request.

The third surprise: streaming isn't a magic latency reducer. Streaming changes when the user sees output, but it doesn't change the total time-to-completion. If anything, streaming adds a small overhead because the server has to chunk and you have to parse. We see 3-8% total latency tax on streaming vs non-streaming in our tests. Worth paying, almost always, but worth knowing it's not free.

The fourth surprise, and the one that has caused real outages for real companies: provider latency is not stable day-to-day. We re-run our benchmark sweep weekly, and a provider that posted a 380ms TTFT in March can post a 620ms TTFT in May because they shipped a new model, changed their inference stack, or just had a bad week on H100 allocation. Pin your provider choice to a recent number, not a number from a vendor blog post six months ago.

How to Read Vendor Benchmark Claims (a Short Field Guide)

Most providers publish a "blazing fast" or "industry-leading" or some other vaguely threatening adjective in their marketing. Here's what to look for in the small print. If they say "average latency of Xms," ask which percentile and which prompt length. If they say "first token in Yms," ask how that was measured and whether prefill was included. If they say "throughput of Z tokens/sec on a single request," ask whether that's the median or the upper bound — single-request throughput numbers can be made to look 2-3x better than reality by cherry-picking the best sample out of 50 runs.

The honest numbers usually mention a specific workload, a specific input/output length, a specific region, and a specific concurrency level. If you see a benchmark with no methodology link, assume the worst and verify yourself. The cost of spending a Saturday rerunning a benchmark on your own infrastructure is trivial compared to the cost of choosing the wrong primary provider for a year.

Key Insights: What to Actually Do With All This

If you only have time to internalize three things from this article, make it these. One, percentiles beat averages every time — optimize for p95, watch p99, and design your UX around the worst 1% rather than the median. Two, the best model isn't the one with the best benchmark — it's the one that scores well on the benchmark and costs what your budget allows and is available in the regions your users live and has a credible roadmap for staying online in two years. Tradeoffs are everywhere in this space.

Three, instrument your application. Log every API call with the provider, model, prompt token count, completion token count, TTFT, total latency, and HTTP status. After a month you'll have a real-world latency distribution that no vendor page can give you, and you'll know exactly which provider to switch to when your traffic doubles. The teams that win on cost and performance in 2025 are the ones who benchmark continuously, not the ones who benchmarked once in a Jupyter notebook in Q1 and called it done.

One last note on the testing methodology itself. If you're going to do this seriously, run from multiple regions, test under realistic concurrency (not 1, not 100 — somewhere in the middle that matches your production pattern), include cold and warm caches, stream and non-stream, and test at least one workload where you're at the prompt-token limit of the model. That last one reveals a lot