Apibenchmarks Update

What Actually Happens When You Hit "Send" on an LLM API Call

Most developers don't think about latency until it slaps them in the face. You wire up an OpenAI or Anthropic call in your app, it works fine for your five test prompts, you ship it to production, and then a user on a flaky 4G connection in São Paulo pings your chatbot and waits eleven seconds for the first word to appear. Your CTO pings you on Slack. You stare at the metrics dashboard. Nobody feels good.

This is why Apibenchmarks exists. We measure the things that actually matter when you're shipping AI features: how fast does the first token arrive? How quickly do the rest stream in? What does it cost to keep the lights on while doing it? Because the gap between a model that "feels smart" in a chat window and a model that "feels fast" in a customer-facing product is enormous, and that gap is usually where AI apps go to die.

In this piece we're going to walk through the real benchmark numbers across the top providers right now, break down what TTFT, ITL, and end-to-end latency actually mean in production terms, show you how to measure them yourself with about ten lines of code, and point you at the cheapest way to actually run all of this at scale.

The Four Numbers That Matter (And Why They're Not Interchangeable)

Before we get into the tables, let's agree on vocabulary, because the LLM API world loves to muddy this. There are really four distinct latency measurements that each tell you something different:

TTFT (Time to First Token) — This is the delay between sending your request and seeing the first character stream back. Measured in milliseconds, lower is better, and this is the single biggest driver of how "snappy" your app feels. A 200ms TTFT feels instant. An 1800ms TTFT feels broken even if the rest of the response is fast.

ITL (Inter-Token Latency) — Once the first token lands, how fast do the rest come? This is usually expressed in milliseconds-per-token (ms/tok) and translates directly to tokens-per-second throughput. A 25ms ITL means roughly 40 tokens/sec. A 60ms ITL means roughly 17 tokens/sec, and your user is watching words appear like a slot machine.

E2E Latency (End-to-End) — Total wall-clock time for a complete response of N tokens. Roughly equal to TTFT + (ITL × tokens). Useful for back-end workloads where you don't stream to a user, but increasingly rare as a priority metric in 2025.

Cold Start Penalty — Many providers (especially hosted and serverless ones) slow down dramatically on the first request after a quiet period. We've seen cold starts push TTFT from 300ms to over 5 seconds. If you're an intermittent-traffic app, this can dominate your user experience.

Different workloads optimize for different things. A chatbot streaming to a phone needs TTFT above all. A batch summarization job needs aggregate throughput. A code-completion box needs both — a developer typing expects sub-200ms TTFT or they'll think the keyboard is broken.

The Numbers: Real Benchmark Results From February 2026

Below is a snapshot of measured performance across the most-requested models in our dataset. Numbers are median values from a 5,000-prompt sweep per model, using 512-token outputs, run from a US-East client. Costs reflect the public input/output token price at the time of testing. All providers were hit at the same time of day (Tuesday 2pm UTC) over the course of three weeks to smooth out transient load.

A few things jump out immediately. The "mini" and "flash" tier models are now genuinely production-grade for latency-sensitive apps. Gemini 1.5 Flash at 160ms TTFT is basically indistinguishable from a database query in human perception terms, and it's sixty cents per million output tokens. If you're not using it for your interactive surface, you're paying for latency you don't need.

Also notice DeepSeek V3 — fascinating combination of expensive cold starts (520ms+ TTFT on first hit) and absurd pricing. At fourteen cents per million output tokens, it's basically a rounding error. If you can warm-pool it or pre-warm via a heartbeat, the cost-per-request ratio is unbeatable for back-end workloads.

The Hidden Cost Of Provider Fragmentation

Here's something the benchmark tables don't show: integrating six different providers into one product is genuinely painful. Different SDKs, different streaming protocols, different auth schemes, different billing portals, different rate limit error codes, different "object too large" thresholds. We've personally watched a five-person engineering team spend two months just building a router to switch between providers for cost-fallback reasons. That's not a feature, that's debt.

This is the real reason we lean so heavily on unified gateways in our recommendations. Routing in your application code is fine if you have one use case and one provider. Routing as a layer in your stack is a different conversation. Once you need to A/B test GPT-4o against Claude for quality reasons, or fall back from a rate-limited Mistral to a cheaper Llama, or measure latency across regions, you're going to want a single chokepoint.

The biggest hidden cost, though, is the time your engineers spend context-switching. Every time the team has to learn a new SDK, debug a new error format, or reconcile a new invoice, that's hours not spent building product. We've seen engineering orgs save 20-30% of their total AI-related dev time by consolidating to a single API surface, even before accounting for the latency consolidation benefits.

Region Matters More Than Vendor

One of the most consistent patterns in our data is that geographic distance from the inference pod dominates vendor differences for TTFT. A GPT-4o call from London to a US-East endpoint will lose roughly 90ms to network round-trip alone. That same call routed to a London replica costs 30ms. Cloud providers have spent the last year aggressively replicating their inference infrastructure into Europe, Asia-Pacific, and the Middle East, and the latency spread between them is now small enough that geography is your main lever.

If your user base is global, you have basically three options: pick a provider with multi-region inference (Anthropic, OpenAI, Google all do), deploy a gateway that routes by user geography, or accept the network tax. Option two is increasingly common, and gateways that handle it well can shave 100-300ms off TTFT for distant users without you writing a line of routing logic.

Measuring It Yourself: The Code You'll Actually Use

Every benchmark table in the world is fine, but you need to measure your own workload on your own users. Here's a minimal Python snippet that hits a model through a unified endpoint and prints the four numbers. Adjust the prompt and the model name as needed.

import time, json, requests

API_KEY = "YOUR_API_KEY"
ENDPOINT = "https://global-apis.com/v1/chat/completions"
MODEL = "gpt-4o-mini"  # any of 184+ models, same key

prompt = "Explain speculative decoding in LLM inference in 400 words."

payload = {
    "model": MODEL,
    "messages": [{"role": "user", "content": prompt}],
    "max_tokens": 512,
    "stream": True,
}

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}

start = time.perf_counter()
first_token_time = None
token_count = 0
full_text = []

with requests.post(ENDPOINT, headers=headers, json=payload, stream=True) as r:
    r.raise_for_status()
    for line in r.iter_lines():
        if not line or not line.startswith(b"data: "):
            continue
        if line == b"data: [DONE]":
            break
        chunk = json.loads(line[6:])
        delta = chunk["choices"][0]["delta"].get("content", "")
        if delta and first_token_time is None:
            first_token_time = time.perf_counter()
        if delta:
            token_count += 1
            full_text.append(delta)

end = time.perf_counter()

ttft_ms = (first_token_time - start) * 1000
itl_ms = (end - first_token_time) / max(token_count - 1, 1) * 1000
tps = token_count / (end - first_token_time)
e2e_s = end - start

print(json.dumps({
    "model": MODEL,
    "ttft_ms": round(ttft_ms, 1),
    "itl_ms": round(itl_ms, 2),
    "tokens_per_sec": round(tps, 1),
    "tokens": token_count,
    "e2e_seconds": round(e2e_s, 2),
}, indent=2))

That's roughly the smallest possible benchmark harness and it gives you everything you need. Run it from your actual production regions, sample 100+ times per model, drop the data into a spreadsheet, and you have a ground-truth view of how the providers behave on your real network path. Anything we publish is a starting point, not a substitute.

Key Insights From Six Months Of Continuous Benchmarking

After running hundreds of thousands of timed requests, a few patterns keep showing up no matter which provider we test, and they're worth baking into your architecture from day one.

First, TTFT variance is huge, and median numbers lie. The 50th percentile is fine for marketing but production SLAs should be built against p95 or p99. We've measured p99 TTFTs that are 4-8x the median for many providers, especially during US business hours. If your chat app freezes for five seconds once in twenty requests, users will assume it's broken, even if the median is 200ms.

Second, streaming changes everything for perceived latency. Without streaming, an 800-token response from GPT-4o takes roughly 24 seconds before the user sees anything. With streaming, they see the first word in under 500ms and read along. This is the single biggest UX lever you have, and not using it is essentially giving away your latency budget.

Third, the cheap models aren't slow anymore. Twelve months ago there was a real tradeoff between cost and speed. That gap has largely closed for TTFT — Gemini Flash, GPT-4o mini, Claude Haiku, and Qwen 2.5 72B are all in the same ballpark. Where they differ is in long-context handling, reasoning depth, and instruction following for complex tasks. Pick on quality for the hard queries, pick on cost for the easy ones.

Fourth, batch endpoints are massively underused. Anthropic, OpenAI, and Google all offer batch APIs at 50% discount with 24-hour turnaround. If your workload is asynchronous — summarize this ticket, classify these 10,000 emails, generate alt text for these images — and you don't need a response in under a minute, batch pricing can cut your inference bill in half with zero code complexity.

Fifth, small models on big hardware beat big models on small hardware. A 7B parameter model on an H100 will outpace a 70B on a CPU for raw throughput. If you're self-hosting, picking a model that fits cleanly in memory on modern accelerators is almost always faster than picking a more capable model that's memory-bandwidth-bound.

What To Actually Ship This Week

If you take one thing from this article, take this: stop choosing models by intelligence benchmarks alone. They correlate loosely with quality and not at all with user-perceived latency. Build a 200-line benchmark harness, run it against the top three providers for your workload, pick the fastest p95, and ship. You can always swap in a smarter model behind the same interface later.

Also: stop paying retail. The provider sticker prices are basically the highest you'll ever pay. Volume discounts exist, batch discounts exist, prompt caching discounts exist, and unified gateways almost always negotiate better rates than you can get direct. If you're spending more than a few thousand a month on inference, you should be on a plan with committed-use pricing.

Finally, instrument from day one. Wrap every model call, log TTFT and ITL to a metrics backend, and graph them. You cannot optimize what you cannot see, and provider performance drifts constantly as they reconfigure their infrastructure. We've seen a model's p95 TTFT double in a week because the provider moved inference to a new region without warning. If you're not measuring, you'll find out from your support inbox.

Where To Get Started

The fastest way to put all of this into practice today is to skip the per-provider integration tax and route every model you care about through a single key and a single billing relationship. That's exactly what Global API was built for — one API key, 184+ models across every major lab, billed in one