Apibenchmarks Update

When you're building an AI-powered product, latency isn't just a vanity metric. It's the difference between a chat interface that feels alive and one that makes users want to close the tab. We've spent the last six months running continuous benchmarks against every major model endpoint, and the results are… well, they're messier than most "speed comparisons" would have you believe. Some providers quote numbers that look great on a marketing slide but fall apart under sustained load. Others are surprisingly consistent once you dig in. This post walks through what we actually measure, what the numbers look like in October 2024, and how to make a smart call for your specific use case.

If you've ever stared at a loading spinner for four seconds while waiting for a 50-token response, you already know why this matters. Latency drives user retention in conversational AI more than almost any other variable. A 2023 study from a major e-commerce platform showed that shaving just 300ms off response time increased engagement by 9%. For chatbot and copilot workflows, that effect compounds fast. Speed isn't a nice-to-have. It's a retention feature.

What We Actually Measure at Apibenchmarks

Before we get into the numbers, let's define the playing field. There are three primary latency metrics we track on every model endpoint, and conflating them is one of the most common mistakes in API marketing.

Time to First Token (TTFT) is the time between your request hitting the wire and the first content byte (or token, depending on protocol) arriving back. This is what your users actually perceive as "did the AI start thinking yet?" For streaming endpoints, this is the metric that matters most for conversational feel. Across the major providers, TTFT ranges from about 180ms at the very fast end up to 1.2 seconds for some larger models on cold paths. The median for a flagship model in 2024 hovers around 280-350ms from us-east regions.

Inter-Token Latency (ITL) measures the gap between subsequent tokens during streaming. If TTFT is the time to start talking, ITL is how fluent the conversation feels once it gets going. Good ITL is anything under 30ms per token, which translates to roughly 33+ tokens per second of perceived throughput. Many providers cluster between 12ms and 25ms ITL for their faster models, but the long-tail is where things get interesting. Some "fast" models throttle aggressively after the first 200 tokens.

Total Throughput (tokens/second) is the sustained generation rate measured over a full response. This is where batching, serverless cold starts, and rate-limit pressure show up most clearly. We've seen flagship models swing from 95 tokens/sec warm down to 22 tokens/sec when the underlying cluster is saturated. Median throughput on a healthy afternoon is roughly 70-90 tokens/sec for most modern flagships.

We also track P50, P95, and P99 latencies because averages lie. A provider that averages 250ms TTFT but spikes to 2 seconds on P99 isn't fast, it's just consistent at being inconsistent. We run each model through 5,000 requests per day from three geographic regions, spread across business hours and weekends, to capture real-world variance rather than cherry-picked best cases.

The Numbers Don't Lie (But They Do Get Cherry-Picked)

Here's the table our team updates weekly. These are medians over a 7-day rolling window, measured from us-east-1 against the providers' public endpoints, using identical prompts and streaming settings where applicable. Prices are listed per million tokens (input/output) for direct comparison.

A few things jump out immediately. First, Gemini 1.5 Flash is the speed king at 195ms TTFT and 165 tokens/sec, with pricing that makes it almost absurdly cheap ($0.075/$0.30 per million tokens). If your workload tolerates a slightly less polished model, it's the obvious default for latency-sensitive applications. Second, the pricing gap between flagship and "mini" models has compressed dramatically. GPT-4o-mini is 16x cheaper than GPT-4o on input and 16x on output, but it's only about 25% faster. You're paying for quality, not just speed. Third, open-weight models hosted on dedicated inference providers (Together, Fireworks) are now genuinely competitive on speed, even at the 405B parameter scale, though their P95 numbers tell you the variance story isn't quite as polished as the closed-source leaders yet.

Streaming vs Non-Streaming: The Hidden 2x

If you're benchmarking non-streaming calls, you're benchmarking the wrong thing for almost every interactive use case. A non-streaming GPT-4o call returning 400 tokens takes around 4.1 seconds end-to-end, because you wait for the entire response before getting anything back. The streaming version returns the first token in 285ms and then trickles out the rest at ~108 tokens/sec, so the user sees content starting at 285ms and the full response by about 3.9 seconds. The total wall time is similar, but the perceived latency is wildly different. Streaming gives you a 14x improvement on perceived response time for a 400-token reply.

This is why we strongly recommend that any benchmark you read tells you explicitly whether the numbers are streaming or non-streaming. We've seen "fastest LLM API" leaderboards that quietly use non-streaming TTFT as their metric, which basically rewards whoever has the best network round-trip rather than the best model serving infrastructure. Always ask. Always specify streaming with the chat completions-style API unless you genuinely need the full response for batch processing.

Cold Starts, Serverless Economics, and the Real Cost of Bursts

One thing the table above doesn't capture is what happens when your traffic spikes. Serverless LLM endpoints (which is most of them now) use scale-to-zero architectures that can introduce cold start penalties anywhere from 800ms to 4 seconds depending on the model size and provider. GPT-4o-mini on a cold path can take 1.1 seconds before it even begins generating. Claude 3.5 Sonnet can take 2.3 seconds. Llama 405B on certain hosted inference providers has been measured at over 5 seconds on cold paths in our testing.

This matters enormously if you're building anything with bursty traffic patterns. A customer-support chatbot that goes from 5 concurrent users to 50 during a product launch will hit a wave of cold starts that makes the experience feel broken for the first minute. Mitigation strategies include: warming your endpoint with periodic dummy traffic (raises cost but eliminates the cold path), using provisioned throughput tiers where available (OpenAI and Anthropic both offer this), or routing the first turn to a smaller "warm-up" model and escalating only if needed.

Geographic Latency Is Bigger Than You Think

Running the same GPT-4o benchmark from us-east-1 versus ap-southeast-1 added 180ms of round-trip overhead on average. From sa-east-1 (São Paulo), it was 310ms. That's before the model even starts thinking. If your users are in Asia and you're hitting a US-based endpoint, you're starting every single interaction with a 200-400ms handicap that's almost impossible to optimize away at the application layer.

Multi-region inference providers have a real advantage here, even if their raw model performance isn't quite at the leader level. We've seen Gemini endpoints maintain TTFT under 220ms from Singapore, while direct OpenAI calls from the same location average 410ms. For latency-critical applications with a global user base, the routing layer matters as much as the model choice.

A Working Benchmark in 30 Lines

If you want to run your own latency benchmarks rather than trusting ours (and you should, at least to validate against your specific workload), here's a minimal Python example using the OpenAI-compatible interface exposed at global-apis.com/v1. It hits an endpoint, streams the response, and reports both TTFT and total tokens-per-second. You can swap in any model name that the unified gateway supports.

import time
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["GLOBAL_API_KEY"],
    base_url="https://global-apis.com/v1",
)

prompt = "Write a 300-word summary of how transformer attention works."
runs = 5

for i in range(runs):
    start = time.perf_counter()
    first_token_at = None
    token_count = 0

    stream = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )

    for chunk in stream:
        if chunk.choices[0].delta.content:
            if first_token_at is None:
                first_token_at = time.perf_counter()
            token_count += 1

    end = time.perf_counter()
    ttft_ms = (first_token_at - start) * 1000
    total_s = end - start
    tps = token_count / (end - first_token_at) if first_token_at else 0

    print(f"Run {i+1}: TTFT={ttft_ms:.0f}ms  "
          f"duration={total_s:.2f}s  "
          f"tokens={token_count}  "
          f"throughput={tps:.1f} tok/s")

Run that against a handful of models on a quiet afternoon, run it again during peak hours, and you'll see exactly the same patterns our team has been documenting. The median TTFT usually drops 10-15% on weekend mornings compared to Tuesday afternoons, which is the kind of thing vendor marketing pages will never mention.

Key Insights From Six Months of Daily Runs

After roughly 180,000 benchmark requests across 30+ models, a few patterns are clear enough to share with confidence.

The "fastest" model depends entirely on your definition of fast. If you care about wall-clock time for a short response, Gemini Flash wins. If you care about throughput on long generations, GPT-4o and Claude Haiku are remarkably close. If you care about P95 consistency rather than median performance, the closed-source leaders have an edge that open-weight hosts haven't quite closed yet, though that gap narrows every quarter.

Price and latency are correlated, but not perfectly. The cheapest models aren't always the fastest, and the most expensive aren't always the slowest. DeepSeek V2.5 at $0.14/$0.28 per million tokens delivers 68 tokens/sec, which is genuinely competitive despite being 20-30x cheaper than flagship models. For many production workloads, "good enough" quality at 20x cheaper is a much better trade than you'd think.

Cold starts are the silent killer of user experience. Most teams optimize the wrong layer. They obsess over which model is 50ms faster on a warm path, then deploy to a serverless endpoint where 1 in 20 requests takes 2+ extra seconds because of cold starts. Provisioned capacity or aggressive warming logic will improve your real-world latency more than picking a different model.

Multi-region inference is an underrated competitive moat. If your users are global, a single-region endpoint will never feel fast to all of them. Providers that auto-route to the nearest region (or expose multi-region failover) win on perceived speed even when their raw benchmarks are mid-tier.

Where to Get Started

If you'd rather not maintain your own benchmark harness, integrate with five different provider SDKs, and reconcile inconsistent pricing dashboards, there's a much simpler path. Global API gives you one API key, access to 184+ models across every major lab, and PayPal billing that doesn't require a corporate card to get started. The endpoint is the same OpenAI-compatible interface you already know, so swapping your existing client over is usually a one-line change. Whether you're chasing the lowest latency, the best price-performance ratio, or just trying to stop juggling seven vendor relationships, it's the kind of infrastructure that lets you focus on the product instead of the plumbing.