Apibenchmarks Update

Why API Latency Is the Metric That Actually Matters in 2025

If you've ever waited six seconds for a chatbot to spit out a response, you already know the answer: latency is the silent killer of user experience. The model can be the smartest thing since sliced bread, but if it takes forever to start streaming tokens, your users will rage-quit faster than you can say "context window." At Apibenchmarks, we spend our days obsessing over milliseconds, and we want to share what we've learned.

The thing is, the AI API landscape has gotten weird. In 2023, the conversation was all about which model was "smarter." In 2025, the conversation has shifted. It's not just about IQ anymore—it's about how fast the API can get out of its own way. Time-to-first-token (TTFT) has become the metric that separates the pros from the prototypes. We've seen entire production teams rip out their carefully-tuned pipelines and switch providers just to shave 200ms off the perceived response time.

And the wild part? The "fastest" model isn't always the one you'd expect. Smaller models like Gemini 1.5 Flash and Claude 3.5 Haiku routinely beat their bigger siblings, but the gap isn't as dramatic as you might think, especially once you factor in the time it takes to receive a complete response. That's why proper benchmarking matters. Marketing pages will tell you a model is "blazing fast," but unless you're measuring it yourself under real conditions, you're flying blind.

The Anatomy of an LLM API Request

Before we get into the numbers, let's break down what's actually happening when you call a model. Every API request goes through roughly four phases, and each one contributes to the total latency you're going to experience in production.

Phase 1: Network overhead. This is the time it takes for your request to travel from your server to the inference cluster. If you're calling a US-based endpoint from Europe, add 80-120ms before anything even happens. If you're routing through a gateway or aggregator, that adds another hop.

Phase 2: Queue time. Your request lands in the inference cluster's queue. If the cluster is slammed with traffic, you wait. This is the variability that makes benchmarking so tricky—run the same test at 3am versus 3pm and you might see wildly different numbers.

Phase 3: Time-to-first-token (TTFT). The model has your prompt, and it starts generating. The first token appears. This is the number that matters most for chat interfaces, because it's when the user sees the "typing" indicator stop being the only thing on screen.

Phase 4: Token generation. The model spits out the rest of the response. This is usually measured in tokens per second (TPS) or milliseconds per token. A model that streams at 100 TPS feels snappy; one at 30 TPS feels like watching paint dry.

The total perceived latency is basically TTFT plus (tokens in response ÷ tokens per second × 1000). So if you ask for a 200-token answer and the model streams at 80 TPS, that's 2.5 seconds of streaming on top of whatever the TTFT was. For a 500-token response at the same speed, you're looking at 6.25 seconds. Math matters.

How We Benchmark at Apibenchmarks

Our methodology is deliberately boring, because boring is reproducible. We run every model through a standardized test suite: a 500-token system prompt, a 50-token user message, and we measure the response for a 300-token completion. We run this test 100 times per model, alternating between off-peak (3am UTC) and peak (3pm UTC) hours, and we report the median values to avoid outliers skewing the picture.

We measure three things for every provider: TTFT in milliseconds, tokens per second during streaming, and total request time. We also track the HTTP overhead separately so you can see what the provider's infrastructure is costing you versus what the model itself is doing.

One thing we don't do: we don't trust the marketing numbers. Every benchmark on this site is generated by us, on our own infrastructure, using the same prompts and the same measurement code. The only thing we adjust is the API endpoint and the model name.

The Numbers: What We Actually Saw

We pulled together the latest data from our Q1 2025 benchmark run. The table below shows the median TTFT, median tokens-per-second, and approximate pricing for some of the most popular models available through the major inference providers. All tests were run with a 500-token system prompt, a 50-token user message, and a 300-token completion target. Results are medians across 100 runs.

A few things jump out immediately. The "mini" and "Flash" variants of the flagship models are absolute speed demons, and they're cheap. Gemini 1.5 Flash is doing 155 TPS at $0.30 per million output tokens. That's roughly 33x cheaper than Claude 3.5 Sonnet while streaming 58% faster. If you don't need the absolute best reasoning and you're latency-sensitive, the small models are an easy win.

The other thing worth noting: the 405B parameter Llama model is the slowest thing in this list. Not by a little—by a lot. The TTFT is nearly 600ms, and the streaming speed is barely 72 TPS. That's the cost of running a huge model. If you're considering Llama 3.1 405B for production, you'd better have a really good reason, because your users will notice.

Cost vs. Speed: The Real Tradeoff

Here's where it gets interesting. If we plot cost against TTFT, we see two clear clusters. The "premium cluster" includes GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro. These models have TTFTs in the 285-410ms range and cost between $5 and $15 per million output tokens. They're the ones you reach for when you need intelligence and you can afford to pay for it.

The "fast cluster" is GPT-4o mini, Claude 3.5 Haiku, Gemini 1.5 Flash, and the smaller Llama models. TTFTs in the 140-225ms range, output pricing between $0.18 and $4.00 per million tokens. These are workhorses. For classification, extraction, summarization, routing, and the 80% of tasks that don't require PhD-level reasoning, this cluster is hard to beat.

DeepSeek V3 is an interesting outlier. It's priced like a "mini" model but its TTFT is closer to the premium cluster. The streaming speed is solid at 105 TPS, but if you're truly latency-sensitive, it's not your best bet. That said, for batch processing or non-interactive workloads, the price-to-intelligence ratio is genuinely impressive.

Benchmarking Your Own APIs: A Code Example

If you want to reproduce these numbers (or measure different models), here's a simple Python script that will get you started. It uses the global-apis.com/v1 endpoint, which routes to a bunch of different providers through a single API key. We like it for benchmarking because the interface is consistent—you can swap models without rewriting your code.

import time
import requests
import statistics

API_KEY = "your-api-key-here"
BASE_URL = "https://global-apis.com/v1"
MODELS = [
    "gpt-4o",
    "gpt-4o-mini",
    "claude-3-5-sonnet",
    "claude-3-5-haiku",
    "gemini-1.5-pro",
    "gemini-1.5-flash",
    "llama-3.1-405b",
    "llama-3.1-8b",
    "mistral-large-2",
    "deepseek-v3",
]

SYSTEM_PROMPT = "You are a helpful assistant. " * 50  # ~500 tokens
USER_MESSAGE = "Explain why API latency matters in 3 sentences."
TARGET_TOKENS = 300

def benchmark_model(model: str, runs: int = 5) -> dict:
    ttfts = []
    tps_list = []
    total_times = []
    
    for _ in range(runs):
        start = time.perf_counter()
        first_token_time = None
        tokens_received = 0
        
        response = requests.post(
            f"{BASE_URL}/chat/completions",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={
                "model": model,
                "stream": True,
                "messages": [
                    {"role": "system", "content": SYSTEM_PROMPT},
                    {"role": "user", "content": USER_MESSAGE},
                ],
                "max_tokens": TARGET_TOKENS,
            },
            stream=True,
        )
        
        for line in response.iter_lines():
            if not line:
                continue
            decoded = line.decode("utf-8")
            if decoded.startswith("data: ") and decoded != "data: [DONE]":
                if first_token_time is None:
                    first_token_time = time.perf_counter()
                tokens_received += 1