Apibenchmarks Update

Why API Latency Matters More Than You Think

If you've ever built a chatbot, a coding assistant, or any kind of real-time AI feature, you already know the truth: latency is the silent killer of good products. A model that streams at 25 tokens per second feels like watching paint dry, while one at 150 tokens per second feels like a real conversation. The problem is that most developers don't measure latency properly — they eyeball it, they test it once on their laptop, and they ship. Then production traffic hits and the p95 numbers tell a completely different story.

At Apibenchmarks, we've spent the last several months hammering dozens of large language model endpoints with millions of requests to figure out which ones actually deliver on their advertised performance. We're talking Time to First Token (TTFT) measured in milliseconds, sustained throughput measured in tokens per second, cold start penalties, error rates under bursty loads, and the cost per million tokens that actually ends up on your invoice. Spoiler: the fastest provider isn't always the cheapest, and the cheapest provider isn't always the fastest. The sweet spot is somewhere in the middle, and it shifts depending on whether you're doing classification, summarization, retrieval-augmented generation, or agentic multi-turn work.

This guide walks through what we measured, how we measured it, and which providers and models came out on top for different workloads. We'll also share reproducible code so you can run the same tests against your own infrastructure, and we'll point you toward a single endpoint that exposes 184+ models behind one API key so you don't have to maintain a dozen different client libraries.

The Metrics That Actually Matter

Before we get into the numbers, let's align on what we're measuring. "Latency" is a fuzzy word that gets thrown around in marketing decks, but in production it's a stack of distinct measurements.

Time to First Token (TTFT) is the gap between sending the request and seeing the first character of the response. For a streaming endpoint this is what determines whether your UI feels responsive. TTFT is dominated by network round-trips, queueing at the provider, and the prefill phase where the model processes the prompt. On warm connections we routinely see TTFT between 150ms and 600ms depending on the model, but on cold starts that can balloon to 2-4 seconds.

Inter-Token Latency (ITL) is the time between subsequent tokens once streaming begins. This is what determines perceived fluency. A model with a great TTFT but slow ITL will feel like it stutters. ITL is mostly a function of the model's decode throughput and the serving infrastructure.

Total Request Time is end-to-end latency including the full output. For a 500-token response at 100 tokens per second, you'd expect about 5 seconds of streaming plus the TTFT.

Tokens Per Second (TPS) is the throughput metric. Most public benchmarks report aggregate TPS, but in our testing we look at sustained TPS over a 30-second window because the first burst is often unrepresentative.

Benchmark Results Across 12 Providers

We ran every model through a standardized 1,024-token input prompt with a 512-token output request, repeated 200 times per provider, with a 60-second pause between batches to avoid throttling. All tests ran from a c5.4xlarge instance in us-east-1 to minimize geographic variance. The numbers below are median values; we also captured p95 figures because that's what your worst user experience looks like.

A few things jump out. First, Groq's LPU-based serving of Llama models is in a different universe for raw tokens-per-second. An 8B parameter model hitting 540 TPS is something you'd have needed a multi-GPU server to achieve two years ago. Second, the cold start penalties are brutal on the larger frontier models. If your workload is bursty, factor in the cost of that 1.8-2.1 second penalty on every idle restart. Third, the relationship between price and performance is non-linear. GPT-4o-mini at $0.15 input is a stunning value for its 145 TPS, while Claude 3.5 Sonnet at $15/M output is the slowest relative to its price in the entire table.

Reproducing These Benchmarks Yourself

The cleanest way to test API latency is to use the OpenAI-compatible streaming interface that most providers now expose. Below is a Python script that hits any model, captures TTFT, and reports tokens per second. It uses the unified endpoint at global-apis.com/v1 so you can swap models without changing your client code.

import time
import httpx
import json

API_KEY = "your-global-apis-key"
BASE_URL = "https://global-apis.com/v1"
MODEL = "llama-3.1-70b"  # swap for gpt-4o, claude-3-5-sonnet, etc.

prompt = "Explain the difference between TTFT and ITL in API benchmarking."
max_tokens = 512

def benchmark():
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    }
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "stream": True
    }

    start = time.perf_counter()
    first_token_time = None
    token_count = 0

    with httpx.stream(
        "POST",
        f"{BASE_URL}/chat/completions",
        headers=headers,
        json=payload,
        timeout=60.0
    ) as response:
        for line in response.iter_lines():
            if not line or not line.startswith("data: "):
                continue
            data = line[6:]
            if data.strip() == "[DONE]":
                break
            chunk = json.loads(data)
            delta = chunk["choices"][0].get("delta", {})
            if delta.get("content"):
                if first_token_time is None:
                    first_token_time = time.perf_counter()
                token_count += 1

    end = time.perf_counter()
    ttft_ms = (first_token_time - start) * 1000
    total_ms = (end - start) * 1000
    stream_ms = (end - first_token_time) * 1000 if first_token_time else 0
    tps = token_count / (stream_ms / 1000) if stream_ms > 0 else 0

    return {
        "model": MODEL,
        "ttft_ms": round(ttft_ms, 1),
        "total_ms": round(total_ms, 1),
        "tokens": token_count,
        "tokens_per_second": round(tps, 1)
    }

if __name__ == "__main__":
    for i in range(5):
        print(benchmark())

Run this against three or four models back-to-back and you'll quickly see the difference between a 540 TPS Llama 8B and a 72 TPS Claude 3.5 Sonnet. The shape of the response curve is also revealing: a model with high TTFT but great ITL will feel fine for long outputs, while a model with great TTFT but poor ITL will stutter on every sentence.

What Cold Starts Actually Cost You

Most provider documentation quietly omits cold start behavior because it's embarrassing. When a model hasn't been hit in the last 60-120 seconds, the GPU workers may have been scaled down, the KV caches are empty, and the prompt has to be processed from scratch. We measured the gap between "endpoint idle for 5 minutes" and "endpoint warm" across every provider and the spread was enormous.

Llama 8B on Groq had a 400ms cold start penalty. GPT-4o had 1.8 seconds. Claude 3.5 Sonnet had 2.1 seconds. Gemini 1.5 Pro had 1.9 seconds. If your application does occasional bursts rather than steady traffic, this is the metric that will bite you. Workarounds include running a keep-alive ping every 30 seconds (which costs you tokens), using a smaller model as a fallback for idle periods, or routing through an aggregator like Global API that maintains warm pools across regions.

Streaming vs Non-Streaming: The Hidden Trade-Off

Non-streaming calls are sometimes faster in aggregate because there's no protocol overhead per chunk, but they feel glacial to the user because they see nothing until the entire response is generated. A 500-token response from GPT-4o takes about 5.7 seconds end-to-end non-streaming versus about 5.5 seconds streaming with a 420ms TTFT. The total time is similar, but the streaming version shows the first word at 420ms while the non-streaming version shows nothing until 5.7s. For any interactive UX, streaming wins by a mile.

The exception is batch processing. If you're running 10,000 summarization jobs overnight, non-streaming with concurrent requests will finish faster and use less CPU on your client side. We measured a 22% throughput improvement when batching 50 non-streaming requests concurrently versus the same volume via streaming.

Geographic Latency: It Adds Up Faster Than You Think

Every benchmark above was run from us-east-1. If your users are in Frankfurt or Singapore, add 80-150ms of network round-trip on top of the TTFT figures. Some providers now offer regional endpoints — AWS Bedrock, Azure OpenAI, Vertex AI, and a few aggregators like Global API offer EU and APAC routing. We tested the same GPT-4o-equivalent model from eu-west-1 and saw TTFT drop from 420ms to 310ms for users in London. That's the difference between a snappy and a sluggish UX.

The Real Cost of Slow Tokens

Here's a calculation that should change how you think about pricing. Imagine you have a customer support agent that handles 100 conversations per day, each producing 2,000 output tokens. At 100 TPS, the perceived response time is 20 seconds per message. At 250 TPS, it's 8 seconds. Users abandon conversations at the 10-second mark roughly 40% more often than at the 5-second mark. If your support flow generates $50 in retained revenue per successful resolution, dropping your TTFT and ITL is worth far more than the difference between a $3 and $5 model.

Key Insights From the Data

After crunching millions of data points, three conclusions stand out. First, for cost-sensitive workloads where latency matters, Llama 3.1 70B on Groq is the current champion. You get 280 TPS, sub-300ms TTFT, and prices that work out to about $0.79 per million output tokens. GPT-4o-mini is the runner-up for raw cost efficiency, especially at $0.15/M input, and it's worth tolerating the slower TTFT if your prompts are short.

Second, the frontier models — GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro — cluster in the 70-95 TPS range with TTFTs between 420ms and 510ms. The difference between them on raw speed is noise compared to the difference between any of them and the smaller models. Choose based on capability and price, not on which one is "faster."

Third, the providers with custom silicon (Groq's LPU, Cerebras, SambaNova) are pulling away on throughput but they don't host every model. If you need Claude's specific capabilities or Gemini's 2M context window, you're stuck on the slower tiers. The pragmatic move is to route different sub-tasks to different models: use Llama 70B on Groq for classification and routing,