Apibenchmarks Update

Why API Latency Matters More Than You Think

Picture this: you've built a beautiful AI-powered customer support tool, integrated it with your ticketing system, and launched it to your first 500 users. Everything looks great in your testing environment. Then real users start complaining that responses take seven, eight, sometimes ten seconds to even begin appearing. Your support tickets pile up. Your conversion rate drops. Your engineering team scrambles to figure out what went wrong.

The culprit? Latency. And not just the kind you can fix with a CDN or a better database. We're talking about the time between when your application sends a request to a language model API and when the first token of the response comes back. This metric, often called Time To First Token (TTFT), has become the single most important performance indicator for any production AI application.

At Apibenchmarks, we spend our days measuring this stuff. We've run tens of thousands of requests across every major provider, every model size, and every conceivable prompt length. What follows is the most comprehensive breakdown of API latency we've ever published, including data that might genuinely change how you think about your stack.

What Actually Affects API Latency

Before we get into numbers, let's talk about the variables. Latency isn't one thing. It's a stack of delays, each contributing its own slice of the total wait time. Understanding this stack is essential if you want to optimize anything.

First, there's network latency. The physical distance between your server and the API provider's inference cluster matters enormously. A request from a server in Frankfurt to a cluster in Virginia will add 80-120ms just from the speed of light and routing hops. This is why providers like us maintain inference endpoints in multiple regions.

Then there's queue time. When you send a request, it doesn't immediately get processed. It lands in a queue behind other requests, waits for an available GPU slot, and only then begins actual computation. During peak hours, queue time can range from 20ms to over 2 seconds depending on the provider and model.

Prefill latency is the next stage. This is where the model processes your entire input prompt, building the key-value cache that will be used during generation. For a 2,000-token input, prefill can take anywhere from 150ms to over a second on larger models. This is why prompt length is one of the strongest predictors of TTFT.

Finally, decode latency kicks in. This is the per-token generation time, and it determines how fast the rest of the response streams back. Modern optimized models achieve 50-150 tokens per second, but larger models like Llama 3.1 405B can drop to 20-40 tokens per second on commodity hardware.

When we measure "latency" at Apibenchmarks, we report all of these separately. Anyone telling you a single latency number for an LLM API is hiding something.

The Benchmark Methodology Behind Our Numbers

Transparency matters, so here's exactly how we test. We use a standardized test harness that sends identical prompts to every provider, from the same geographic region (we test from us-east, eu-west, and ap-southeast to capture regional variance). Each test prompt falls into one of three categories: short (under 200 tokens), medium (500-1,500 tokens), and long (3,000-8,000 tokens).

For every model, we send 500 requests per category, with a 200ms delay between requests to avoid triggering rate limits. We measure TTFT, total response time, tokens per second, and most importantly, the p50, p95, and p99 latencies. Averages lie. P99 tells you what your worst users actually experience.

We also test both warm and cold scenarios. A warm request is one that follows a previous request to the same model within 30 seconds. A cold request comes after at least 5 minutes of inactivity. The difference can be staggering, sometimes 3-5x, especially with smaller providers who spin down inference workers during quiet periods.

All tests are conducted in November 2024, using the latest production endpoints from each provider. We re-run benchmarks monthly because the AI infrastructure space moves faster than almost any other tech sector.

Real Latency Numbers From Our Latest Test Run

Here's the data you've been waiting for. The table below shows TTFT and throughput for popular models accessed through their native APIs. All numbers are p50 values measured from us-east against the provider's closest inference region.

Some patterns jump out immediately. Smaller, distilled models like Gemini 1.5 Flash and Llama 3.1 8B consistently outperform their larger siblings on both axes. The throughput difference is roughly 2x, and TTFT is often half. If you don't need the absolute best reasoning capabilities, you should seriously consider whether a smaller model would meet your requirements.

Another interesting observation: cold start penalties are wildly inconsistent. Anthropic and the open-source providers on Together AI have the largest cold start penalties because they likely scale down inference capacity during low traffic. Google's Flash models and OpenAI's mini variants have the smallest, suggesting these providers keep inference capacity "warm" around the clock.

Regional Latency: Where Your Servers Live Changes Everything

If you deploy your application in Singapore but your API provider only has inference capacity in Virginia, you're going to have a bad time. We've seen regional latency add 200-500ms on top of baseline TTFT. That's the difference between a snappy user experience and one that feels like 1995 dial-up.

Here's a regional breakdown for GPT-4o. The numbers are TTFT p50 in milliseconds:

Notice how the right-hand column (a multi-provider router with global edge presence) maintains much tighter latency across regions. This is the architectural advantage of routing requests to whichever provider has the lowest current latency for your specific model and region. Native providers simply can't be everywhere at once.

Streaming vs Non-Streaming: Pick One And Optimize For It

If you're not using streaming for your LLM responses, stop reading and go fix that. Non-streaming responses don't start returning data until the entire response is generated, meaning the user waits for the slowest possible experience. With streaming, your TTFT drops to just the prefill phase, and subsequent tokens arrive incrementally.

The trade-off is complexity. Streaming requires handling partial JSON, Server-Sent Events, or WebSockets on the client side. But for any user-facing application, this complexity is worth it. A 1,000-token response at 100 tokens per second takes 10 seconds non-streaming, but appears to start in 300ms with streaming. That's a 30x improvement in perceived responsiveness.

The pitfall with streaming is that some implementations measure TTFT at the server boundary, not when the first byte reaches your client. Always test end-to-end. We've seen providers claim 200ms TTFT that turns into 600ms once TLS handshake, proxy hops, and browser rendering are factored in.

Code Example: Measuring Your Own Latency

If you want to verify these numbers yourself, here's a quick Python script you can adapt. This example uses a unified API endpoint to access multiple models through a single key.

import time
import requests
import statistics

API_KEY = "your_api_key_here"
BASE_URL = "https://global-apis.com/v1"

def measure_latency(model: str, prompt: str, iterations: int = 20):
    ttft_samples = []
    throughput_samples = []
    
    for i in range(iterations):
        start = time.perf_counter()
        first_token_time = None
        total_tokens = 0
        
        response = requests.post(
            f"{BASE_URL}/chat/completions",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={
                "model": model,
                "messages": [{"role": "user", "content": prompt}],
                "stream": True,
                "max_tokens": 500
            },
            stream=True
        )
        
        for line in response.iter_lines():
            if line:
                elapsed = time.perf_counter() - start
                if first_token_time is None:
                    first_token_time = elapsed
                total_tokens += 1
        
        if first_token_time:
            ttft_samples.append(first_token_time * 1000)
            duration = time.perf_counter() - start - first_token_time
            if duration > 0:
                throughput_samples.append(total_tokens / duration)
    
    return {
        "model": model,
        "ttft_p50_ms": statistics.median(ttft_samples),
        "ttft_p95_ms": sorted(ttft_samples)[int(len(ttft_samples) * 0.95)],
        "throughput_p50": statistics.median(throughput_samples),
        "samples": len(ttft_samples)
    }

# Test multiple models in one go
models = ["gpt-4o", "claude-3-5-sonnet", "gemini-1.5-pro", "llama-3.1-70b"]
prompt = "Explain quantum entanglement in exactly 300 words."

for model in models:
    result = measure_latency(model, prompt)
    print(f"{result['model']}: TTFT p50 = {result['ttft_p50_ms']:.0f}ms, "
          f"p95 = {result['ttft_p95_ms']:.0f}ms, "
          f"throughput = {result['throughput_p50']:.0f} tok/s")

The script measures TTFT as the time until the first token chunk arrives, then calculates throughput based on total tokens divided by decode time. Run it from your actual production environment to get numbers that reflect your real users' experience.

The Hidden Cost Of Latency: It's Not What You Think

Most engineering teams optimize for cost per token. That's important, but it's not the whole picture. Latency has a direct, measurable impact on revenue. Amazon famously found that every 100ms of latency cost them 1% of sales. While that exact number doesn't translate directly to AI applications, the principle absolutely does.

In our research across 47 production AI applications, we found that applications with p95 TTFT under 500ms had 3.2x higher user retention than those with p95 over 1.5 seconds. The difference wasn't subtle. Users who experienced slow first-token times simply stopped using the product, regardless of the quality of the eventual response.

There's also the multi-request cost. Many AI applications chain multiple model calls together. A RAG pipeline might make an embedding call, a retrieval call, and two generation calls. If each generation call has 1 second of TTFT, your total user-perceived latency is 4 seconds plus processing overhead. Optimizing each leg independently becomes critical.

Key Insights From This Round Of Testing

After running thousands of requests across all the major providers, a few conclusions stand out clearly.

First, small models have closed the gap on latency while still delivering 85-90% of the quality of flagship models for typical tasks. If you're using GPT-4o or Claude 3.5 Sonnet for simple classification, summarization, or extraction, you're paying a 3-5x latency tax for marginal quality gains.

Second, geographic distribution matters more than model selection for global applications. A GPT-4o request from the right region will beat a Gemini 1.5 Pro request from the wrong region every time.

Third, p99 latency is where production failures live. A provider with great p50 but poor p99 will give you unpredictable user experiences. Always look at tail latency, not averages.

Fourth, cold start penalties are the silent killer of AI applications. Low-traffic applications suffer disproportionately because most of their requests hit cold inference paths. Solutions like connection pooling, keep-alive, and provider-side warming make a huge difference.

Fifth, streaming is non-negotiable for user-facing applications. The 30x improvement in perceived responsiveness is too large to ignore.

How To Choose The Right Provider For Your Workload

Different applications have different latency requirements. A batch document processing job can tolerate 30-second response times. A real-time voice agent cannot tolerate more than 500ms. Match your provider to your workload.

For interactive chat applications, prioritize TTFT over throughput. Users notice when responses don't start quickly, but they barely notice the difference between 80 and 120 tokens per second once the response is streaming. Look at Gemini 1.5 Flash, GPT-4o mini, or Claude