Section 01 · Framing
The two metrics you must measure separately
LLM latency optimization means different things depending on which number you are trying to move. Conflating TTFT and total latency is the single most common cause of regressions in production AI systems.
Quick answer
How do you reduce LLM latency in production? The fastest wins come from the application side: enable response streaming to cut perceived TTFT immediately, add semantic caching for repeated queries (25 to 45 percent cache hit rates in FAQ-style patterns), and route simple tasks to smaller models for 40 to 70 percent TTFT reduction on those tasks. Serving-side levers — continuous batching, KV-cache prefix reuse, quantization, speculative decoding — require inference engine control but cut total throughput latency by 50 percent or more.
Time to first token (TTFT) is the elapsed time from request submission to the arrival of the first character of the response stream. For conversational applications and chat interfaces, TTFT dominates perceived responsiveness. A user staring at a blank response box experiences every millisecond of TTFT as a direct wait. Total response time is secondary — once the response starts streaming, users perceive progress and tolerate longer completions. For conversational use cases, optimizing total latency while ignoring TTFT is a mistake.
Total response latency is the elapsed time from request submission to the arrival of the last token. For batch document processing, code generation for a CI pipeline, or any use case where the application waits for the full response before acting on it, total latency is the relevant metric. Streaming provides no benefit when the downstream process requires the complete output. For these use cases, throughput per second and total latency are the numbers to optimize.
For applications that display partial responses — chat, copilots, inline completion — measure both. TTFT p95 above 800 milliseconds typically registers as slow in user perception research. Total latency p95 is workload-specific; set the target from your actual use case, not a generic benchmark. Track them separately before and after every change.
Section 02 · Application-Side
Application-side levers
These four techniques reduce latency without requiring access to or control over the inference engine. They operate above the API boundary and work against any hosted LLM API.
Response streaming
Streaming returns tokens to the client as they are generated rather than waiting for the full completion. The model generates at the same speed either way — the difference is entirely in perceived responsiveness. With streaming enabled, users see the response begin within 200 to 500 milliseconds of a well-routed request, even if the full completion takes eight seconds. Without streaming, the user waits the full eight seconds before seeing anything. Streaming is the highest leverage change for conversational applications and costs almost nothing to implement. It should be the first change on the list for any application that displays responses as they arrive.
Semantic caching
Semantic caching stores previous responses and returns them for semantically similar new queries without calling the model. A new request is embedded as a vector and compared against the cache. If the similarity score exceeds a configured threshold — typically 0.92 to 0.96 using cosine distance — the cached response returns immediately. For a request that would otherwise take two seconds to complete, a cache hit takes under 20 milliseconds. FAQ-style interactions and repeated analytical queries on stable knowledge see cache hit rates of 25 to 45 percent in production. Threshold calibration matters: too high misses valid cache opportunities; too low returns subtly wrong cached responses.
Prompt trimming and token hygiene
Every input token the model processes adds inference time. System prompts that accumulate instructions over time — fixing edge cases, adding examples, documenting behavioral constraints — commonly grow to 2,000 to 5,000 tokens or more. A structured audit removing redundant instructions and obsolete examples typically recovers 10 to 20 percent of input tokens with no behavioral change. For retrieval-augmented generation (RAG) applications, the retrieved context is often the larger contributor. Aggressive retrieval that pulls the top 20 chunks when the top 3 are sufficient adds thousands of input tokens on every request. Setting tighter retrieval budgets and filtering chunks by a minimum similarity score reduces input token count and directly reduces TTFT.
Model routing by task complexity
Not all tasks in an application require the same model. Classification tasks — determining intent, categorizing a request, routing to a handler — are handled reliably by smaller, faster models. Structured extraction tasks — parsing dates, extracting entities, converting natural language to structured parameters — also route well to smaller models. Synthesis, multi-step reasoning, and nuanced judgment calls benefit from frontier models. A routing layer that matches task type to the appropriate model tier cuts TTFT by 40 to 70 percent on the tasks that route to smaller models. A frontier model might take 800 milliseconds TTFT for a request that a smaller model resolves in 150 milliseconds. Routing correctly does not degrade quality on tasks that genuinely do not need frontier capability.
Section 03 · Serving-Side
Serving-side levers
These techniques require control over the inference engine or infrastructure. They are available to teams hosting models with engines like vLLM, SGLang, or TensorRT-LLM, and increasingly as configuration options on managed inference platforms.
Continuous batching. Traditional static batching processes a batch of requests and waits for all of them to complete before accepting the next batch. A single long request blocks shorter ones in the same batch, inflating their latency. Continuous batching (also called iteration-level scheduling) processes each completed sequence out of the batch immediately and fills the freed slot with a waiting request. There is no wait for the slowest request. Continuous batching improves throughput significantly — up to 85 percent lower per-token cost at scale — with its effect on individual request latency depending on queue depth. At high concurrency, it prevents the queuing penalty that static batching imposes on short requests blocked behind long ones.
KV-cache and prefix caching. During autoregressive generation, the model computes key-value attention matrices for every token in the input sequence. These matrices are expensive to compute but deterministic for a given input prefix. KV-cache stores these matrices in GPU memory so they do not need to be recomputed on subsequent tokens in the same sequence. Prefix caching extends KV-cache reuse across requests: if multiple requests share a common prefix — a long system prompt, a fixed knowledge-base document, a recurring user context — the prefix KV-cache is computed once and reused across all requests that share it. For applications with a large static system prompt of 2,000 to 10,000 tokens, prefix caching reduces TTFT substantially. Anthropic and OpenAI both expose prompt caching on their hosted APIs.
FlashAttention and quantization. FlashAttention rewrites the attention computation to use GPU memory bandwidth more efficiently, reducing attention computation time by 2x to 4x on typical sequence lengths. It is now the default in most open-source inference engines. If your inference stack is not using FlashAttention, enabling it is a configuration change. Quantization reduces model weight precision from 16-bit to 8-bit or 4-bit representations, reducing memory footprint and speeding up matrix multiplications. INT8 quantization typically shows less than one percent quality degradation. INT4 shows two to five percent degradation on average, with larger drops on reasoning-heavy tasks. Calibrate on your actual workload before committing to aggressive quantization.
Speculative decoding. Speculative decoding uses a small draft model to propose multiple tokens at once, then uses the full model to verify them in parallel. When the draft model predicts correctly — which it does for common phrases, code patterns, and formulaic text at 60 to 80 percent accuracy — the full model accepts the batch without additional inference steps, effectively generating multiple tokens per step. The net effect on TTFT is modest. Speculative decoding primarily reduces total generation time for longer completions, with speed improvements of 1.5x to 2x common for code and structured output. It requires a compatible draft model and adds infrastructure complexity; evaluate whether the workload justifies it before deploying.
Section 04 · Sequencing
Optimization order of operations
Apply techniques in the order that maximizes impact for the least infrastructure change. Each step compounds the previous.
| Step | Technique | Metric | Typical improvement | Effort |
|---|---|---|---|---|
| 1 | Enable streaming | TTFT (perceived) | 60–80% perceived wait reduction | Hours |
| 2 | Semantic caching | Both | 25–45% cache hit rate | Days |
| 3 | Model routing | TTFT | 40–70% cut on routed tasks | 1–2 weeks |
| 4 | Prompt audit + trim | Both | 10–20% input token reduction | Days |
| 5 | Prefix caching (hosted) | TTFT | Significant for long system prompts | Hours |
| 6 | Continuous batching (engine) | Total / throughput | Up to 85% cost reduction | Weeks |
| 7 | Quantization (INT8) | Total | Larger batches, faster matrix ops | Weeks |
| 8 | Speculative decoding | Total (long outputs) | 1.5–2× for patterned completions | Months |
The ordering matters because of compounding. Streaming comes first because it costs nothing and immediately changes the user experience. Semantic caching comes second because it eliminates entire inference calls — every subsequent technique only needs to apply to the uncached workload. Model routing comes third because it reduces the baseline cost and latency for all remaining calls. Prompt trimming comes fourth because it reduces per-call token count across all non-cached, correctly-routed requests. Serving-side optimizations come last because they require infrastructure access and longer implementation cycles.
For a deeper look at how latency and cost tradeoffs interact across the full inference stack, including the specific cost impact of routing and caching, see the LLM inference cost optimization guide, which covers the same five-lever framework from the cost angle.
Section 05 · Observability
What to measure before you optimize
Optimizing without measurement produces improvements that look good in benchmarks and degrade in production. Two practices prevent this.
Measure TTFT and total latency separately at p95
Average latency hides the tail behavior that degrades user experience. A change that reduces average total latency by 40 percent while increasing TTFT p95 from 600 milliseconds to 1,400 milliseconds is a regression for conversational users, even though the headline number improved. Track both percentiles before and after every change. P50 tells you what a typical user experiences; p95 tells you what your slowest users experience — and slow outliers disproportionately drive churn.
Measure against your own workload, not synthetic benchmarks
Standard LLM benchmarks use fixed sequence lengths and request patterns that rarely match production traffic distributions. Your actual prompt length distribution, request concurrency, and task mix determine whether a technique improves production latency. Run optimization experiments on production traffic samples or realistic load tests before drawing conclusions. A technique that improves latency on 512-token prompts may have no effect on your 4,000-token RAG context workload.
If your production LLM system does not yet have the instrumentation to capture TTFT and total latency p95 per request, address that before implementing latency optimizations. Optimizing without measurement is guesswork. The LLM observability guide covers what to instrument and how to wire it into production systems.
Section 06 · When to Engage
When to bring in an AI systems architect
Most application-side levers can be implemented by a backend engineering team with focused effort. The serving-side layer is a different category.
Choosing and configuring an inference engine, calibrating quantization for a specific model and workload, implementing prefix caching with the right invalidation strategy, and evaluating speculative decoding for a particular task distribution requires infrastructure expertise that most product engineering teams do not carry. The risk is not that the techniques are complex in isolation — it is that applying them incorrectly produces regressions that are difficult to diagnose without deep familiarity with inference engine internals.
If your latency problem requires serving-side work, or if you have implemented the application-side levers and need to understand what is left to gain, the AI Systems Architecture service covers production inference infrastructure as part of an end-to-end architecture engagement. That includes a latency and cost audit, serving-side configuration, and observability wiring — not a generic recommendations report.
FAQ
Frequently asked questions
How do you reduce LLM latency in production?
The fastest wins come from the application side: enable response streaming to cut perceived TTFT immediately, add semantic caching for repeated queries (25 to 45 percent cache hit rates), and route simple tasks to smaller models for 40 to 70 percent TTFT reduction on those tasks. Serving-side levers require inference engine control but cut total throughput latency by 50 percent or more when applied correctly.
What is time to first token (TTFT) and why does it matter?
Time to first token is the elapsed time from request submission to the arrival of the first character of the response stream. For conversational applications, TTFT determines perceived responsiveness more than total response time. A streaming response that begins in 300 milliseconds and takes eight seconds to complete feels fast to most users. Optimizing total latency without measuring TTFT is the most common cause of latency regressions in production chat and copilot applications.
What is continuous batching in LLM inference?
Continuous batching processes completed sequences out of the inference batch immediately and fills the freed slot with a waiting request. Traditional static batching waits for the entire batch to finish before accepting new requests — one long request stalls all the shorter ones behind it. Continuous batching eliminates that queuing penalty, improving throughput by up to 85 percent at scale and reducing the latency of short requests blocked behind longer ones.
What is speculative decoding?
Speculative decoding uses a small draft model to propose multiple candidate tokens at once, then uses the full target model to verify them in parallel. When the draft model predicts correctly the target model accepts the batch and effectively generates multiple tokens per step. The typical speed improvement is 1.5x to 2x for longer code and structured completions. It requires a compatible draft model and adds operational complexity; it is most justified for workloads dominated by long, patterned completions.
Should I use quantization to reduce LLM latency?
INT8 quantization typically shows less than one percent quality degradation and meaningfully increases throughput on the same hardware — worth enabling for most production workloads. INT4 shows two to five percent degradation on average benchmarks, with larger drops on complex reasoning tasks. Calibrate on your specific workload before committing to INT4, particularly if quality on reasoning or instruction-following tasks is important to your application.