Inference

Overview

Inference is the process of running a trained model on new inputs to produce outputs. For LLMs, this means token-by-token generation in response to a user prompt. Model parameters are not updated during inference. Inference cost, latency, and throughput determine the practical viability of an LLM-based product.

Inference optimization

Key optimizations include KV Cache, Speculative Decoding, quantization, FlashAttention, and request batching. Cloud APIs apply these transparently. For local inference, engines such as llama.cpp, vLLM, and TGI handle optimization.