Inference speed isn't just a model-size problem

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. ICML 2023. Teerapittayanon, S., McDanel, B., & Kung, H. T. (2017). BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks.

None of these three papers prunes or quantizes anything, and that’s the point of pairing them here: a transformer’s serving latency is shaped by how autoregressive decoding is scheduled and how memory for the KV cache is managed, not only by how many parameters the model has or how many bits each one costs — axes that shrinking the model doesn’t automatically address.

PagedAttention / vLLM (Kwon et al., 2023): the KV cache is a memory-management problem. During autoregressive generation, a transformer caches key/value activations for every previous token so it isn’t recomputed at each step; for long sequences and many concurrent requests, this cache can be larger than the model weights themselves, and its size changes dynamically as generation proceeds. The paper’s diagnosis is that standard serving systems allocate this cache contiguously per request, which causes memory fragmentation and wasted capacity — conceptually the same problem operating systems solved for process memory decades ago. PagedAttention applies that solution directly: it allocates KV-cache memory in fixed-size, non-contiguous blocks, the way an OS pages virtual memory, which lets the vLLM serving system built on top of it achieve near-zero cache waste and share cache blocks across requests where possible. The reported result is a 2–4x throughput improvement over prior state-of-the-art serving systems at the same latency, with larger gains for longer sequences and larger models — entirely a scheduling and memory-layout change, with the underlying model completely untouched.

Speculative decoding (Leviathan et al., 2023): make the sequential bottleneck parallel. Autoregressive decoding is inherently sequential — one forward pass produces one token, and the next token’s forward pass depends on it. The paper’s observation is that many tokens in a generated sequence are, in retrospect, “easy” — predictable enough that a much smaller, cheaper draft model would have guessed the same token as the large target model. Speculative decoding exploits this: the small draft model proposes several tokens ahead, and the large model then verifies the entire proposed continuation in a single parallel forward pass, accepting whichever prefix of the guess matches what the large model would have produced on its own (with a rejection-sampling correction that guarantees the output distribution is exactly the same as standard decoding from the large model alone — not an approximation). Reported speedups are 2–3x on T5-XXL with no retraining or architecture change to either model, and no change whatsoever to the output distribution.

Early exit / BranchyNet (Teerapittayanon et al., 2017): not every input needs the full network. The paper attaches side-branch classifiers at intermediate layers of a network and lets an input exit through an early branch once its prediction at that branch is confident enough, skipping the remaining layers entirely; only inputs that fail the confidence check at early branches proceed deeper. The premise, distinct from the two papers above, is architectural rather than about scheduling or parallelism: it assumes inputs vary in difficulty, and spends compute proportionally, rather than spending the same fixed depth on every input regardless of how “easy” it is.

Why grouping these three together matters. PagedAttention changes how memory is laid out around a fixed model. Speculative decoding changes how many forward passes are needed to get a fixed amount of output. Early exit changes how much of the network an individual input actually traverses. None of the three touches a single weight value. All three are legitimate “make it faster” techniques, and a serving system that only invests in a smaller model while ignoring scheduling, memory layout, and input-dependent compute is leaving comparable or larger gains unaddressed — these are answers to different bottlenecks than the ones pruning and quantization address, not competing solutions to the same one.