-
Inference speed isn't just a model-size problem
PagedAttention/vLLM, speculative decoding, and early-exit networks all speed up inference without shrinking a single parameter — three papers, one point.
-
Why joint compression beats doing it in sequence
APQ (Wang et al., CVPR 2020) shows what changes when architecture, pruning, and quantization are searched for jointly instead of tuned one after another.
-
Quantization: fewer bits per weight, without losing the model
LLM.int8(), SmoothQuant, and GPTQ — three concrete answers to the same problem, outlier activations that make naive low-bit quantization of LLMs fail.
-
Pruning: removing what a network doesn't need
From Optimal Brain Damage to the Lottery Ticket Hypothesis — what the pruning literature actually removes, and why unstructured sparsity doesn't automatically buy speed.
-
Is it the demonstrations, or the model?
Ling, Zhao, Zhang et al. (NAACL 2024) decompose ICL's predictive uncertainty into a demonstration-selection component and a model component.