Why joint compression beats doing it in sequence

Wang, T., Wang, K., Cai, H., Lin, J., Liu, Z., & Han, S. (2020). APQ: Joint Search for Network Architecture, Pruning and Quantization Policy. CVPR 2020.

The problem with sequential pipelines. The default way to combine compression techniques is to apply them in stages — design or select an architecture, then decide a pruning policy for it, then decide a quantization policy for what’s left, fine-tuning between stages. Each stage in that pipeline makes its decision based on the model as it exists at that point, without knowledge of what the following stage will do to it. A channel judged “important enough to keep” by a pruning stage that only sees the full-precision model isn’t necessarily the channel that’s most worth keeping once quantization error is also accounted for — the two decisions are entangled, but a staged pipeline resolves them independently.

APQ’s approach: search the joint space, but make it cheap. APQ treats architecture choice, per-layer pruning ratio, and per-layer bit-width as one combined search space and searches over it jointly rather than staging the three decisions. The obvious obstacle is cost: naively, evaluating one point in this joint space means training or fine-tuning a candidate (architecture, pruning, quantization) configuration and measuring its accuracy, and the joint space is far larger than any one of the three sub-spaces alone. The paper’s specific contribution is a way to avoid that cost. They first train a single accuracy predictor for full-precision candidate architectures using a “once-for-all” network — a network built so that many sub-architectures can be evaluated without separately training each one, so this predictor itself requires no extra training cost per candidate. They then transfer that full-precision predictor into a quantization-aware predictor using a comparatively small number of actual quantized (architecture, pruning, quantization) evaluations, rather than needing to collect a large quantized-accuracy dataset from scratch.

Reported results. On ImageNet, searching this joint space directly is reported to reach 2.3% higher accuracy than a pipeline that searches architecture, pruning, and quantization separately at a matched efficiency budget, and the resulting models achieve roughly 2x lower latency and 1.3x lower energy consumption than a MobileNetV2 baseline combined with HAQ (a strong prior quantization-search method), while the search and data-collection cost is reported as substantially lower than prior joint neural-architecture-search-plus-compression approaches.

Why the accuracy gain isn’t surprising, given the setup. The point isn’t that joint search finds some clever trick unavailable to staged pipelines — it’s that a staged pipeline is provably leaving information on the table by construction: the pruning decision in a staged pipeline literally cannot see the quantization policy that will be applied afterward, so it cannot account for how the two interact. A joint search, even an approximate one, has access to that interaction. The corresponding cost, which is the real engineering content of the paper, is that a joint search space is combinatorially larger, and most of APQ’s method is specifically about making evaluation of that larger space cheap enough to be tractable — not about a fundamentally different compression technique.

The general pattern. This is one instance of a broader theme in this area: once you frame “how much to prune” and “how many bits to use” as a joint decision under a shared resource budget rather than two independent hyperparameters, the compression problem turns into a constrained combinatorial search/optimization problem, and progress often comes from making that joint search efficient rather than from a better pruning or quantization criterion in isolation.