Batch pruning keeps reasoning models fast at scale

Large reasoning models earn their accuracy by thinking longer, and a new arXiv paper frames the cost of that extended chain-of-thought generation as the central problem it targets arXiv paper. Serving these models in production means paying for inference on every request. The same work, posted to arXiv’s Computation and Language track, lists authors Yongmin Kim, Shota Takashiro, Yusuke Iwasawa, Takeshi Kojima, and Yutaka Matsuo arXiv listing. Its answer is a training-free pruning method that keeps models accurate and fast when many requests run together in a batch arXiv paper.

Why batched inference breaks existing pruning

Training-free adaptive pruning looked like a clean win: shrink the model without retraining. The paper documents where that breaks down — under batched inference, where a shared mask pools activations across samples and a fixed threshold calibrated on single examples no longer matches the aggregated distribution, so realized sparsity drifts and reasoning accuracy collapses arXiv DOI.

What the new method changes

The authors swap threshold selection for periodic top-k selection over aggregated importance scores, add an activation memory that retains neurons re-firing periodically during long reasoning traces, and run selection once per update period rather than at every token to preserve the speedup arXiv DOI.

The measured results

According to the paper, on DeepSeek-R1-Distill-Qwen-7B at batch size 4 with a 50% target sparsity, their method beats the previous state-of-the-art adaptive pruning method by 39.7 percentage points in average accuracy, while reaching 1.40x speedup over dense inference at 50% actual sparsity.

Elsewhere on zBrandco, the site covered an arXiv agent-memory model that bars stale, retracted data zBrandco.