According to a paper posted to arXiv on August 17 arXiv paper, researchers lifted a 3B recipient language model’s average accuracy from 55.5% to 60.6% across 16 benchmarks. The method, called Activation-Prune-Merge (APM), targets a long-standing problem in model fusion: combining models that differ in size and architecture.
How the method works
APM starts from a simple observation: truncating a large donor model down to a smaller architecture and injecting it with a tiny mixing weight already helps the smaller recipient. Instead of trying to align neurons one-to-one between mismatched models, APM builds task-conditioned activation maps on the donor, then selects the most salient layers, hidden dimensions, attention heads, and MLP neurons to prune it to the recipient’s shape. The extracted donor slice is merged back into the original recipient using a micro interpolation coefficient, so the donor acts as a source of concentrated functional components rather than a structure to copy exactly.
What the benchmarks show
On the 16-task suite spanning reasoning, mathematics, code generation, instruction following, and classification, the authors report the overall average rose from 55.5% to 60.6% arXiv paper DOI record. According to the authors, the biggest single jumps came on natural-language understanding: RTE accuracy climbed from 64.3% to 82.3%, QNLI from 52.3% to 65.7%, and BoolQ from 70.8% to 79.2% arXiv paper DOI record. Those gains arrived without fine-tuning the recipient, which the authors say matters most for practical deployment.
Why a training-free transfer matters
The study was led by Jiahe Fan, Si Chen, Yinghao Hou, Aiyuan Zhang, and Hong Xie, according to the arXiv author listing author index. Most capability-transfer techniques assume you can train or distill the smaller model, which costs compute and data. APM’s appeal is that the recipient improves from a donor’s structure alone, suggesting cross-scale fusion can work when the donor’s contribution is concentrated and carefully selected rather than semantically aligned neuron-by-neuron. For teams running small models at the edge or in cost-sensitive settings, a training-free lift of several points could change the build-versus-borrow calculus.
The work joins a recent run of arXiv ML studies tracked by zBrandco, including an agent-memory model that bars stale, retracted data zBrandco.
