Two GPU kernels that implement the same scaled INT8 matrix-multiply interface are usually treated as drop-in replacements. A new arXiv preprint tests that assumption inside vLLM and finds it does not survive contact with real text generation: with the checkpoint, prompts, hardware, inference engine, decoding settings, and quantization configuration all held fixed, the authors swapped only the INT8 linear kernel — CUTLASS in one arm, Triton in the other — and the two arms agreed on no generated sequence at all, across comparisons of 8, 16, and 64 sequences arXiv paper.
Neither arm was flaky on its own. At 1.7B parameters, each kernel reproduced its own output bit-for-bit across cold restarts, the authors report, so the disagreement was not run-to-run noise but a stable consequence of which kernel the engine picked DOI record.
Why the accumulator is not the culprit
The paper’s core argument is what the authors call an integer alibi. For shared INT8 operands under a verified no-overflow bound, the INT32 dot product is exact and independent of summation order, which eliminates the accumulator as a possible source of difference — normally the first suspect whenever two implementations of the same math disagree arXiv paper.
To pin that down layer by layer, the team fed both kernels identical operands taken from every linear layer of Qwen3-1.7B and Qwen3-8B — 196 and 252 layers respectively — and found bit-identical outputs under power-of-two scales, matching a pinned prediction list on all 196 and all 252 layers. Under the checkpoints’ real scales, they documented differences of at most one bfloat16 spacing DOI record. The 1.7B prediction list was pre-registered; the 8B list, the authors note, was pinned but not blind.
Where the divergence actually lives
That evidence moves the blame to scale application and output rounding after the exact accumulator. Applied as a probe checkpoint, the same intervention restored end-to-end bitwise agreement, with all 8 and all 16 compared sequences matching arXiv paper.
Floating-point kernels leave a different fingerprint, the authors report. In cross-implementation FP8 GEMM, both the prevalence and the magnitude of the differences grow with reduction depth, while the INT8 fraction stays at parts per million and within one spacing over a 64-fold range of K DOI record. Teacher-forced replay then tied layers back to tokens: flips concentrate where logit margins are small, and margin size predicted flip risk with a ROC-AUC of 0.94 across 16,384 positions.
What this changes for inference teams
For anyone serving quantized models, the takeaway is that kernel selection belongs in the reproducibility contract alongside the checkpoint hash and the decoding seed. Two deployments can run the same weights, the same prompt, and the same quantization recipe and still return different text, and nothing in a standard configuration dump would explain why. The authors say they will release the pre-registration, the per-layer predictions, manifests carrying kernel-selection evidence, and a conformance procedure that turns these controls into a concrete interchangeability check.
That matters for evaluation harnesses and regression suites in particular: a scoring difference between two runs may reflect a kernel dispatch rather than a model change, and a bug reproduced on one machine may vanish on another with a different library build. Implementation choices this far down the stack shape what a serving engine actually emits, a theme that also runs through recent work on accelerating diffusion-based LLM decoding CORA-Diff speeds up diffusion LLM inference.
