New Trick Lets Mamba Models Steal Transformer Smarts Without Attention Blocks

Researchers have developed a novel two-stage distillation method to transfer knowledge from established Transformer models into newer, more efficient Mamba State Space Models (SSMs). The technique first distills a Transformer into a linearized version of Attention, then transfers that knowledge into a pure Mamba architecture, bypassing the need for hybrid Attention-SSM blocks that have been a common workaround.

This allows engineers to leverage the vast library of pre-trained Transformers while adopting the faster inference and lower memory footprint of Mamba models. The distilled 1B-parameter Mamba model achieved a perplexity of 14.01, nearly matching the teacher Transformer's 13.86, and maintained strong performance on most downstream tasks. This breakthrough provides a practical migration path for teams wanting to modernize their architectures without sacrificing hard-won model intelligence.