Arm servers finally get llama.cpp love with 6.5x speed boost

The popular llama.cpp project, which runs LLMs locally, had a major performance gap on Arm-based CPUs compared to x86. The issue was that a key routine called "tiled K-quant matmul," added in late 2026, only had optimized kernels for x86 instructions like AVX-512. For several important low-bit quantized model formats, Arm CPUs were forced to use a slow fallback path that unpacked weights repeatedly for every calculation.

A new kernel has been added that uses Arm's NEON dot-product instruction, SDOT. Testing on Arm Neoverse server cores shows dramatic speedups for prompt processing, with the largest gain being a 6.5x improvement for the IQ3_M format on an N1 core. This means developers and users running models on Arm servers, Macs with Apple Silicon, or other Arm devices can now get much faster prompt processing for many quantized models, finally closing a significant performance gap that had been holding back the Arm ecosystem for local AI.