Hi everyone,
I'm an independent developer working on NUMALLAMA, a small, dependency-free C inference engine designed to run modern LLMs directly on CPUs, without GPUs or CUDA.
The project focuses on a particular challenge: making effective use of large multi-socket Arm servers instead of treating the entire system as a uniform CPU.
I developed and benchmarked it on a dual-socket Ampere Altra Max M128-30 system (256 Arm Neoverse N1 cores, 1 TB RAM).
Some of the key features:
NUMA-aware weight and KV-cache placement.
Per-socket work queues and synchronization.
Optimized NEON dot-product kernels.
Q4/Q8 quantization and speculative decoding.
Support for Llama 3.x, Qwen 2.5, Qwen 3 and MoE models.
OpenAI-compatible inference server.
In my benchmarks, NUMALLAMA achieves up to 99 tokens/s decoding on Llama 3.2 3B Q4 and 91 tokens/s on Qwen3-30B-A3B Q4, using CPU only.
One interesting observation is that llama.cpp performs better on a single socket than on both sockets on this particular system. NUMALLAMA uses explicit NUMA-aware memory placement and per-socket execution to address this.
I'd really appreciate feedback from the Arm community, especially on a few points:
Are there any Arm-specific optimization techniques for Neoverse N1 that I might be overlooking?
What tools or PMU counters would you recommend for deeper analysis of memory bandwidth, cache behavior and inter-socket traffic?
Are there recommended approaches for improving NUMA-aware workloads on Ampere Altra and other Neoverse-based servers?
Are there any Arm developer programs, communities or resources where a project like this could receive technical feedback and broader visibility?
The project is open source, and I'd be happy to discuss the implementation and benchmark methodology.
GitHub: https://github.com/miloserdev/numallamaBenchmarks: https://github.com/miloserdev/numallama/blob/master/docs/BENCHMARKS.md
Any suggestions, technical criticism or pointers to relevant Arm documentation would be very welcome.
Thanks!