NUMA-aware LLM inference on Neoverse N1 - looking for technical feedback

Hi everyone,

I'm an independent developer working on NUMALLAMA, a small, dependency-free C inference engine designed to run modern LLMs directly on CPUs, without GPUs or CUDA.

The project focuses on a particular challenge: making effective use of large multi-socket Arm servers instead of treating the entire system as a uniform CPU.

I developed and benchmarked it on a dual-socket Ampere Altra Max M128-30 system (256 Arm Neoverse N1 cores, 1 TB RAM).

Some of the key features:

  • NUMA-aware weight and KV-cache placement.

  • Per-socket work queues and synchronization.

  • Optimized NEON dot-product kernels.

  • Q4/Q8 quantization and speculative decoding.

  • Support for Llama 3.x, Qwen 2.5, Qwen 3 and MoE models.

  • OpenAI-compatible inference server.

In my benchmarks, NUMALLAMA achieves up to 99 tokens/s decoding on Llama 3.2 3B Q4 and 91 tokens/s on Qwen3-30B-A3B Q4, using CPU only.

One interesting observation is that llama.cpp performs better on a single socket than on both sockets on this particular system. NUMALLAMA uses explicit NUMA-aware memory placement and per-socket execution to address this.

I'd really appreciate feedback from the Arm community, especially on a few points:

  1. Are there any Arm-specific optimization techniques for Neoverse N1 that I might be overlooking?

  2. What tools or PMU counters would you recommend for deeper analysis of memory bandwidth, cache behavior and inter-socket traffic?

  3. Are there recommended approaches for improving NUMA-aware workloads on Ampere Altra and other Neoverse-based servers?

  4. Are there any Arm developer programs, communities or resources where a project like this could receive technical feedback and broader visibility?

The project is open source, and I'd be happy to discuss the implementation and benchmark methodology.

GitHub: https://github.com/miloserdev/numallama
Benchmarks: https://github.com/miloserdev/numallama/blob/master/docs/BENCHMARKS.md

Any suggestions, technical criticism or pointers to relevant Arm documentation would be very welcome.

Thanks!