NUMA or Non-Uniform Memory Access is a system design that allows processors to have access to their own local memory as well as remote memory of other processors in a shared-memory multiprocessor system. In NUMA systems, each processor has a local memory bank that it can access quickly, and remote memory banks that it can access more slowly through interconnects. Linux has specific support for NUMA hardware and it can improve system performance by utilizing the locality of memory access.
Here are some key differences and considerations when working with NUMA systems in Linux:
Memory Management: In NUMA systems, memory management is divided into two parts, local and remote memory. Linux kernel assigns memory to each processor based on the local and remote memory access cost. The memory allocation and deallocation policies are different for local and remote memory.
Process Scheduling: Linux uses the Completely Fair Scheduler (CFS) for process scheduling. In NUMA systems, the scheduler has to be aware of the locality of the memory access to minimize the number of remote memory accesses. Linux kernel uses a technique called NUMA-aware scheduling to schedule processes on the processor that has the closest access to the memory.
Interconnects: In NUMA systems, processors are connected to each other through interconnects. The interconnects are used for communication between processors and for accessing remote memory. The interconnects can be a bottleneck for performance, and it is important to ensure that they are configured correctly for optimal performance.
Tools: Linux provides several tools for monitoring and managing NUMA systems. The numactl tool is used for controlling memory allocation policies for NUMA systems. The numastat tool provides information about the memory usage and the number of remote memory accesses. The hwloc tool can be used to visualize the topology of the NUMA system.
In summary, working with NUMA systems in Linux requires an understanding of the memory management policies, process scheduling, interconnects, and specialized tools. By correctly configuring the system and using the right tools, performance can be optimized on NUMA systems.