Regularities considered harmful: Forcing randomness to memory accesses to reduce row buffer conflicts for multi-core, multi-bank systems

We propose a novel kernel-level memory allocator, called M3 (Mcube, Multi-core Multi-bank Memory allocator), that has the following two features. First, it introduces and makes use of a notion of a memory container, which is defined as a unit of memory that comprises the minimum number of page frames that can cover all the banks of the memory organization, by exclusively assigning a container to a core so that each core achieves bank parallelism as much as possible. Second, it orchestrates page frame allocation so that pages that threads access are dispersed randomly across multiple banks so that each thread's access pattern is randomized. The development of M3 is based on a tool that we develop to fully understand the architectural characteristics of the underlying memory organization. Using an extension of this tool, we observe that the same application that accesses pages in a random manner outperforms one that accesses pages in a regular pattern such as sequential or same ordered accesses. This is because such randomized accesses reduces inter-thread access interference on the rowbuffer in memory banks. We implement M3 in the Linux kernel version 2.6.32 on the Intel Xeon system that has 16 cores and 32GB DRAM. Performance evaluation with various workloads show that M3 improves the overall performance for memory intensive benchmarks by up to 85% with an average of about 40%. Categories and Subject Descriptors B.3.1 [Memory Structures]: Semiconductor Memories-Dynamic memory (DRAM); D.4.2 [Operating Systems]: Storage Management-Allocation/deallocation strategies; D.4.2 [Operating Systems]: Storage Management-Main memory; D.4.8 [Operating Systems]: Performance- Measurements General Terms Algorithms, Design, Experimentation, Measurement, Performance.