Before diving in to Zen, I'd like to talk briefly CPU memory-caching systems.

Modern CPUs contain a variety of fast memory which cache accesses to the larger more expensive RAM. Usually we call the fastest, smallest cache L1, the next fastest L2, and the largest, but slowest cache L3. These caches communicate with each other to move data around (see MESI protocol).

Assume for simplification that the L3 cache knows how to get data from main memory or from the L1 and L2 caches. Also assume that the L3 cache is the only part of the memory system which is able to talk main memory.

If Core 0 wants to read from memory, it will ask the local L1 cache to perform the read. If the data is in the L1 cache, the cache sends the data to the CPU registers. If the data is not in the L1 cache, the cache asks L2. Finally, the L2 cache asks L3 and the L3 cache goes and fetches the data.

Next, assume that all data is written into the L1 cache then propagated out to other caches or main memory as needed. When the L1 cache is out of space, it will evict the data into the L2 cache. When the L2 cache is out of space, it will evict the data into the L3 cache. Finally, if L3 runs out of space, it will evict the data back to main memory.

Generally, each CPU has it's own L1 and L2 cache, but the L3 cache is shared by all of the cores. Something like this:

/-----------------------------------\
|                                   |
|   L3 cache (shared by all cores)  |
|                                   |
+--------+--------+--------+--------+
|        |        |        |        |
| L{1,2} | L{1,2} | L{1,2} | L{1,2} |
|        |        |        |        |
+--------+--------+--------+--------+
|        |        |        |        |
|  Core  |  Core  |  Core  |  Core  |
|        |        |        |        |
\--------+--------+--------+--------/

This style of L1/L2/L3 cache has been common in CPU designs for many years.

To talk about AMD's Zen architecture, there's a few terms we need to define:

• CCX: Core Complex. Contains a max of 4 cores and a slice of the L3 cache.
• CCD: Core Chiplet Die: A single die which contains 2 CCX (on Ryzen 3/5/7)
• IO controller: A separate die containing DDR controllers, PCIe controllers. Also the hub for the "Infinity Fabric"
• Infinity Fabric: High performance interconnect connecting CCXs

Each Ryzen processor has one IO controller and some number of CCD dies. This CCD/CCX/IO controller strategy allows AMD to mass produce CCD chips, performance/correctness test the cores (this is called "binning"), then build a wide variety of different processor configurations.

The particular Ryzen 3800X I own has a single IO controller and a single CCD. The CCD contains two CCXs. Each CCX in my particular CPU has all 4 physical cores enabled. Each of those physical cores is capable of running two threads (see Simultaneous Multithreading (SMT)).

Here's how this looks in hwloc's lstopo output:

Interpreting the output, we can see that:

• CCX1 contains the cores that the OS calls 0, 1, 2, 3, 4, 5, 6, and 7
• CCX2 contains the cores that the OS calls 8, 9, 10, 11, 12, 13, 14, and 15

This likely means that cores in the same CCX can communicate with each other by reading and writing to their shared L3 cache. If cores from different CCXs need to communicate, they will have to use the infinity interconnect.

To verify my hypothesis, lets start with a classic pingpong roundtrip latency test.

This test runs two threads. The first thread flips a value in memory to some value "Ping", then waits to see another value "Pong." The second thread waits to see "Ping", then flips the value to "Pong."

The first thread starts timing right before sending "Ping", and stops timing once it sees "Pong."

I've plotted the mean core-to-core pingpong round-trip time for all pairs of cores (including the SMT cores). Latency for A->B and B->A were both tested (for no particular reason).

That is:

for core1 in cores:
    for core2 in cores:
        if core1 == core2: continue # skip self, both threads spin
        results[ (core1, core2) ] = mean_of_many_tests( core1, core2 )

This plot is a heatmap of core->core pingpong roundtrip time, in nanoseconds:

As we can see, there's a clear difference in round-trip latency between pairs of cores in the same CCX and pairs of cores in a different CCX. For example, core 0->core 3 round-trip latency is pinkish, but core 0->core 4 latency is dark blueish. The actual values I got around ~96ns for 0->3 and ~196ns for 0->4.

I do not have a good explanation for the upper and lower diagonal lines (at around 48ish ns), but, we can clearly see that pairs of cores for which latency was low share a common L3 cache (they are in the same CCX).

For full details about this benchmark, please see the source and my results here.

So, we know that there is a difference in latencies, but, does this really matter? Few applications actually have any reason to bounce a single value back and forth between cores.

When discussing latency, programmers who are more bandwidth focused will often (rightly) say something like "the latency doesn't matter, I only care about bandwidth." Most programmers are more focused on bandwidth than latency, because generally we care a lot more about throughput than latency.

This sort of behavior will not come as a surprised for programmers used to programming for multi-socket (multiple physical CPU) server systems (see NUMA). However, for those writing parallel applications, I feel like this effect is still notable and warrants slightly different considerations than traditional NUMA.

In a traditional multi-socket/multiple-physical CPU system, RAM is physically connected to only one of the many CPUS. If core0 of cpu0 needs to access memory physically attached to cpu1, it will pay a latency penalty to access this memory (this is NUMA - Non-Uniform Memory Access). However, we've also generally assumed that core0 can access any part of memory attached to cpu0 with the same latency. Therefore, applications built for NUMA architectures often focused on getting their data into the memory CPU attached to a CPU, then using the cores on that CPU to process the data.

For platforms like AMD's (and recent Intel, see Intel Mesh) we have to now make NUMA-style considerations any time we are operating over a piece of data on multiple cores, not just when dealing with multiple sockets.

Consider the specs for the EPYC 7542:

• 32 cores
• 512 KiB L2 d-cache per core
• 128 MiB L3 cache for the entire chip

On the 7542, we have to divide this 128 MiB of L3 cache up into CCX cache slices. Assuming each CCX has 4 cores (which I think is correct), we'd have 32/4=8 CCXs on this CPU. Therefore, each CCX (4 cores) only has 16 MiB of "close L3 cache memory," accessing the rest of the cache will incur a small amount of extra latency.

For comparison, a 2016 Intel Broadwell Xeon could be configured with 22 cores and 55 MiB of L3 cache. This means that the AMD has a core/fast-shared-l3 ratio of 0.25 MiB/core and the Broadwell has a core/fast-shared-l3 ratio of 2.5 MiB/core. Building high performance code with 2.5 MiB of super fast shared memory per core is a slightly different game than building high performance applications with only 0.25 MiB of super fast shared memory per core.

I suspect that "networked" cpu architectures will become the new normal in the next few years. These new platforms are going probably going to be increasingly more and more sensitive to data access and data movement than the systems of the past.