I was trying to find if Skylake-X has a page-crossing prefetcher that loads PTEs into STLB, and if yes what access pattern activates it. I wrote the following program to test this:

#include 
#include 
#include 
#include 
#include "../../events.h"
#include "../../utils.h"

#define PAGE_SIZE 4096
#define MAP_BASE 0x1000000000
#define THOUSAND 1000

using u8 = uint8_t;
using u64 = uint64_t;

struct Data {
    u8* map_base;
    int target_cacheline;
    int n_pages;
    int zero;
};

void kernel(void* data) {
    Data d = *(Data*)data;
    int dummy = 0;
    // Cross target_cache line in every page and hope to see an STLB hit for the next page.
    for (int i = 0; i < d.n_pages; i++) {
        u8* page_base = (d.map_base + d.zero) + (i * PAGE_SIZE);
        u8* cacheline_offset = page_base + (d.target_cacheline * 64);
        // Issue two loads to induce a cache line crossing access.
        // Access the last byte in target_cacheline and first byte in (target_cacheline + 1).
        u8* access_offset = cacheline_offset + 63;
        for (int k = 0; k < 2; k++) {
            dummy += access_offset[k + d.zero];
            d.zero = d.zero & dummy;
        }
    }
    asm volatile("" :: "r"(dummy));
}

int main(int argc, char** argv) {
    Data d;
    d.zero = argc > 0 ? 0 : 1;
    d.n_pages = 1 * THOUSAND;

    int map_size = d.n_pages * PAGE_SIZE;
    void* addr = mmap((void*)MAP_BASE, map_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0);
    if (addr == MAP_FAILED) {
        printf("Memory map failed!\n");
        exit(EXIT_FAILURE);
    }
    d.map_base = (u8*)addr;
    prefault_range(d.map_base, map_size);

    // Run `kernel` for every cache line in the range [0, 63).
    for (int i = 0; i < 63; i++) {
        flush_tlb_range(d.map_base, map_size);
        d.target_cacheline = i;
        u64 event = SKX_MEM_INST_RETIRED_STLB_MISS_LOADS;
        u64 counter = 0;
        perf_measure(&event, &counter, /*nevents=*/1, /*workload=*/kernel, /*args=*/(void*)&d);
        printf("MAPPED PAGES = %lld, TARGET_CACHELINE = %d, STLB_LOAD_MISS = %lld\n", d.n_pages, d.target_cacheline, counter);
    }

    return 0;
}

The program uses a few utility functions such as perf_measure(), flush_tlb_range() etc. that come from an external library I use for my uarch experiments. If someone would generously offer try this experiment on other uarchs, I would happily post a standalone reproducer.

From earlier experiments, I had developed a hypothesis that there's a critical band of offsets within a page where if loads are issued, a prefetcher is "activated" that loads the PTE of the next page(s?) to STLB. I was consistently observing the band was between bytes 3390 to 3410 into the page.

This hypothesis was too random, so with further experiments I refined it to this:

There is a critical cache line within a page which when crossed, activates a prefetcher that loads next page PTE to STLB.

The program above essentially tests the refined hypothesis. kernel is invoked with a target_cacheline and it accesses 2 bytes in every mapped page: the last byte of target_cacheline and the first byte of target_cacheline + 1.

The output is quite interesting:

MAPPED PAGES = 1000, TARGET_CACHELINE = 0, STLB_LOAD_MISS = 1000
MAPPED PAGES = 1000, TARGET_CACHELINE = 1, STLB_LOAD_MISS = 1000
MAPPED PAGES = 1000, TARGET_CACHELINE = 2, STLB_LOAD_MISS = 1000
...
MAPPED PAGES = 1000, TARGET_CACHELINE = 52, STLB_LOAD_MISS = 2
...
MAPPED PAGES = 1000, TARGET_CACHELINE = 53, STLB_LOAD_MISS = 1000
MAPPED PAGES = 1000, TARGET_CACHELINE = 54, STLB_LOAD_MISS = 1000
MAPPED PAGES = 1000, TARGET_CACHELINE = 55, STLB_LOAD_MISS = 1000

The output is consistent across many runs and two SKX machines. The prefetcher in question could be a next-page prefetcher which Intel admits to include in Atom based E-cores and Redwood Cove, or a stride prefetcher that's allowed prefetch beyond page boundaries, but it's always only the 52nd cache line which when crossed, reports 3 > MEM_INST_RETIRED.STLB_MISS_LOADS > 0.

I am asking this question because I don't fully understand why would a prefetcher do this. To be specific, why is there only a single key cache line, rather than a threshold on a range of cache lines that once crossed, activates the prefetcher. It would've made more sense if crossing cache lines after threshold=52 (e.g. cache line 52, 53, ..., 62) activated the prefetcher -- reason being we are getting closer to accessing the next page. But that's not what I observe; it's always only the 52nd cache line. Has anybody come across any resource that describes a similar prefetcher behavior?

It is also possible that I am attributing a completely different microarchitectural effect to "prefetcher activates at 52nd cache line crossing". I'd also like to be corrected if that's the case.

For the sake of completeness, here's the assembly that kernel compiles to after some cleanup (-O2):

kernel:
        movsxd r10,DWORD PTR [rdi+0xc]
        test   r10d,r10d
        jle    .L1;
        mov    r9d,DWORD PTR [rdi+0x8]
        mov    r8,QWORD PTR [rdi]
        shl    r10,0xc
        xor    edx,edx
        mov    ecx,DWORD PTR [rdi+0x10]
        xor    edi,edi
        shl    r9d,0x6
        movsxd r9,r9d
        add    r9,0x3f
.L2:
        movsxd rsi,ecx
        lea    rax,[rsi+rdi*1]
        add    rsi,r8
        add    rdi,0x1000
        add    rax,r9
        movzx  esi,BYTE PTR [rsi+rax*1]
        add    rax,r8
        add    edx,esi
        and    ecx,edx
        lea    esi,[rcx+0x1]
        movsxd rsi,esi
        movzx  eax,BYTE PTR [rax+rsi*1]
        add    edx,eax
        and    ecx,edx
        cmp    r10,rdi
        jne    .L2;
        ret
.L1:
        xor    edx,edx
        ret