Skip to content
Memory Region / Key Table

Memory Region / Key Table

A remote RDMA operation doesn’t address memory the way a local pointer does – it carries an rkey, opaque to the requester, that the target RNIC has to resolve against its own Memory Region table before anything moves. This part covers that resolution end to end: the Memory Region descriptor a registration produces, the key table that stores it, the on-chip cache that makes resolving a key cheap, the registration handoff that installs one, what happens when a resolved key’s permissions or bounds don’t match the operation asking to use it, what changes when several Queue Pairs share one region, how the key cache decides what to evict under pressure, what actually executes that decision, what happens when a region is deregistered while an operation against it is still in flight, and, closing it out, how many regions a real chip can actually hold registered at once and what it costs in silicon to let registration and per-DMA lookups share the same structure safely.

Ten chapters here, not eleven – a deliberate deviation from Queue Pair Management’s shape, not a gap. A QP’s install-then-run lifecycle earns its own dedicated state-machine chapter because RESET→INIT→RTR→RTS is genuinely rich enough to need one; a Memory Region’s own lifecycle – registered, valid, deregistered – doesn’t have that richness, so the two questions Queue Pair Management answers in separate chapters (installation, and the state machine governing it) collapse into one here, Registering a Memory Region, without skipping either question.

One Memory Region, start to finish – what MR-owned state actually does across one operation, from registration through steady-state key caching to the hazards that can interrupt it (permission faults, shared arbitration, eviction, mid-transaction deregistration), closing on how many regions can exist at once. Each step names the chapter that covers it.

Chapters

The Memory Region Descriptor
A registered memory region hands back two keys, not one – and they are not two views of the same number. One unlocks local Work Requests against this process’s own memory; the other is what a remote peer presents to touch it. This chapter is what’s actually in the entry both of them point at.
The Key Table
A QP number is a dense index handed out by the adapter itself. A key is neither – it’s an opaque 32-bit value a remote peer presents from across the wire, and the table it has to resolve against is built to fail cleanly on a wrong one rather than fail dangerously into someone else’s memory.
Key Caching
The RNIC’s on-chip cache doesn’t have an MTT-only section – it’s the exact same SRAM Queue Pair Management already showed you caching QP context. A memory-region-heavy workload and a QP-heavy workload are fighting over the same shelf space, and losing that fight is a confirmed cost on shipped hardware, not a hypothetical one.
Registering a Memory Region
A memory region has none of a QP’s five-state richness – it’s usable or it isn’t. This chapter covers the one call that makes it usable, the real hardware cost the verbs manual doesn’t describe, and the one alternative – On-Demand Paging – that defers that cost instead of paying it upfront.
Permission and Bounds Violations
An RDMA write that lands outside a memory region’s registered range and one that lands inside it but requests an operation the region was never registered to allow surface through the exact same completion status – no public source says these are two separate hardware checks, or which one runs first. A local Work Queue Element’s own bad lkey fails a structurally different way, and never reaches the wire at all.
Shared-MR Arbitration
A shared receive buffer registered once and addressed by a dozen QPs turns one key-table entry into something many owners touch at once – concurrent reads are presumably free, but no public source says what happens when one of those owners tries to invalidate the entry while the others are still reading it.
Key Cache Eviction Policy
The QP Cache only ever loses one entry at a time to eviction pressure. The Key Cache carries that same policy question, plus a second and harsher one: a single deregistration call is reasoned to invalidate the whole cache at once, not just the key being retired.
Hardware-Assisted Key Cache Eviction
Queue Pair Management asked what runs an eviction policy: fixed silicon, a programmable engine, or a trap to software. This chapter asks that question once, in one paragraph, and then asks a second one Queue Pair Management never had to: what runs a flush of the entire cache, and what does that flush cost the lookups it interrupts.
Mid-Transaction Deregistration
The verbs contract is unambiguous: stopping use of a key before deregistering it is software’s job, not hardware’s. What a DMA burst already mid-flight against that key does when the table entry disappears underneath it is a question the contract doesn’t have to answer, and no public source does either.
MR Capacity and Timing
How many Memory Regions can a real chip hold registered at once isn’t even a well-formed question on its own – a registration’s hard cost scales with how many pages it covers, not with whether it exists. This closing chapter asks what that actually bounds, and picks up Registering a Memory Region’s open question about whether a key can ever resolve against a half-populated entry.
Last updated on