Registering a Memory Region
Overview
A Queue Pair climbs through a hardware-enforced sequence of states before it can carry a single packet – RESET, INIT, RTR, RTS, each transition its own gate. A memory region doesn’t. It is registered, at which point its lkey and rkey are immediately usable in a Work Request, or it is deregistered, at which point they are not. There is no INIT-equivalent holding state, no negotiated handshake with a peer, no multi-attribute mask a driver assembles across several calls. This chapter covers the entire lifecycle – registration, validity, deregistration – in one place, in place of a separate state-machine chapter, precisely because the underlying lifecycle doesn’t have QP State Machine’s richness to justify splitting it up the same way.
Locator
The Key Table covered the MPT/MTT structure a memory region’s entry lives in once it exists. This chapter covers how it gets there.
Analogy
Registering a Memory Region is conceptually the same act as installing a page-table mapping and pinning those pages against eviction or swap – just performed once, explicitly, for a range about to be exposed outside the process’s own instruction stream, instead of continuously for the process’s own working set.
Block diagram
Registration walking page tables and pinning pages into a populated MPT/MTT entry, which then becomes a valid MR with usable lkey/rkey, and later deregistration retiring it – with a dashed branch showing the deferred-resolution (ODP) path skipping upfront pinning for a page-fault-style first touch instead, and rejoining the same MPT/MTT population step.
All 9 steps
- Pinned registration. Software registers a 64 KiB buffer: a Protection Domain, a virtual address range, and access flags. Nothing exists in the adapter’s tables yet. Registering a Memory Region
- The driver walks the process’s page tables and pins all 16 pages so they can’t move. This cost grows with the size of the region and is paid now, whether or not every page is ever touched. Registering a Memory Region
- One MTT entry per page holds its translation, and the MPT entry holds the region’s address, length and permissions. The Memory Region Descriptor
- Only if the key goes live first, and this book doesn’t settle whether hardware prevents that window. If it can open, a lookup in it resolves against an incomplete translation, which is why the keys come last here. Registering a Memory Region
- Keys are assigned and the MR is valid. There is no further state to step through, unlike a QP: the lkey can go into a Work Request and the rkey to a peer right away. Registering a Memory Region
- On-Demand Paging. The same call with the deferred-resolution permission set skips pinning. Registration costs about the same whatever the region’s size. Registering a Memory Region
- It rejoins the same path: the descriptor is populated and the keys go live, with pages still unresolved. Registering a Memory Region
- The cost moves to first touch: the first access to each page pauses for a page-fault-style resolution, then resumes. An implicit-ODP rkey covering a whole address space also exposes all of it if leaked. Single-Burst Fault Handling
- Deregistration. Deregistration removes the MPT and MTT entries, drops them from the Key Cache if resident, and retires the keys. It promises nothing about operations already in flight that use them. Mid-Transaction Deregistration
Deep walkthrough
What registration actually costs, and why a software-facing contract doesn’t have to say. IBTA-V1 requires only that a successful registration produce a valid, checkable descriptor with these fields correctly populated; it says nothing about how many descriptors one registration operation can install, or how many the underlying hardware could accept in a single pass if a workload needed to register many ranges at once. A software registration call could reasonably register exactly one range per call, or more than one – nothing in the public record documents a shipping RNIC batching registrations this way, and this book doesn’t assert one does, but “one range per call” is a software-interface convention some real implementations happen to converge on, not an architectural ceiling. The call takes a Protection Domain, a virtual address range, and a set of access flags, and returns a populated descriptor carrying an lkey and rkey (The Memory Region Descriptor). What it does to get there – same abstraction-boundary gap this book has already hit at Overview’s doorbell and Queue Pair Management’s QP table organization – is not described at the software-contract level. That contract is deliberately a boundary: software states what it wants registered and gets back keys, and the real cost sits entirely on the hardware side of that boundary. Architecturally, what has to happen is not in serious doubt even without a vendor’s exact sequence to cite: the process’s page tables get walked to resolve the requested virtual range down to physical pages, those pages get pinned so the OS can no longer swap or relocate them out from under a device that holds direct physical-address translations to them, and the MR/Key Table gets populated – an MPT entry recording address, length, and permissions, plus one or more MTT entries covering however many physical pages the registration spans. None of that is this book asserting a specific implementation’s exact op sequence; it’s the minimum architectural work a “return a valid, hardware-usable key” contract has to do.
Valid or invalid – not a state machine. Once registration completes, the MR is valid: its keys can appear in a local Work Request or be handed to a remote peer immediately, with no further transition to wait through. QP State Machine exists because a QP needs a port bound, a partition-key index supplied, and a peer named before it’s safe to use – three separate hardware-enforced gates. A memory region has none of that to negotiate; the moment its table entry is populated, its keys are live – though whether that population is atomic with respect to a concurrent lookup is itself an open question this book doesn’t resolve. Registration is a multi-step construction (page tables walked, pages pinned, an MPT entry populated, one or more MTT entries populated behind it), and if the key becomes externally resolvable before every MTT entry backing it exists, a lookup landing in that window would resolve against an incomplete translation – the same shape of torn-state hazard Mid-Transaction Deregistration treats with real depth for the teardown side, with no equivalent treatment here for construction. No source this book found addresses it either way. Deregistration is the only other transition this object has: it removes the MPT/MTT entries and retires the keys. That’s the entire lifecycle – two states, not five – and it’s stated plainly rather than left for a reader who just finished Queue Pair Management to wonder where this book’s version of the QP state machine went. It didn’t get skipped; the object underneath it doesn’t have that shape.
On-Demand Paging: the real alternative to pinning upfront. Requesting the deferred-resolution permission at registration creates an MR that skips upfront page pinning entirely. The common implicit ODP case – one MR covering a process’s entire address space rather than one fixed range – follows a specific recipe documented in the RDMA man pages: the deferred-resolution permission set, a base address of zero, and a length spanning the full address space.
The tradeoff here is precise: a pinned registration pays its full cost once, upfront, at registration time, and every access after that is guaranteed not to stall on a page fault. An ODP registration pays close to nothing at registration time and instead defers that cost to first touch – the first access to a given page triggers a page-fault-style resolution, handled cooperatively by hardware and the OS against the OS’s own page tables, exactly when that page is actually needed rather than speculatively for the whole range up front. The first touch, not registration time, is when the cost lands.
Analogy
This is the same shape as a demand-paged TLB miss handler, just triggered by a network-visible access instead of a local one. That’s a latency-shape tradeoff, not a strictly better option: ODP is attractive when a registered range is large relative to what actually gets touched, or when registration-time latency matters more than a possible stall later; pinned registration is attractive when every page will be touched anyway and a predictable, fault-free access pattern matters more than registration speed. This chapter’s main treatment stays on the standard pinned path, consistent with this book’s current scope – ODP gets this one paragraph as a named alternative, not a second full treatment.
Gotcha
Implicit ODP’s whole point is covering a process’s entire address space with one registration – which means a single successfully-guessed or leaked rkey for an implicit-ODP region exposes the whole process, not one buffer. That’s a materially larger blast radius than the bounded-range case Permission and Bounds Violations covers next.
Memory Windows: a related but distinct mechanism, out of scope here. The Memory Window bind permission appeared in The Memory Region Descriptor’s access-flags list. Here’s precisely why it gets no further treatment in this chapter: a Memory Window is not a Memory Region. Where an MR is registered once, through the relatively heavyweight path this chapter just walked, and stays valid until explicitly deregistered, a Memory Window is a lightweight sub-scope carved out of an already-registered MR – its own rkey, its own narrower address range or access-rights subset, created and torn down through a cheap bind/rebind/invalidate operation rather than a full registration cycle. The use case is letting software narrow or revoke a remote peer’s access to part of an already-registered region without paying deregistration’s cost every time – a different tool built for a different job (fine-grained, frequent scope changes) than what this chapter’s registration path is built for (coarse-grained, infrequent). Memory Windows are scoped out of this book’s first pass the same deliberate way ODP’s implicit/explicit distinction and the newer flush-related permissions are – named here so a reader who knows the mechanism exists doesn’t wonder if it was simply missed.
Deregistration: removing the entry, not the responsibility. Deregistration reverses registration – the MPT/MTT entries are removed from the table (and from the Key Cache, if resident there) and the keys are retired. What it does not do is guarantee anything about operations already in flight that reference those keys, or protect against software that keeps using a retired key after the call returns; that’s Permission and Bounds Violations’s subject next, not this chapter’s.
Why
Pinning pages at registration time rather than defaulting every MR to ODP-style lazy resolution is a deliberate choice about where to put a cost that has to be paid somewhere. Upfront pinning makes every subsequent access to that region fault-free and its latency predictable – exactly what a performance-sensitive RDMA path wants, since a stall on the data path is precisely what registration exists to move out of the way in the first place. The cost is that registration itself becomes proportional to region size, paid whether or not the whole region ever actually gets touched. ODP inverts that trade: registration is cheap regardless of size, but the first-touch cost is deferred, not eliminated, and lands unpredictably wherever an access pattern first reaches a given page. Neither option is free; the choice is really about which latency a given workload can afford to have – a known one, paid once, or a smaller, spread-out one, paid on demand.
Registration and deregistration are what put keys into circulation and take them back out. What happens when a key that’s still in circulation gets presented for an operation it was never granted, or against an address range outside what it covers, is Permission and Bounds Violations’s subject next.
References
- InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications).
- RDMA man pages, memory-registration section (man7.org), for the implicit On-Demand Paging registration recipe – a software-interface convention, not an IBTA-V1 requirement.