The Memory Region Descriptor
This chapter starts with the Memory Region descriptor itself, field by field, before the next chapter covers the table structure holding every one of them.
Overview
A CPU never lets a running instruction touch physical memory directly – every virtual address it issues is translated and permission-checked by the MMU against page tables the OS maintains, on every single access, precisely so a process can’t reach memory it was never given. RDMA needs the same guarantee for a far less trusted case: a packet arriving from a remote machine, not a local instruction, asking to touch this machine’s physical memory directly, with no OS and no local CPU anywhere in that access’s path to check it. A Memory Region descriptor is what a successful registration produces to make that safe to allow at all: a record of one virtual address range, the physical pages backing it, and exactly which operations – local, remote, read, write, atomic – are allowed against it, functioning as the RNIC’s own equivalent of a page-table entry for exactly this one exposed range. Software gets back two keys pointing at that one record, not one, and getting precise about what those two keys actually are, and are not, is this chapter’s whole job.
Locator
Every field below lives inside one entry of The Key Table, which the next chapter covers – this chapter is about what’s in that entry, not the structure holding it.
Analogy
A Memory Region descriptor is the RNIC’s own equivalent of a page-table entry – the same job a CPU’s MMU does when it translates and permission-checks a virtual address against page tables the OS maintains, on every single access. The difference is what’s issuing the access: here it’s a packet arriving from a different machine entirely, checked against a key presented over the wire instead of a virtual address issued by a local instruction.
Block diagram
Deep walkthrough
Why this exists at all – the same job an MMU does for a CPU, checked a different way, and why skipping it would open a security hole. Under a conventional OS, a process’s virtual address space is exactly as private as the OS’s own page tables make it: no other process, and certainly no other machine, has any path to read or write it, because there is no hardware route by which an external actor could even reach in – every access is mediated by the CPU’s MMU checking permissions the OS alone controls, with a TLB caching the translations actually in active use. RDMA’s entire premise breaks that isolation on purpose: it lets a packet arriving from a different machine entirely reach directly into this machine’s physical memory, with no system call, no OS, and no local CPU instruction anywhere on that specific access’s path to check it. Left unchecked, that is not a feature – it is a remote-memory-disclosure vulnerability by construction, so the OS has to explicitly consent to it, once, in advance, for exactly the range being exposed and exactly the operations being permitted, rather than the exception being silently available to any packet that shows up.
Memory registration is that act of consent: a system call the application makes to its own OS, which pins the physical pages so they can’t move or be reclaimed out from under a remote reference, and instructs the adapter that this specific range, with these specific permissions, may now be reached from outside the process’s normal protection boundary – producing a record the adapter holds and checks against every future access the way an MMU holds and checks a page-table entry, just triggered by a key presented over the wire instead of a virtual address issued by a local instruction.
Get this wrong – register more than intended, grant looser permissions than intended, or fail to synchronize a deregistration against an operation still in flight (Mid-Transaction Deregistration covers that last one directly) – and the exact vulnerability this design exists to contain reappears, not through the OS’s normal enforcement (deliberately bypassed here, for sanctioned reasons) but through this one specific boundary the OS was asked to open.
This is also precisely the boundary a confidential-computing threat model has to reason hardest about: a workload running inside a memory-encrypted, hypervisor-untrusted enclave has to decide whether an RNIC sitting outside that trust boundary can be relied on to honor these same permissions – a question this book’s current scope doesn’t resolve, but one a reader coming from that background should recognize sitting exactly here. Everything below is what the record produced by registration’s one grant actually contains.
The full descriptor, and the field that’s easy to conflate with the two that matter. A registered Memory Region’s software-visible record carries: which device and Protection Domain this MR belongs to, the registered virtual range (address and length), a software-side identifier for the MR object itself, and the two hardware-visible keys, lkey and rkey.
Gotcha
The software-side identifier for the MR object is easy to fold into “the key” by accident – it isn’t. It identifies the MR object to the local software stack only: never presented on the wire, never checked by hardware against an incoming operation. Only lkey and rkey are the two things this chapter is actually about.
lkey and rkey – two keys, one entry, and they are not required to match.
Concept
lkey and rkey are two independently-assigned keys pointing at the same one entry – not a matched pair, not two views of one number. lkey is what the local process presents to reference its own memory; rkey is what a remote peer presents to reach the same memory. Nothing requires them to differ in value, only that they stay independently scoped: who may present which one, for what.
Every successful registration returns a unique lkey and a unique rkey, independently assigned, not two views of the same underlying number. The clean way to hold this: one MR descriptor, one entry in the Key Table the next chapter covers, and two differently-scoped keys pointing at it – lkey for the local process to reference its own memory in a Work Request it posts itself, rkey for a remote peer to present in an incoming RDMA operation against that same memory. Same entry, different callers, independently issued keys, not a matched pair that happens to carry two names. (RDMA’s iWARP lineage calls this same concept a “steering tag,” or STag, a single term covering what RoCE/IB splits into the local/remote lkey/rkey pair – useful to know if a vendor doc or paper uses that word instead.) Nothing in the standard requires the two values to differ, only that each be independently assigned and independently scoped – a specific implementation is free to make lkey and rkey numerically identical for a given registration as an internal shortcut, provided the two-key semantic split (who may present which one, for what) still holds. Assume the two keys are independent unless a specific device’s documentation says otherwise.
How much a rkey actually protects – a measured answer, not a guess. The design rationale below treats a rkey as a capability an untrusted remote peer has to actually possess – precisely how hard that possession is to come by matters, rather than assuming a 32-bit key space means 32 bits of protection. ReDMArk (Rothenberger, Taranov, Perrig, Hoefler, USENIX Security 2021), which directly measured key generation across real RNIC implementations, found none of them generate keys with anything close to full entropy: one hardware vendor’s implementation increments the key by a fixed 0x100 per registration (0 bits of entropy in the difference between successive keys – fully predictable); another hardware vendor’s older implementation is similarly weak (about 1 bit); a software-only implementation combines a fixed increment with an 8-bit LFSR whose seed and internal state are knowable (about 2.04 bits); even the strongest hardware implementation tested lands on one of two increment values more than 60% of the time (2.16-2.85 bits of entropy on the key difference, not the key itself). Every tested implementation’s rkey sequence is also strictly increasing, which is itself information: knowing one valid rkey narrows the search space for a neighboring registration’s rkey drastically below a 32-bit brute force.
Protection Domains don’t isolate as much as the field name implies. Among the fields making up a Memory Region’s descriptor is pd, and it’s tempting to read a PD as a hard isolation boundary between unrelated MRs. ReDMArk’s measurements say otherwise on two counts. First, the RDMA connection manager most applications use defaults to a single Protection Domain for every QP and every memory registration within one process – and the paper found that even developers using the lower-level native connection interface directly tend to converge on the same single-PD-per-process pattern in practice, rather than partitioning registrations across multiple PDs. Second, and more fundamental: the key generator itself is shared across PDs on a given device regardless of how many PDs an application actually uses – so even a deployment that carefully partitions its own registrations into separate PDs doesn’t gain protection against the entropy weakness above, because an attacker who obtains one rkey from any PD on that device can still predict rkeys minted for a different PD on the same device. A PD’s real isolation guarantee is narrower than its presence in the descriptor might suggest.
Food for Thought
If a single Protection Domain covers every registration in a process, and the key generator itself is shared across every PD on a device regardless of how an application partitions its own registrations – what is a Protection Domain actually protecting against? What would it take to make PD isolation real in the way its name implies?
Access flags – what the entry actually allows, checked at the point of use.
What: two different tiers, separated here rather than read as one flat list. IBTA-V1 itself defines the core tier at the architecture level – a Memory Region’s access control grants local write, remote write, remote read, and remote atomic access independently, each enable-able or not at registration, with remote write and remote atomic both requiring local write to already be enabled underneath them. That four-permission model is the standard’s, not a software convention – it’s what Memory Region / Key Table’s whole permission story (this chapter through Permission and Bounds Violations) actually rests on. A fifth permission belongs in this same standard-grounded tier: authorization to bind a Memory Window against this MR (Registering a Memory Region explains why that’s a distinct mechanism from the MR itself), since Memory Windows are themselves an IBTA-defined object, not a software invention layered on top.
A second tier exists, answering needs the standard never anticipated: real deployments have found reasons to extend what a registration can request beyond what IBTA-V1 itself defines – addressing conveniences this book doesn’t dwell on, and one permission that earns more than a name-drop. That one is an ordering relaxation: an optional, capability-gated permission a device has to be queried for before it can be requested, and it isn’t cosmetic – setting it permits the adapter to complete incoming network writes to this region out of order relative to each other, which voids write-after-write ordering an application might otherwise assume held (send-message ordering against a receive queue is unaffected; this is specifically about back-to-back network-initiated writes to the same region racing each other). It’s a correctness-relevant flag, not a tuning knob, and deserves reading closely rather than skimming past – and it’s confirmed absent from the IBTA specification text itself, a real answer to a real PCIe-ordering performance question the standard never posed. Deferred-resolution registration (ODP), similarly outside the standard, gets its own real treatment in Registering a Memory Region rather than a mention here. Newer flush-related permissions, oriented at persistent-memory access, are scoped out of this book’s first pass, the same deliberate way The Operation Catalog scoped
ATOMIC_WRITE/FLUSHout of the opcode catalog.Where this is checked: against the flags stamped into this same entry, whenever an operation presents this MR’s
lkey(locally) orrkey(remotely) and asks to do something with it.When it fails: an operation asking for something the flags don’t allow – a remote write against an MR registered without remote-write permission, say – is a permission violation. Permission and Bounds Violations is the chapter that owns everything downstream of that, in depth.
What happens next: this chapter’s job stops at naming what’s in the entry and what it permits; whether a specific access attempt against it is honored is the next few chapters’ subject, not this one’s.
Why
Issuing lkey and rkey as two independent values, rather than one shared key checked differently depending on who presents it, is what lets the two permission surfaces move independently. A local process’s own access to memory it registered itself is a much less sensitive question than what a remote peer – untrusted, on a different machine – is allowed to do to that same memory. Splitting the key means revoking or never granting the remote-facing one (rkey) doesn’t have to touch the local-facing one at all, and an application that only ever needs local access never has to hand out anything a remote peer could present. A single shared key would work too, checked against a caller-identity bit the hardware would then have to carry alongside every operation – independent keys sidestep needing that extra piece of state at all, at the cost of managing two numbers instead of one. What this design choice does not buy, per the measurements above: a remote peer is supposed to be authenticated by possession of a 32-bit rkey, but real drivers hand that key out with only a few bits of actual unpredictability, which is a hardware-adjacent design decision (how the key generator is seeded and advanced) sitting entirely outside what the two-key split itself was ever meant to solve.
The Key Table is next: the structure that holds one of these entries per registered region, and how a presented lkey or rkey finds the right one.
References
- InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications).
- J. Recio, B. Metzler, P. Culley, J. Hilland, D. Garcia, A Remote Direct Memory Access Protocol Specification, RFC 5040, IETF (October 2007) – defines “Steering Tag” (STag), the iWARP term mentioned above for this same local/remote key-split concept.