MR Capacity and Timing

Overview

The Key Table named a hard ceiling on registrations and moved on; Key Caching named a shared, cross-resource working-set pressure and moved on too. Neither chapter asked what a “maximum registered MR count” even means once a registration’s real cost depends on how large it is, not just that it exists. This closing chapter asks that question, then asks a second one this part has left completely open: how a slow, infrequent registration call and a fast, constant per-DMA key resolution end up touching the exact same structure safely – and, along the way, answers the open question Registering a Memory Region raised without resolving: whether a key can ever become resolvable before every MTT entry behind it actually exists.

Locator

Block diagram

Deep walkthrough

“How many MRs” is the wrong question to start with. The QP Table answers a flat question: one QP costs one table entry, so “maximum QP count” and “table size” are the same number. The Key Table doesn’t work that way. One MR costs one MPT entry plus however many MTT entries its registration spans – Registering a Memory Region already established that an MTT entry exists per physical page the registration covers, and that registration cost is proportional to region size, not a flat per-call charge. (Some real implementations likely reduce that per-page ratio for a large contiguous range by covering it with fewer, larger-granularity translation entries – a superpage-style optimization in the same spirit as a CPU’s own large-page TLB entries – but no source found pins down whether or how any specific RNIC supports that, so treat it as a plausible refinement, not an established fact.) A chip that can hold ten thousand small, single-page registrations might only hold a few dozen registrations the size of implicit ODP’s whole-address-space case before the same MTT storage runs out. There is no single “maximum MR count” independent of what those MRs actually look like – the real resource being spent is total MTT entries, and MR count is just one way of spending it.

The one ceiling that is flat: MPT entries. Permission-and-bounds metadata – address, length, access flags, per The Memory Region Descriptor – is a fixed-size record regardless of how many pages the registration covers, so the MPT half of this table does impose a flat ceiling on registration count, independent of size. A registration can fail two different ways against two different limits: out of MPT entries (too many distinct registrations, regardless of their size) or out of MTT entries (registrations too large in aggregate, regardless of how few there are). Software sees one failure either way – a rejected registration call – with no guarantee it can tell which limit it hit.

Where this connects to Queue Pair Management’s version of the same question. QP Capacity and Timing argued that a real max-QP number is the minimum across several hard, per-QP-allocated resources owned by different parts of this book, and that the QP Cache’s soft working-set bound doesn’t belong in that count at all. The Key Table side of that argument is narrower: nothing about what a memory region actually is – a target range, not a connection with its own send schedule, retry state, or injection rate – suggests it reserves a WQE ring slot, a Replay Buffer share, or a Rate Limiter entry the way a QP does. TX Pipeline, Reliability Engine, and Congestion Control Engine between them cover every other hard-allocated structure this book names, and none introduces a resource keyed to a live registration – WQE ring slots and Replay Buffer entries key to a live QP the same way Queue Pair Management’s do, and The Rate Limiter Table’s own addressing argument reasons its way to the identical QP-keyed granularity, not something registration-shaped. MPT and MTT entries remain the best-supported hard-ceiling story for this resource, consistent across every Part that’s since been drafted, though still this book’s own reasoned position rather than a source stating it as a settled architectural fact. What does cross into other territory is the soft side, and it crosses further than Queue Pair Management’s did: Key Caching already established, with real citations (the uRDMA paper’s cache model, corroborated in part by Kong et al.), that the Key Cache isn’t just conceptually similar to the QP Cache – it is the identical physical SRAM, shared with QP context, Completion Queue context, and Event Queue context all at once. A registered-MR population large enough to pressure that shared cache doesn’t only slow down other memory-region lookups; it degrades QP context lookups too, and the reverse holds just as this book’s Queue Pair Management chapter had to leave unconfirmed for lack of a source – Memory Region / Key Table’s own sourcing settles that question more directly than Queue Pair Management’s could.

Where this is checked, and the open question it finally answers. Nothing about capacity is checked per DMA. It’s checked at registration, against whichever of MPT or MTT entries runs out first, and the software-facing result is a failed registration call – the same shape The QP Table already described for its own entries. But Registering a Memory Region left a sharper question open: whether a key becomes externally resolvable before every MTT entry behind it is actually populated, since registration is a multi-step construction (page tables walked, pages pinned, one MPT entry written, then one or more MTT entries written behind it) – not a single atomic act. This chapter’s timing argument, below, answers that architecturally: no.

A second question this part has left completely open: timing. Nothing in this part has asked what clock a Key Table lookup runs on relative to the software call that populated the entry it’s reading. Those are two very different kinds of access to the same structure, and, as in Queue Pair Management, they don’t share a clock domain by default.

Two write paths – but only one of them writes on the hot path. Here the Key Table’s story diverges from the QP Table’s. A QP’s live state gets written by the datapath itself on nearly every packet – send and receive PSN counters advance continuously (QP Context). An MPT/MTT entry has no equivalent on the translation and permission payload itself: the DMA Engine’s Address Translation stage only ever reads that payload to authorize and route a transfer; nothing about a routine DMA writes back into the address, length, permission, or translation fields it consulted. Shared-MR Arbitration already flagged the one asterisk on this claim and left it open rather than resolving it: if an LRU-style eviction policy piggybacks recency metadata onto the same cache line, a read-only lookup does become a write to that metadata, even though the translation payload stays untouched – a question no source resolves, and this chapter doesn’t resolve it either. Setting that open bookkeeping question aside, the payload’s hot-path side of this crossing is simpler than Queue Pair Management’s – a pure read against the data that actually gates a DMA, with no read-modify-write race to reason about there. The control-plane side is where all the payload writing happens: registration populates an entry for the first time, and deregistration retires one, both arriving over a register or command interface almost certainly clocked independently of the DMA Engine’s own core datapath clock – architecturally the same control-plane-vs-datapath split QP Capacity and Timing already named, applied to a resource whose hot-path payload never talks back.

Why the answer to registration’s open question is no. A registration populating one MPT entry and however many MTT entries a large region requires is precisely the multi-bit, multi-write crossing QP Capacity and Timing already argued a plain synchronizer can’t safely carry, and it’s a harder version of that problem than a modify-QP call ever was – a modify-QP request writes one bounded attribute struct; a large registration writes a variable, potentially long chain of MTT entries one at a time. If any of those entries became visible to a concurrent lookup as soon as it landed, a DMA arriving mid-registration could resolve against a torn, partially-populated translation chain – the construction-side mirror of the teardown race Mid-Transaction Deregistration already covers in depth. No public source confirms this for a real RNIC, but the architecturally sound answer is almost certainly the same shape as Queue Pair Management’s: the whole chain – MPT entry and every MTT entry behind it – gets built in a staging area on the control-plane side of the boundary and only gets swapped into the structure the DMA Engine actually reads once construction is complete, through a handshake, not a bare synchronizer. A lookup that arrives before that swap simply doesn’t find the key yet, the ordinary and correct failure mode The Key Table already covers; nothing observes a half-built entry.

Pipeline depth, briefly. A Key Cache lookup is naturally at least two conditional stages, not one – The Key Table’s own MPT-then-MTT split, mirrored in Key Caching’s working set, means a permission check against the MPT entry has to resolve before it’s even meaningful to walk into MTT territory behind it. A rejected permission check can stop after the first stage; only a lookup that’s actually authorized to move data pays for the second. This book states that shape as reasoned inference about how a two-part structure like this is ordinarily pipelined, not a specific vendor’s cycle count; the DMA Engine’s own scatter-gather walk once a translation is in hand belongs to DMA Engine, forward-referenced here rather than re-derived.

This chapter closes Memory Region / Key Table. TX Pipeline is next.

References

• No IETF or IBTA standard governs per-MR resource capacity, silicon clocking, or pipeline staging – all of it is implementation-specific hardware design, not a wire-format or software-interface contract any standards body defines. This chapter’s capacity reasoning is grounded in the same architectural inference The Key Table, Key Caching, and Registering a Memory Region already used.
• Clifford E. Cummings, “Clock Domain Crossing (CDC) Design & Verification Techniques Using SystemVerilog,” SNUG (Synopsys Users Group), Boston, 2008 – the same general CDC methodology reference QP Capacity and Timing cites, applied here to a Key Table/Cache boundary instead of a QP Table/Cache one. Not an RNIC-specific source.