Shared-MR Arbitration
Overview
A memory region registered once and handed out to many QPs — a shared receive buffer is the natural example — faces the same shape of problem: it turns one MPT/MTT entry into a piece of state with multiple independent owners, all capable of hitting it concurrently, and a concurrent invalidation is this chapter’s version of the shootdown question – with no public source saying whether a real RNIC actually solves it the way a CPU does. That’s a different arbitration shape from anything Queue Pair Management covers, distinct enough to name rather than treat as a re-skin of a chapter already written. Reasoning through the shape gets somewhere useful even without a documented hardware answer: concurrent read-only lookups against one entry look, on the face of it, trivial to arbitrate, because a lookup doesn’t mutate the entry it reads. The hard case is a concurrent invalidation — an MW rebind, or a deregistration caught mid-execution having only removed some of the entry’s translations so far — racing against lookups still in flight against the old translation. That race’s specific hardware resolution is not documented anywhere public, and this chapter doesn’t invent one.
Locator
Permission and Bounds Violations covered what happens when one QP’s access to one MR entry is rejected. This chapter asks a different question: what happens when more than one QP is addressing that same entry at the same time, none of them necessarily doing anything wrong. This is confirmed thin territory — no source found describes how a real RNIC internally arbitrates simultaneous accesses from multiple QPs against one shared MR/key-table entry, and this chapter says so plainly rather than presenting reasoned analysis as documented fact.
Analogy
Multiple CPU cores can cache the same page-table entry in their own TLBs safely, right up until that entry changes – at which point the kernel has to shoot down every cached copy before the change is safe to rely on, because a core still executing against a stale translation is a genuine correctness hazard, not a footnote. A concurrent invalidation against a shared MPT/MTT entry is this chapter’s version of the same shootdown question.
Block diagram
Deep walkthrough
Why this isn’t Queue Pair Management’s multi-QP arbitration wearing a different label. Multi-QP Error Arbitration is about many QPs, each owning its own separate QP-context entry, all needing to report a fault through one shared downstream path at the same moment — many entries, one reporting channel, contention only at the funnel’s narrow end. This chapter’s shape is upstream of that and structurally different: many QPs are not each touching their own entry, they’re all touching the same single MPT/MTT entry directly, concurrently, as part of ordinary (non-faulting) operation. Many owners, one entry, not many entries, one reporting path. The two chapters are not answering the same question at different scales — one is about fan-in to a shared reporting structure after the fact, the other is about simultaneous access to one shared piece of state during the fact. That distinction matters, so a reader who’s just seen Queue Pair Management’s chapter doesn’t assume this one is the same problem restated.
The easy case: concurrent read-only lookups, scoped precisely to the translation data itself. A lookup — resolving an incoming rkey to its address range and permission bits to validate a remote operation — doesn’t write anything back into the translation or permission data the MPT/MTT entry holds. Multiple QPs’ lookups against the same entry, at the same instant, are each independently reading that same unchanging payload; nothing about one lookup can observe or be corrupted by another lookup happening in parallel against the same data. That claim is scoped deliberately to the payload, not to the entry’s every bit: if Key Cache Eviction Policy imports an LRU-style policy from Eviction Policy, recency metadata on that same entry needs updating on every touch, including read-only ones — a real write, even if the translation data itself never changes, and a real question about whether two QPs’ simultaneous lookups can update that metadata atomically. This chapter’s “no coordination needed” claim holds for the payload; it doesn’t extend to whatever bookkeeping an eviction policy piggybacks onto the same entry, and no source resolves that piece either. Payload contention aside, this case is trivial to arbitrate — whatever hardware mechanism exists for reading a cache line or table entry from multiple requesters at once (replicated read ports, a lookup pipeline that simply serializes reads with no side effects) is adequate by construction, without needing anything specific to the shared-MR case.
The hard case: a lookup racing an invalidation. The entry stops being read-only the moment something needs to change it while it’s live — a memory window rebind that repoints the same key at a new address range, or a single deregistration call caught partway through its own work, having only removed some of the translation entries an in-flight lookup might still be resolving against. A lookup that started before the invalidation and one that starts during or after it can, in principle, see different answers for what should be one consistent piece of state, and an in-flight lookup that’s already past the entry and mid-DMA when the invalidation lands raises the same question Mid-Transaction Deregistration takes up directly. Whether hardware serializes invalidation against all in-flight lookups on that entry (blocking new lookups, draining old ones, then applying the change), lets an in-flight lookup complete against a now-stale copy of the translation it already fetched, or does something else entirely is not stated anywhere public. This chapter doesn’t have a documented answer to hand the reader — only the shape of the question.
Where the deregistration side of this gets picked up properly. Mid-Transaction Deregistration is the chapter that takes the deregistration half of this exact race further — what a DMA engine holding a translation it already fetched is supposed to do when the entry backing that translation gets torn down mid-burst. This chapter’s job was narrower: establishing that the race exists the moment an MR has more than one concurrent owner, and that read-only concurrency is not the part of the problem this chapter needs to spend more words on.
Why
The reasoned split between “trivial” and “hard” here tracks a general principle that shows up anywhere multiple readers share one piece of mutable state: readers alone never need coordination, because nothing about a read can be observed by another read. The moment a writer (here, an invalidation) enters the picture, some coordination has to exist — the only choices are where that coordination cost gets paid. A design could pay it on every lookup (check a busy/pending-invalidate bit on every access, even the overwhelming majority that never race anything), or pay it only when an invalidation actually happens (accept that a lookup and an invalidation can interleave, and instead guarantee the invalidation itself waits for or fences off in-flight lookups before completing). The first makes the common case slightly slower to make the rare case simple to reason about; the second keeps the common case free but pushes the complexity onto the invalidation path specifically — which is a smaller, less latency-sensitive piece of the datapath to spend that complexity on. Neither is documented as correct; this is analysis, not a report of what any real RNIC does.
Key Cache Eviction Policy leaves shared ownership behind and returns to the numbers from Key Caching: what decides which cached translation gets evicted when the working set runs out of room.
References
- No IETF or IBTA standard governs concurrent-lookup-vs-invalidation arbitration on one key-table entry – it is implementation-specific silicon design, not a wire-format or software-interface contract any standards body defines. This chapter’s treatment is grounded in the sources cited in-text above, not a standards document.