Mid-Transaction Deregistration
Overview
Mid-Transaction QP Reset closed Queue Pair Management on the same shape of question this chapter closes Memory Region / Key Table with: a resource torn down while something is still actively using it. There, the resource was a Queue Pair and the thing still in flight was a packet, a replay-buffer entry, a partially-reassembled message. Here, the resource is a Memory Region and the thing still in flight is a DMA burst that has already resolved an rkey against the key table and started moving bytes, at the exact moment software deregisters that same MR on the other end. Same shape, different resource, and — unlike Queue Pair Management’s version — one sourced anchor to start from before the chapter has to reason on its own.
Locator
Block diagram
Deep walkthrough
What the contract actually says. RDMAmojo’s documentation for deregistration is direct about where responsibility sits: “It is up to the user to stop using the keys (lkey and rkey) that are associated with this MR. If those keys will be used in local Work Requests or by remote operation requests, it will lead to Work Completion errors.” That sentence is this chapter’s one solid factual anchor. Here is precisely what it does and doesn’t establish. It establishes that avoiding a stale-key access is a software obligation — synchronize with the remote peer, drain outstanding operations, then deregister — not a condition hardware is documented to detect and safely reject on its own. It does not establish what a stale-key access actually produces at the hardware level if the contract is violated anyway. RDMAmojo names “Work Completion errors” as the consequence, but a specific completion status for this exact case — a rejection distinguishable from any other remote-access error, versus something less well-defined — is not stated anywhere found. Permission and Bounds Violations flagged the same gap for the general permission/bounds case; it applies here too, sharpened by timing.
The specific race this chapter is about. Permission and bounds violations are static: an incoming operation’s rkey resolves to a still-valid entry, and the check is whether the requested access matches that entry’s permissions and range. Mid-transaction deregistration is dynamic: the entry itself stops being valid while a resolved operation against it is still executing. A remote RDMA write’s rkey is checked once, early in the operation — the RETH carries it, the responder’s RNIC (the target whose memory the write is landing in, and whose software later runs the deregistration this chapter is about) looks it up against its own MPT/MTT, and if the lookup succeeds, the operation proceeds to move data. That lookup and the DMA burst it authorizes are not necessarily the same instant. A large write is not a single atomic act — it is a burst, plausibly split across multiple internal transfers, occupying some nonzero span of time on the PCIe/memory side after the rkey check has already passed. If deregistration runs on the local software stack during that span, the MPT/MTT entry the burst is still relying on can be invalidated out from under it — and per Key Caching’s sourced finding from Kong et al., deregistration is suspected to invalidate the entire MTT/MPT cache’s working set, not just the one entry being torn down, which makes the blast radius of a badly-timed deregistration considerably larger than “this one region becomes unavailable.”
What a hardware implementation has to decide, reasoned honestly. Two shapes of answer are available, and nothing public commits to either for a real RNIC. One: the DMA engine, once it has validated an rkey and begun a burst, holds a private, already-resolved copy of the translation for the duration of that burst — the lookup happens once, the burst executes against a local copy, and a concurrent invalidation of the shared MPT/MTT entry (or the whole cache) simply doesn’t reach an operation that’s already past the check. This is the same shape of guarantee DMA Engine’s DMA Engine chapter and this part’s caching chapters would need to describe explicitly if either committed to it, and neither does. Two: there is no such private copy, the DMA engine re-touches the shared table (or a cache line backed by it) at some granularity during a long burst — per sub-transfer, say — and a concurrent invalidation lands inside a live burst’s execution window, producing a genuine race whose outcome (the burst faults partway through, silently continues against stale metadata, or something else) is undefined by any source this book found. Both are physically plausible designs. A private per-burst copy is the safer answer and the one that would make RDMAmojo’s software contract sufficient in practice — software staying disciplined about starting a deregistration is enough, because hardware then finishes what it already started cleanly. A no-copy design would mean the software contract is necessary but not obviously sufficient — correct sequencing avoids triggering the race, but doesn’t by itself prove there’s no race to trigger if timing goes wrong. This book cannot resolve which of the two is true for any real RNIC, and says so plainly rather than picking the more comforting answer.
Food for Thought
A private per-burst translation copy is the safer design; re-touching the shared table costs nothing extra in dedicated storage. If you were sizing the private-copy option, what would drive how much buffer space to spend – typical burst length, expected deregistration frequency, something else? At what point does “safer” stop being worth its silicon cost?
Where this leaves the reader. The pattern is identical to Mid-Transaction QP Reset’s conclusion: the problem is real and, unusually for this part’s thinner corners, acknowledged in writing at the API-contract level. The resolution mechanics — what the DMA engine actually does when the table it depends on changes underneath an operation already in progress — sit past the edge of what any public source documents, for this resource exactly as much as for the QP.
Why
The honest way to reason about which of the two shapes above a real design would pick is to ask what a private per-burst translation copy would cost, since “always safer” isn’t free. Holding a resolved translation locally for the life of a burst means the DMA engine needs somewhere to put it — a small per-in-flight-operation register or buffer, sized to whatever the maximum number of concurrent bursts the engine supports, not a large structure, but not nothing either, and it has to be populated at exactly the right moment (after the rkey check passes, before the burst can be preempted by anything else touching the shared table). The alternative — re-deriving the translation from the shared table at intervals during a long burst — costs nothing extra in dedicated storage, but reopens the same table on every re-touch to whatever churn is happening on it concurrently, including a deregistration event that Key Caching already showed is disproportionately disruptive to that table’s cache. Neither option is obviously dominant in the abstract; a design’s actual choice would depend on typical burst length relative to expected deregistration frequency, and on how much silicon budget the vendor is willing to spend guaranteeing an invariant (in-flight operations are immune to concurrent table changes) that the verbs-layer software contract already tells applications not to rely on triggering in the first place. That last point cuts both ways: RDMAmojo’s contract makes a hardware guarantee less necessary for correctness of well-behaved software, which is exactly the kind of pressure that could push a real vendor toward the cheaper, race-permitting design rather than the safer one — this book has no way to know which way any specific vendor actually went.
This is not a new class of problem. Mid-Transaction QP Reset — Queue Pair Management’s closing chapter — covers the same shape of hazard against a different resource: a Queue Pair reset with a packet, a replay-buffer entry, and a DMA write all still in flight against it, versus this chapter’s Memory Region deregistered with a DMA burst still in flight against its key. Neither chapter can point at a documented resolution mechanism; both can point at exactly what has to be true for the outcome to be correct, and name the race that a real implementation has had to close one way or another.
This is the last chapter in this part built around one Memory Region at a time. MR Capacity and Timing, next, closes Memory Region / Key Table by asking how many of these can exist together, and what it costs in silicon to let them share the same structure safely.
References
- InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications).
- No IETF or IBTA standard governs the exact hardware sequencing of a mid-transaction deregistration race (as opposed to the deregistration software contract itself, which is a verbs-API convention, not an IBTA-V1 wire/state requirement) – it is implementation-specific silicon design, not a wire-format or software-interface contract any standards body defines. This chapter’s treatment is grounded in the sources cited in-text above, not a standards document.