Permission and Bounds Violations

Overview

Two distinct real-world conditions are this chapter’s main subject: a bounds violation, where the address-plus-length an incoming RDMA operation names falls outside the MR’s registered range, and a permission violation, where the operation type itself — a remote write, say — isn’t enabled by the flags the MR was registered with (an MR registered with only remote-read permission has no obligation to accept a remote write). Both are named, distinct conditions, not one vague “bad remote access” idea. What’s undocumented – and possibly nonexistent – is a hardware distinction between them: every source available treats a rejected remote access as one umbrella status, regardless of which of the two conditions actually triggered it. This chapter says that plainly rather than inventing a check order the specification doesn’t commit to. A third condition closes out the chapter: a local Work Queue Element’s own lkey failing the same kind of check, entirely on this side, before anything reaches the wire at all.

Locator

Registering a Memory Region covered how an MR gets its base address, length, and access-permission bits installed into the key table in the first place. This chapter is about what happens when an already-registered entry gets looked up by an incoming remote operation that doesn’t fit it — either the address and length it names fall outside what was registered, or the operation type it’s asking for isn’t one the registration allowed.

Block diagram

Deep walkthrough

The two conditions, named precisely. A bounds violation is an addressing problem: the incoming operation’s virtual address and length, taken together, reach past either end of the range the MR registered — reading or writing bytes the application never handed to registration in the first place. A permission violation is an authorization problem on an address range that’s otherwise entirely valid: the bytes are inside the MR, but the requested operation isn’t one the registration’s access flags enabled — a remote write against remote-read-only memory, a remote atomic against an MR that never enabled remote-atomic permission. These are conceptually independent failures. An operation can violate bounds without touching permissions at all (a well-permissioned MR, wrong offset), and it can violate permissions on a perfectly in-range address (the right bytes, the wrong verb).

The underlying fact is wire-level, not software-level. IBTA-V1’s AETH syndrome encoding is what actually classifies a rejected remote access at the point the NAK carrying it crosses back to the requester – a standard-defined condition, not a software invention. The responder’s own local access-error event and the requester’s own remote-access-error completion status, named below, are simply how that same standard-defined condition surfaces locally on each side of the connection – not the origin of the classification itself.

One umbrella status covers both, confirmed directly against IBTA-V1’s own fault-class table, not inferred from its absence. IBTA-V1 §9.7.5.2.4 defines a single condition, Remote Access Error, as covering all of: an invalid R_Key, an address-and-length-or-type-of-access outside the R_Key’s registered bound, and a protection-domain violation – three causes that span both this chapter’s bounds and permission conditions (the second cause alone bundles both: an out-of-range address is the bounds half, a disallowed access type is the permission half), folded into one wire-level classification with one NAK code. Table 58 (Responder Error Behavior Summary) carries that through to the fault-class level: an RC R_Key Violation maps to exactly one NAK code (Remote Access Error) and one fault class (Responder Class C), with no finer-grained split for which of the three underlying causes actually fired. The completion status the requester’s QP sees is the same story from the other side of the wire: one status, covering the same class of remote-side rejection, regardless of which specific condition the responder’s hardware actually tripped on.

This is a two-sided event, not a one-QP fault. Single-QP Error Handling is about one QP’s own local fault and its own local flush. A permission or bounds violation isn’t that shape: the check happens on the responder’s side, against the responder’s MR, and it’s the responder’s own QP that fires the local access-error event and auto-transitions to ERROR — at which point everything that chapter already covers about flushing outstanding work with a flush-status completion applies to the responder’s QP exactly as written. But the requester — a different QP, possibly on a different machine entirely — never sees that event. What it sees is a NAK crossing the wire and a remote-access-error completion on its own send queue. Two QPs, two local artifacts, one underlying violation. The single-QP chapter’s framing doesn’t stand in for this one.

Here is that event on the wire, for a WRITE against a read-only MR, with both QPs ending in ERROR:

All 7 steps

1. The requester sends a WRITE whose rkey names an MR on the responder. That MR was registered with remote read permission only. The Memory Region Descriptor
2. PSN 1000 is the one expected, so the packet passes the sequence gate. PSN Check and Out-of-Order Arrival
3. No. The MR exists, but its access flags leave out remote write. A permission failure and a bounds failure are reported the same way, and nothing is written. Permission and Bounds Violations
4. The responder NAKs PSN 1000 with code 2, Remote Access Error. Permission and Bounds Violations
5. In the same moment, the responder’s own QP raises a local access-error event and moves to ERROR; it does not wait for the NAK to arrive. Single-QP Error Handling
6. No. Only sequence-error and RNR NAKs are retried. The WQE completes at once with a remote access error status. The Retry and RNR Timeout State Machines
7. The requester’s QP moves to ERROR too, and every WQE posted after this one completes as flushed. One bad rkey has stopped both ends of the connection. Single-QP Error Handling

The local half of this chapter’s own title: a violation that never reaches the wire at all. Everything above is the remote half of permission and bounds checking – a peer’s incoming operation, validated against the responder’s own MR. The Memory Region Descriptor already established lkey and rkey as independently-scoped keys for the same registration, and a local Work Queue Element’s own scatter-gather list entries carry an lkey, not an rkey – the source buffer for a SEND or WRITE’s local gather, or the destination buffer a RECV or an RDMA READ response writes into. IBTA-V1 names this failure mode directly and separately from everything above it: a Local Memory Protection Error, “an implementation specific memory protection error in [the requester’s] own local memory subsystem” (Table 56), reported as Requester Class B – the same immediate, no-retry, current-WQE-completed-in-error terminal shape The Retry and RNR Timeout State Machines’s own direct path already covers for wire-level faults, reached here by an entirely local route instead.

Why this is a one-QP event, the mirror image of the two-sided story above. A bad lkey never produces a packet at all – the local channel interface validates the scatter-gather list’s own keys before a WQE’s bytes ever reach the wire, the same before-the-wire timing this book’s own address-translation chapters already establish for every local lookup this engine performs. There is no NAK, no remote-side event, no peer that ever learns an invalid access was attempted, because nothing crossed the connection in the first place. Where a remote rkey violation is unavoidably two QPs’ business – the responder’s local fault and the requester’s remote-access-error completion, as above – a local lkey violation is entirely this side’s own problem, caught and reported without the peer’s involvement at any point.

No documented check order, and no invented one here. Does hardware check bounds first and only evaluate permissions on an in-range address, or the reverse? No source found states an order, and none states that the two are even implemented as separate stages rather than one combined table-entry validation that either passes or doesn’t. Given that the only software-visible signal either way is the same umbrella status, there’s no observable behavior that would let an application (or this book) infer an order empirically. The honest position: this is an implementation detail the spec doesn’t mandate a visible order for, and this chapter isn’t going to assert one anyway.

Stale keys are a software problem, not a hardware-guaranteed-safe one. A related but distinct failure mode: a remote peer holding an rkey to an MR that’s since been deregistered. RDMAmojo’s documentation for deregistration is explicit that this is the application’s problem to avoid, not hardware’s problem to catch: “It is up to the user to stop using the keys (lkey and rkey) that are associated with this MR. If those keys will be used in local Work Requests or by remote operation requests, it will lead to Work Completion errors.” That’s a software contract, stated plainly — synchronize before deregistering — not a claim that hardware detects every possible stale-key race and rejects it cleanly. What a stale-key access actually produces at the hardware level, in a badly-sequenced race where deregistration and a remote access to the same rkey happen close together, isn’t stated anywhere found. Mid-Transaction Deregistration picks this exact race up in depth; this chapter’s job is just naming it as real and distinct from the bounds/permission checks above, not re-deriving it.

A bad rkey isn’t always a bug or a race – it can be an attacker’s best guess. Everything above frames a rejected remote access as originating from a stale key or a software mistake. There’s a third origin: The Memory Region Descriptor already covers ReDMArk’s finding that real drivers generate rkeys with only a few bits of measured entropy, sequentially – which means an attacker who has legitimately registered one memory region on a shared host can predict the rkey values neighboring registrations were assigned, and combine that with the fact that RDMA-accessible buffers for different clients are typically allocated in close proximity (consecutive chunks of one larger region, not scattered by ASLR the way normal heap allocations are) to guess both the address and the key of another client’s buffer. ReDMArk documents this as a demonstrated attack (its “A3: Unauthorized Memory Access”), not a theoretical one. From this chapter’s own hardware perspective, a guessed-and-correct rkey presented against a currently-valid MPT entry with matching permissions and range is indistinguishable from a legitimate access – there is no hardware signal at all for “this key was guessed rather than possessed legitimately,” which is exactly why the entropy question belongs in the threat model at all, not just the permission/bounds logic this chapter otherwise documents.

The two conditions this chapter named are what happens when one QP’s access to one MR is rejected. Shared-MR Arbitration asks the harder question underneath that: what happens when many QPs are addressing that same MR at once, and one of them is trying to invalidate it while the others are still mid-lookup.

References

• InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications) – §9.7.5.2.4 (Remote Access Error) defines the single wire-level condition covering both this chapter’s bounds and permission failures, including the access-type clause that names the permission half specifically; Table 58 (Responder Error Behavior Summary) confirms it maps to one NAK code and one fault class (Responder Class C for RC), with no documented finer-grained split and no documented check order between the two conditions. Table 56 (Requester Side Error Behavior) is the source for the Local Memory Protection Error fault class this chapter’s local-lkey section covers, reported as Requester Class B.