Single-QP Error Handling
Overview
A QP that reaches ERROR does not stop processing — it starts flushing.
Analogy
A CPU exception or branch misprediction doesn’t strand the instructions still in flight behind it in limbo – it squashes them, in order, each with a defined outcome, rather than letting them vanish silently or complete against a state that’s no longer valid. A QP in ERROR does the same thing for its own in-flight work. Every work request already posted and not yet completed, and every work request posted afterward, drains through the completion queue carrying a status that says the operation never happened, until nothing is left in either queue. Nothing about this touches any other QP on the same adapter: ERROR is scoped to exactly the QP that faulted, by construction, and that isolation guarantee is this chapter’s real subject.
Locator
QP State Machine covered how a QP reaches ERROR and named this chapter as the place that explains what ERROR actually does to a QP’s outstanding work. This is that chapter — for exactly one QP, in isolation. Multi-QP Error Arbitration picks up immediately after with the harder question of many QPs faulting together.
Block diagram
Deep walkthrough
What a flush actually is. Once a QP is in ERROR, hardware doesn’t discard outstanding work silently — it completes every one of it, in order, with a flush-in-error completion status instead of a normal success status. This includes work already posted and awaiting processing when the fault occurred, and work posted afterward, for as long as the application keeps posting to a QP it hasn’t yet noticed is dead. A flush is not a hardware fault of its own; it’s the QP telling software, one completion at a time, “this didn’t happen, and neither will anything you post from here.” Draining continues until both queues are empty, at which point the QP is fully quiesced in ERROR and waiting for a RESET.
Two kinds of fault, and why the distinction matters. Be precise about which layer each name below actually belongs to: the underlying condition – a transport-layer violation in an arriving packet, or a permission/bounds/resource violation on an operation – is wire-level and standard-defined, the same AETH-syndrome territory Permission and Bounds Violations covers in depth; the two local fault-event classes named below are how each condition surfaces locally, as an asynchronous event on the responder’s own QP, not the origin of the classification itself. Not every path into ERROR means the same thing about what went wrong. An invalid-request error (the verbs event IBV_EVENT_QP_REQ_ERR, where “REQ” names the incoming request, not the requester side) fires when this QP, acting as responder, detects a transport-layer violation in a request arriving for it — an unsupported or out-of-sequence opcode. It’s RC-only, and is rare enough in practice to be a sign of a problem in the subnet, or an RDMA device sending illegal packets — something is wrong with what’s arriving, not with how this QP was set up. An access-violation error is a different animal entirely: a misaligned atomic request, more outstanding RDMA read/atomic requests than the responder-side ceiling allows, a remote-key violation, or a length error on an operation without immediate data. This class usually reflects a bug in the requesting application’s own usage, not the network or the peer doing anything wrong. Both auto-transition the QP to ERROR identically from hardware’s side – confirmed directly, not just observed as a pattern: IBTA-V1’s Table 58 (Responder Error Behavior Summary) maps every condition named in both groups above, for RC, to the identical fault class (Responder Class C – current WQE completed in error, subsequent WQEs flushed, queue to error state). The distinction this chapter draws between the two groups is entirely about where a diagnosing engineer should look afterward, protocol/wire problem versus local-programming problem, not about how the QP itself behaves once it’s there – the standard itself draws no behavioral line between them at all.
Isolation is the property, not an accident. Nothing in either error path touches a sibling QP on the same adapter. The flush behavior, the ERROR state itself, and the eventual RESET-to-recover path are all scoped to the one QP context that faulted — a second QP carrying on normal RTS traffic through the exact same TX/RX pipeline stages, cached in the exact same QP Cache, keeps running without observing anything. That isolation is precisely the property Multi-QP Error Arbitration complicates: isolation between two QPs faulting independently is one problem; many QPs faulting from one shared cause, all needing to report through the same channel at once, is a different one.
A third trigger, and the one that doesn’t stay single-QP. A QP-fatal event is more than a generic “something prevented further completions” catch-all – it has a cause the other two triggers don’t have: an error on a Completion Queue or a Shared Receive Queue this QP is associated with. When a CQ or SRQ itself hits an error, every QP associated with it – not just the one that happened to trigger the condition – transitions to ERROR and fires its own QP-fatal event, simultaneously. The SRQ half of this is directly confirmed: IBTA-V1’s Table 58 names “Local QP Error” (Responder Class A) explicitly as including the case where “a receive queue which is associated with a shared receive queue is unable to fetch a WQE from the shared receive queue due to an error condition in the shared receive queue.” The CQ half – a CQ error alone, with no SRQ involved – isn’t confirmed by that same table entry; it’s sourced separately (see CQ Overflow, which documents one concrete way a CQ itself reaches an error condition that cascades outward this way). This chapter’s isolation story still holds for the requester-side and access-violation triggers, both scoped to the one QP that faulted; the QP-fatal trigger via a shared CQ/SRQ is the one documented, concrete mechanism by which a single upstream cause takes out many QPs at the exact same instant. Multi-QP Error Arbitration is where that scenario actually gets reasoned through.
SRQ-associated QPs need one more signal before teardown. A QP that shares a Shared Receive Queue with other QPs can’t simply be destroyed the moment it reaches ERROR — some of its outstanding receive-side work may still reference WQEs pulled from that shared queue, and destroying the QP before that work finishes draining risks leaving the SRQ itself in an inconsistent state for the QPs still using it. A dedicated last-SRQ-WQE-consumed event is the signal that closes this gap: it fires once this QP has consumed the last SRQ-sourced WQE it will ever consume, and it’s the event software waits for before it’s safe to tear the QP down without disturbing its SRQ siblings.
All 8 steps
- One QP faults. Three QPs share one Shared Receive Queue. QP 1 has 3 send WQEs outstanding and has already pulled 2 receive WQEs from the SRQ. Single-QP Error Handling
- QP 1’s first outstanding send fails, a remote access error, and QP 1 moves to ERROR. Nothing outside QP 1 changes state. Single-QP Error Handling
- No. Each one gets a CQE, in order: an error status for the send that failed, and a flush status for the two behind it. Every WR the application posted ends with a completion. Single-QP Error Handling
- The receive side flushes the same way: the 2 WQEs QP 1 already took from the SRQ complete with a flush status. Single-QP Error Handling
- QP 1 will never take another WQE from the SRQ, so the adapter raises last WQE reached. Destroying an SRQ-attached QP before this event can leave the shared queue inconsistent for QP 2 and QP 3. Single-QP Error Handling
- Meanwhile QP 2 keeps taking receive WQEs from the same SRQ, through the same pipeline. One QP’s fault stays inside that QP. Single-QP Error Handling
- The shared queue faults. Every QP attached to it, at once. IBTA-V1 Table 58 counts a receive queue that can’t fetch from its SRQ as a Local QP Error, so QP 2 and QP 3 both move to ERROR and each raises its own QP-fatal event. Single-QP Error Handling
- QP 2 and QP 3 each flush to their own CQs the way QP 1 did, and each raises last WQE reached before it is safe to tear down. A fault in a shared structure is the one path out of single-QP isolation. Single-QP Error Handling
Why
Flushing every outstanding operation with an explicit status, rather than simply halting the QP and leaving posted-but-unprocessed work in limbo, is what makes fault recovery a software-observable, boundedly-long process instead of an open question the application has to poll for indefinitely. It costs hardware a real obligation — walk both queues to completion, in order, generating a real CQE for each entry, even though none of them succeeded — instead of just setting a flag and stopping. That cost buys the application a guarantee: post-then-poll always terminates, one way or another, for every request it ever issued, fault or no fault. The alternative (silently dropping queued work with no completion at all) would turn every fault into a hang the application has no event to wake up on.
One QP faulting is a contained, well-defined event. Multi-QP Error Arbitration asks what happens when the isolation boundary this chapter just described has to hold under many simultaneous faults instead of one.
References
- InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications) – Table 58 (Responder Error Behavior Summary) confirms this chapter’s requester-side-transport-error and access-violation-error groups both map to the identical fault class (Responder Class C, RC) despite their different underlying causes, and its “Local QP Error” row (Responder Class A) directly names the SRQ-WQE-fetch-failure case this chapter’s QP-fatal trigger describes.