Single-Burst Fault Handling

Overview

Address Translation Handoff already drew the line precisely: a bad key, an out-of-range address, or a disallowed operation type is Permission and Bounds Violations’s fault, caught before this engine ever begins moving bytes. What’s left, once a transfer clears that gate and actually starts running, is narrower and genuinely this engine’s own: one expected, recoverable stall, and one real, unrecoverable fault.

Locator

Block diagram

All 10 steps

1. First touch: a pause. A 24 KiB SEND arrives and is scattered into a receive buffer spanning six pages. The buffer is registered with On-Demand Paging, and page 3 has never been touched, so it isn’t mapped yet. Single-Burst Fault Handling
2. Pages 1 and 2 are written. The engine’s in-flight state moves forward with them. In-Flight Transfer State
3. The scatter reaches page 3 and finds no mapping. The transfer pauses and hands the page to the OS’s own page-fault handling. Registering a Memory Region
4. The OS maps page 3 and the transfer is released. Single-Burst Fault Handling
5. No. Current address and bytes left say exactly where it stopped, so it picks up at page 3 and writes pages 3 to 6. In-Flight Transfer State
6. The receive WQE completes with success. The first touch cost latency and nothing else: no error and no error completion. Single-Burst Fault Handling
7. Memory failure: an abort. The same SEND again, with every page already mapped. Pages 1 to 3 are written. Single-Burst Fault Handling
8. The write to page 4 fails: an uncorrectable host-memory error. This isn’t a missing mapping, so there is nothing to wait for and nothing to resume. Single-Burst Fault Handling
9. Nothing. No source this book found describes an undo, so the first 12 KiB stay where they landed. A partial transfer stays partial. Single-Burst Fault Handling
10. The receive WQE completes in error. If this counts as IBTA’s Local QP Error, as the chapter argues, the following WQEs are flushed and the QP moves to ERROR. The CQE says the SEND failed, not how much of it landed. Single-QP Error Handling

Deep walkthrough

The expected case first: an On-Demand-Paging first touch is a stall, not a fault. Registering a Memory Region already covers this precisely – an ODP-registered range defers page pinning, and “the first access to a given page triggers a page-fault-style resolution, handled cooperatively by hardware and the OS against the OS’s own page tables.” When this engine’s own gather or scatter reaches such a page for the first time, that resolution is what runs. In-Flight Transfer State’s current address and remaining length are exactly what let the transfer resume where it paused rather than restart – the same state this chapter’s genuine-fault case, below, has no equivalent recovery for.

The real fault: a host-memory access that simply fails. An authorized, already-translated transfer can still fail on the memory side – an uncorrectable error, or a memory-fabric access that never returns a good result. This is the one genuinely new fault class this Part contributes. Single-QP Error Handling names three triggers in total, but only two are packet-triggered and comparable to this one: an invalid-request error (a wire-level violation in an arriving packet) and an access-violation error (a bad key, bounds, or permission, caught before execution) – both about what a packet asked for. Its third, a QP-fatal event cascading from a shared Completion Queue or Shared Receive Queue’s own fault, is a different shape entirely – not a property of any one operation, but a consequence one QP’s own error inflicts on every other QP sharing that resource – and this comparison isn’t reaching for it. What this chapter adds is narrower than any of the three: whether the memory this engine was correctly asked to touch actually cooperated once execution began – a distinct failure surface, local to this side’s own memory subsystem, with no wire-level or permission-level cause at all.

IBTA-V1’s own fault taxonomy has a real, if not perfectly named, home for this: Table 58’s “Local QP Error” (Responder Class A, reported as NAK-Remote Operational Error) is defined in general terms broad enough to cover it – “a local QP related error while executing the request message… [that] prevented the responder from completing the request.” That general clause isn’t scoped to any one cause; the specification’s own concrete example under that heading is a receive queue failing to fetch a WQE from a Shared Receive Queue, not a host-memory access failure. This chapter’s classification is therefore a plausible fit against the general definition, not a confirmed instance of the specification’s own named example – worth stating precisely rather than either overclaiming a direct citation or, as an earlier draft of this chapter did, understating that any source addresses this at all.

No rollback – a partial transfer stays partial. Nothing this book found describes an undo mechanism for bytes a burst already placed before the fault that ends it. The honest, reasoned position: a multi-segment transfer that fails partway through leaves whatever it already wrote in place, unrolled by nothing, and reports failure for the operation as a whole regardless of how much of it actually succeeded. An application that needs to know precisely how much landed before a mid-burst failure has no completion-status field this book found that tells it – the completion says the operation failed, not where.

Confirmed, conditional on that classification: this fault forces the owning QP to ERROR. Single-QP Error Handling’s flush-and-drain obligation exists precisely so a faulted operation always terminates with a real completion rather than a hang, and if this fault is indeed IBTA-V1’s Local QP Error, Table 59 removes the guesswork: Responder Class A’s own compliance behavior is “current receive WQE completed in error, subsequent flushed, queue to error state” – the identical shape Single-QP Error Handling already describes for its own two packet-triggered classes. What remains genuinely unconfirmed is only the classification itself, not what happens once it’s made – the QP-to-ERROR consequence is mandatory for Local QP Error however this specific memory-side condition ends up counted.

Where this connects. A faulted transfer’s completion is CQE Generation’s to write, carrying whatever error status this class resolves to. What happens when this fault hits a burst attributed to a QP that’s simultaneously being reset out from under it – a narrower, timing-specific version of the same underlying question – is Mid-Transaction Burst Teardown’s subject, not this chapter’s.

Interface reference

References

• Registering a Memory Region and its RDMA man-pages citation for On-Demand Paging’s first-touch behavior – reused here, not re-derived.
• InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications) – Table 58’s “Local QP Error” (Responder Class A, NAK-Remote Operational Error) is defined broadly enough to plausibly cover this chapter’s host-memory-execution fault class, and Table 59 confirms its mandatory QP-to-ERROR, flush-and-drain consequence. The specification’s own only named example under that heading is a Shared Receive Queue WQE-fetch failure, not a host-memory access failure – this chapter’s classification is a plausible fit against the general definition, not a confirmed instance of the named example, and is stated with that precision rather than as either an unsourced claim or a settled one.