CQ Overflow

Overview

Replay Buffer Capacity and Backpressure already established that some resources can’t answer a capacity question by evicting something – an unacknowledged replay entry is unsafe to sacrifice, so hardware enforces admission control instead, refusing to transmit past what its Replay Buffer can hold. A Completion Queue has no equivalent gate: nothing stops a QP from finishing an operation the instant its CQ is already full. What happens next is not left to hardware’s discretion – IBTA-V1 mandates a specific consequence, and it’s a real one: the affected QP is driven to ERROR.

Locator

The Completion Queue already named the trigger: a producer index that’s caught all the way up to the consumer index, with nowhere left to write a new entry. This chapter is what happens when that trigger actually fires.

Block diagram

All 8 steps

1. The burst. CQ 1 is empty and the application has just polled it. Each QP has 4 Work Requests outstanding. Their ceilings add to 12 and the CQ has room for 8: fine at the usual completion rate, but not if they all complete at once. CQ Overflow
2. QP A hits a remote access error and moves to ERROR. Its queue is flushed: one error CQE for the faulting WR and a flush CQE for each WR behind it, whether or not they were signaled. CQE Generation
3. QP B’s four Work Requests complete normally, one after another, before the application gets round to polling. Their CQEs take the last four slots. CQ 1 is full. The Completion Queue
4. Nowhere. The completion attempt finds CQ 1 full. QP C did nothing wrong, but it is the QP whose completion couldn’t be written. CQ Overflow
5. IBTA C9-218 (C9-225 on the responder side) is mandatory here: the affected QP, QP C, moves to ERROR and an asynchronous error is raised. The WQE that tried to complete, and every one after it, is left in an unknown state. CQ Overflow
6. QP C in ERROR flushes too, and its flush CQEs are aimed at the CQ that just ran out of room. The standard says only “unknown”. This book’s inference from the ring: a producer that keeps writing overwrites CQEs the application hasn’t read yet. Single-QP Error Handling
7. What stays contained. QP D reports to CQ 2 and completes normally: overflow of one CQ must not reach a QP or CQ that isn’t party to it. QP B stays in RTS too, unless the implementation treats the overflowed CQ itself as in error; then every QP on CQ 1 goes to ERROR. CQ Overflow
8. Room for 12, the sum of every outstanding ceiling. Then QP A’s whole flush, all four of QP B’s completions and all of QP C’s work fit with nothing polled. Size for every possible outstanding WR completing at once, because a flush is exactly when they do. CQ Overflow

Deep walkthrough

IBTA-V1’s own position, stated directly rather than inferred. Two direct compliance statements settle this. C9-218 (Requester Class F): “A Requester Class F error occurs when the CQ is inaccessible or full and an attempt is made to complete a WQE. The Affected QP shall be moved to the error state and affiliated asynchronous errors generated… The current WQE and any subsequent WQEs are left in an unknown state.” C9-225 says the identical thing for the responder side (Responder Class G). Hitting a full CQ during a completion attempt isn’t a gap the standard stays silent on – it’s a named fault class with a mandatory consequence: the affected QP shall move to ERROR. What’s genuinely left open is narrower than “does anything happen at all”: the content of the current and any subsequent WQEs is explicitly left unknown, and nothing in the standard requires hardware to proactively detect an approaching overflow before it happens – only to react correctly once a completion attempt actually finds the CQ full. Nothing in The CQE or The Completion Queue changes because of this; it’s a separate fact layered on top of both, about what happens at the one moment those chapters’ structures run out of room.

What “left in an unknown state” most likely looks like, reasoned rather than sourced. IBTA-V1 mandates the QP’s fate but doesn’t describe a concrete failure shape for the WQE content itself – a ring-based structure gives away the likely one anyway: a producer index that keeps advancing past a full ring has nowhere to write but over an existing slot, silently overwriting a CQE for a different, already-succeeded operation that software simply hasn’t polled yet. That’s a materially worse outcome than the new completion just failing to appear – it’s an existing, valid result quietly corrupted underneath an application that has no way to know it happened. This specific mechanism is this book’s own inference from the ring structure The Completion Queue already established, not a claim IBTA-V1 makes explicitly – the compliance statement itself commits only to “unknown,” not to a specific corruption shape.

What the mandatory QP-to-ERROR outcome actually connects to. C9-218/C9-225 aren’t a status value sitting unconnected to everything else this book has already described. The compliance statements themselves name only the affected QP – the one whose completion attempt found the CQ full – and that QP’s move to ERROR is the part every compliant implementation shares. Where an implementation also treats the overflowed CQ itself as in error, the fault joins the path Single-QP Error Handling and Multi-QP Error Arbitration already established for a CQ in error generally: every QP associated with that CQ transitions to ERROR simultaneously, the cascade Event and Multi-CQ Arbitration treats as a live scenario, later in this Part. Either way, the damage stays among the QPs reporting to this CQ. And because ERROR is sticky per Single-QP Error Handling’s own flush behavior, the cost of overflow isn’t contained to the one completion that couldn’t be written: every WQE the affected QP has outstanding, and every one posted to it afterward until software notices and resets it, gets flushed in error too. Those flush completions are aimed at the same CQ that just ran out of room, which is exactly why the compliance statement leaves “any subsequent WQEs” in an unknown state rather than promising each one a flush CQE. Overflow doesn’t just cost software one lost completion record – it costs the QP itself.

The provisioning rule that keeps this from being a real problem in practice. The RDDP verbs source above states exactly what makes overflow avoidable by construction: software should size a Completion Queue so that if every Work Request currently capable of being outstanding actually completed at once, there would still be room for all of their CQEs. Stated as a concrete inequality: for deterministic behavior, the sum of the maximum outstanding Work Request counts across every Queue Pair pointing at one CQ should never exceed that CQ’s own depth. Get that provisioning right, and overflow simply can’t happen, no matter how completions happen to be timed. Get it wrong, and C9-218/C9-225’s mandatory QP-to-ERROR consequence, with the triggering WQE’s own content left unknown, is what actually happens.

“Every possible outstanding WR completing at once” is not a hypothetical – this book has already named the mechanism that does it. Single-QP Error Handling already established that a QP forced to ERROR gets both its queues flushed, and CQE Generation already named what that means for this Part specifically: a flush writes a CQE for every remaining entry, in order, regardless of any individual Work Request’s own signal flag. A provisioning scheme that only accounts for the steady-state rate of signaled completions, and not for what a single QP’s full outstanding count would dump into its CQ all at once during a flush, is exactly the gap the sizing rule above is warning against.

The one guarantee that survives overflow regardless of provisioning. Even granting that the triggering WQE’s own content is left unknown, overflow’s blast radius is not left equally unbounded: overflow of one CQ must not affect a Queue Pair that doesn’t report completions to it, and must not affect any other Completion Queue. Whatever happens to this CQ and the QPs actually pointing at it, it stays contained to them – the same shape of isolation boundary Single-QP Error Handling already drew around one QP’s own fault, applied here to a shared structure’s overflow instead.

Why this Part doesn’t get a preventive admission-control gate the way Reliability Engine did, even though the QP-to-ERROR consequence is real. Replay Buffer Capacity and Backpressure argued that hardware has to gate transmission itself, because a lost replay entry breaks the reliability protocol’s own ability to retransmit – the protocol depends on that data surviving, for every QP on the connection, indefinitely. A CQ overflow’s cost is real but bounded differently: the wire-level correctness of the operation that overflowed isn’t in question – it was still correctly sent, acknowledged, and retried exactly as if its completion had been recorded. What’s genuinely lost is this QP’s own ability to keep operating, and everything still outstanding or posted to it afterward, until software notices and resets it – a real cost, just one Single-QP Error Handling’s own isolation guarantee already contains to this one QP and CQ rather than the protocol’s own guarantees to every connection sharing the fabric. That containment, not any doubt about whether the consequence is real, is what lets the standard leave the preventive half of this to software provisioning discipline instead of mandating a hardware gate: getting provisioning wrong costs this QP its own future, not the wire protocol’s correctness for anyone else.

References

• InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications) – C9-218 (Requester Class F Fault Behavior) and C9-225 (Responder Class G Fault Behavior) are the primary source for this chapter’s central claim: a CQ found inaccessible or full during a completion attempt mandates that the affected QP move to ERROR and generate an affiliated asynchronous error, while leaving the current and any subsequent WQEs in an unknown state. This is the authoritative statement for that mandate.
• Hilland, Culley, Pinkerton, and Recio, RDMA Protocol Verbs Specification (IETF Internet-Draft draft-hilland-rddp-verbs-00, expired, never advanced to RFC status) – confirms the same behavior at the software-interface layer and is the source for this chapter’s provisioning guidance: a Consumer should size a CQ’s depth against the sum of every associated QP’s maximum outstanding Work Request count for deterministic behavior, and overflow of one CQ must not affect QPs or CQs not party to it. Kept as a secondary, software-interface-level confirmation, not the primary source for the QP-to-ERROR mandate above.
• The ring-wraparound mechanism describing what “an unknown state” most likely looks like in practice is this book’s own reasoned inference from the ring structure The Completion Queue already established, not a claim either source above makes explicitly.