Event and Multi-CQ Arbitration

Overview

Multi-QP Error Arbitration already established a real, documented scenario as fact: when a Completion Queue or Shared Receive Queue itself hits an error, every QP associated with it transitions to ERROR and fires a QP-fatal event simultaneously, not one at a time. That chapter named the shape of the problem this Part would have to answer – many simultaneous producers, one serialized reporting structure – without answering it. This chapter is where that shape gets named in full, alongside a related but distinct question: many Completion Queues, sharing one notification channel, competing for the same interrupt.

Locator

Two different structures this Part has already covered both funnel into a shared resource under the right conditions. CQE Generation is one serialized write path several QPs can converge on at once. Creating a Completion Queue’s event channel and completion vector are one notification path several CQs can converge on too. This chapter covers what decides servicing order when either actually happens.

Block diagram

Deep walkthrough

The write-side scenario, restated precisely from where it was left. Single-QP Error Handling already named the trigger: a QP-fatal event caused by an error on a Completion Queue or Shared Receive Queue itself, rather than anything specific to one QP. When that happens, every QP associated with the faulted structure transitions to ERROR at the same instant, and CQE Generation already established what each of those QPs then has to do: flush every remaining entry through as a CQE, in order, regardless of signal flags. The problem this chapter actually owns is what happens when several QPs need to do that at the same moment, through what is, at that instant, one shared, serialized write path. Something has to decide whose flush gets serviced first.

The notify-side scenario – a quieter version of the same shape. Creating a Completion Queue already named the mechanism: an optional completion vector steers which interrupt line a given CQ’s notifications land on, and a device supporting more than one vector can spread different CQs across different lines specifically to reduce contention. That spreading is a real mitigation, not a complete answer – a device only supports so many vectors, and an application can create far more CQs than that. Multiple CQs sharing one vector is the ordinary, expected case for any application with more Completion Queues than the hardware has interrupt lines to give them, not a misconfiguration. When two of those CQs both have something to report at nearly the same instant, the single interrupt line they share can only carry one signal at a time – something has to decide which CQ’s notification the host actually services first.

The same three candidate shapes, because it’s the same underlying question. Multi-QP TX Scheduling already worked through round-robin, weighted round-robin, and priority for a different shared gate – whose WQE gets fetched next – and Eviction Policy worked through the same three shapes again for cache victim selection. Round-robin services every waiting QP-flush or CQ-notification in a fixed rotation, fair and starvation-free but blind to whether one of them is more urgent than the rest. Weighted round-robin fixes that blindness at the cost of needing something to own the weights. Strict priority serves whichever QP or CQ matters most predictably fast, at the structural starvation risk Multi-QP TX Scheduling already named for the exact same tradeoff. This book doesn’t have a citation pinning down which of these a real RNIC actually runs for either the write-side or notify-side question here, the same honest position taken every other time this shape has come up.

The walk below runs both gates: two armed CQs whose events reach one vector together, first under round-robin and then under strict priority, and then three QPs flushing through one CQE write path at once.

All 13 steps

1. Three CQs, one vector. Three CQs were created on vector 0. More CQs than vectors is the normal case, not a misconfiguration. Software has armed A and B for their next completion; C is polled and never armed. Creating a Completion Queue
2. A CQE nobody asked about. A QP that reports to CQ C completes a request, and its CQE is written there. C is not armed, so nothing goes to the notify arbiter. The CQE waits in the ring for the next poll. The Completion Queue
3. Two at once. CQEs land in A and B back to back. Both are armed, so each asks for a notification, and both requests are pending when the arbiter next decides. Arming is one-shot, so A and B are both disarmed now. Event and Multi-CQ Arbitration
4. Whichever the policy says; no standard decides it. Under round-robin it is A’s turn, so A goes, and the turn moves to B. Event and Multi-CQ Arbitration
5. The interrupt reaches the host. The event channel tells software which CQ the event is for: A. Software polls A, drains it, and re-arms it. Creating a Completion Queue
6. Next decision: B is the only request pending and it is B’s turn anyway. B’s event goes out.
7. Strict priority. A and B are both pending again. Priority picks A. Event and Multi-CQ Arbitration
8. Before the arbiter decides again, software has handled A’s event and re-armed A, and A’s next CQE has already fired it. A is pending again, next to B.
9. And A wins again. B’s event waits as long as A keeps a request pending: the starvation risk Multi-QP TX Scheduling found for the same policy. Round-robin would have served B here. Multi-QP TX Scheduling
10. A shared fault. Now the write side. QPs 1, 2 and 3 share a Shared Receive Queue, and it faults. All three go to ERROR at the same instant, and every WQE they still hold has to be flushed out as a CQE. All three report to CQ A. Single-QP Error Handling
11. Round-robin takes one from each QP in turn: 1a, 2a, 3a, then 1b, 2b, 3b, then 1c. CQE Generation
12. They reach CQ A in that order. Each QP’s own flushes are still in order, which is what CQE Generation requires; how the three QPs interleave is the arbiter’s choice. Priority for QP 1 would have written 1a, 1b, 1c first. The Completion Queue
13. Three QPs and seven CQEs are easy. A fault on a structure hundreds of QPs share makes all of them eligible in the same instant, which is when the choice of policy matters most. It is also the event CQ Overflow’s sizing rule is written for. CQ Overflow

Why a QP-fatal storm specifically raises the stakes on getting this right. An ordinary moment of two QPs’ signaled completions landing close together is unremarkable – The Completion Queue already established that a shared CQ’s ring is a strict FIFO regardless of which QP fed it, and nothing about ordinary traffic requires a policy decision beyond “whoever’s write actually arrives first.” A shared-fault storm is different in kind: potentially dozens or hundreds of QPs (per Multi-QP Error Arbitration’s own framing) all becoming eligible to flush at the exact same instant, not spread out over time the way ordinary traffic naturally is. Whatever policy governs this gate matters most precisely when it’s under the most simultaneous demand – the same lesson CQ Overflow’s provisioning rule already drew from the identical event.

What happens next. Once a CQE actually gets written and, if applicable, a notification actually gets raised, both this Part’s earlier chapters and this one are done with it – Mid-Transaction CQ Reset, the last chapter in this Part, is what happens to a Completion Queue itself while it’s still in this kind of active use and something tries to tear it down underneath that activity.

References

• No IETF or IBTA standard governs which arbitration policy a real RNIC runs for either simultaneous CQE writes or simultaneous CQ notifications – it is implementation-specific silicon and firmware design, not a wire-format contract any standards body defines. This chapter’s treatment reasons from the same candidate-policy shapes already sourced and cited in Multi-QP TX Scheduling and Eviction Policy, not a new citation of its own.