Completion Queue / Event Management
Completion Queue / Event Management
The TX and RX pipelines both end at the same kind of structure: a Completion Queue Entry gets generated, written into a per-CQ ring, and reported – by polling or by interrupt – to whatever’s waiting on it. This part covers that path end to end: CQE generation from both the send and receive sides, what happens when a CQ fills up faster than software drains it, and how hardware arbitrates event reporting when many queues have something to report at once.
Chapters
A WQE is deliberately thin: hardware produces everything downstream of it. A CQE runs that discipline in reverse – it’s hardware’s own report of what actually happened, and on a failed completion, most of it isn’t even required to be trustworthy. This chapter is the field list, and the one status check that decides which fields you’re allowed to read at all.
A Send Queue is software producing, hardware consuming. A Completion Queue is the exact same ring shape with both roles swapped – hardware produces, software consumes – and it’s usually not even feeding from one QP: many QPs’ completions land in the same ring, interleaved in whatever order they actually finish in.
Installing a QP needs a state machine. Registering a Memory Region needs a Protection Domain. Creating a Completion Queue needs neither – just a depth request and, optionally, a way to be told about new entries instead of having to go looking for them.
Nothing else in this Part is fed by both sides of this engine’s own pipeline at once. This chapter is the convergence point – and the one place a real, asymmetric rule decides whether a completion gets written at all: every receive-side event does, unconditionally, while a send-side event only does if something actually asked for it.
The Replay Buffer can’t safely evict, so hardware enforces admission control instead. A Completion Queue has no equivalent gate – but IBTA-V1 is specific about what happens when one fills anyway: the affected QP doesn’t just lose one completion, it’s mandated into ERROR.
Multi-QP Error Arbitration named this Part as the place a real, documented storm scenario gets its general treatment: many QPs forced to ERROR by one shared fault, all needing to report at once through what is, at that instant, one serialized path. This chapter names that problem in full – and its quieter, steady-state counterpart on the notification side – without a citation pinning down which policy a real RNIC actually runs for either.
Mid-Transaction TX Reset already gathered the citation this chapter needed before this Part even had a name: destroying a QP doesn’t guarantee its CQEs get cleaned out of the Completion Queue it reported to. This closing chapter is what that leaves behind, and the one real dispute about whether a shared CQ’s own cleanup path can be trusted not to touch a QP that never asked to be reset at all.
Last updated on