Creating a Completion Queue

Overview

Creating a Completion Queue is the software-to-hardware handoff that produces the ring The Completion Queue already described. It’s a noticeably thinner handshake than either QP installation or memory registration – no state machine to drive through, no Protection Domain to belong to, just a depth request and two optional pieces of routing for how software finds out about new entries at all.

Locator

The previous chapter’s ring has to come from somewhere. This chapter is that somewhere – what software supplies, what hardware hands back, and what stays fixed versus adjustable afterward.

Block diagram

Deep walkthrough

What creation actually requires – and what it conspicuously doesn’t. A Completion Queue creation call takes a device context and a requested minimum depth, full stop, as far as mandatory inputs go. Installing a QP needs a Protection Domain, and Registering a Memory Region needs one too – both are memory-and-permission-scoped objects, and a PD is the boundary that scopes them. A Completion Queue isn’t either of those things. It doesn’t touch application memory the way a Memory Region does, and it doesn’t carry the remote-access permission mask a QP does – it just holds completion records – so it has no protection boundary to belong to at all.

Depth negotiation, in full. The Completion Queue already named the shape of this: software requests a minimum, hardware is free to allocate more, and the actual allocated depth comes back as part of creation’s own return value rather than a separate query. Public completion-queue interface documentation describes exactly this contract: a creation call allocates at least the requested depth, never less, and the caller is expected to read the returned structure’s own depth field rather than assume the requested number is what it got. There’s no benefit to software over-requesting to compensate for possible rounding – whatever hardware allocates is already reported back, so the honest number is always available immediately after creation returns.

The event channel and interrupt-steering vector – optional at creation, and this is where polling and events actually fork. Two more inputs are optional, and both stay unset by default: an event channel, and an interrupt-steering vector. Neither is required – a Completion Queue created without either is a purely polling-driven structure, exactly what The Completion Queue already described as the baseline case. Supplying an event channel is what makes this CQ capable of generating an asynchronous notification instead of relying entirely on software to keep polling it; an interrupt-steering vector, when a device supports more than one, is what steers which interrupt line that notification actually lands on – letting different CQs’ events spread across different interrupt lines (and typically different CPU cores) rather than all funneling through one. This is where The WQE and the Send Queue Descriptor’s solicit flag actually cashes out: a solicited-event request only has anywhere to go if the responder’s own CQ was created with an event channel in the first place.

An event channel existing isn’t the same as being armed to hear from it. A real interface treats notification as a one-shot request, not a standing subscription: software has to explicitly ask to be told about the next completion each time, and a CQ that isn’t currently armed simply produces no event no matter how many CQEs land in it while unarmed. That request itself comes in two flavors, and this is the actual consumer of the solicit flag named above – software can arm for the next completion of any kind, or narrow the request to only the next solicited one, ignoring ordinary unsolicited completions entirely until something explicitly marked solicit arrives. This Part’s own event-arbitration chapter, later, covers what happens once many CQs are all wired up to notify this way at once.

All 11 steps

1. Poll, then arm, then sleep. The application drains the CQ until a poll comes back empty, intending to arm notification and sleep until more work completes. Creating a Completion Queue
2. Before the arm request reaches the adapter, a Work Request completes and its CQE is written. The CQ isn’t armed, so no event is raised. Creating a Completion Queue
3. The arm request arrives. Notification is one-shot and covers only a CQE written after the request. Creating a Completion Queue
4. No. The waiting CQE was written before the arm, so it never produces an event of its own. Creating a Completion Queue
5. The application sleeps on the event channel. The completed work sits in the CQ until some unrelated later completion happens to wake the thread, and if none comes, it is never handled. Creating a Completion Queue
6. Arm, then poll again. Same start: a poll comes back empty. Creating a Completion Queue
7. A CQE lands in the same gap, before the arm. Creating a Completion Queue
8. The application arms notification. Creating a Completion Queue
9. Poll once more. This poll finds the CQE that landed before the arm and handles it. The arm stays in place for whatever comes next. Creating a Completion Queue
10. A later Work Request completes while the CQ is armed, so this CQE does raise an event. Creating a Completion Queue
11. The application wakes and repeats the same order: poll until empty, arm, poll once more. It never sleeps with a CQE waiting. Creating a Completion Queue

Resizing – a separate act from creation, not a variation on it. A Completion Queue’s depth doesn’t have to stay fixed for its entire life the way The QP Table’s entries or a Send Queue’s ring do – a real interface documents a distinct resize operation, callable after creation, that requests a new minimum depth for an existing CQ. The one hard constraint: the requested new depth has to be at least the number of entries still sitting unpolled in that CQ right now – you cannot resize a queue down below data it’s already holding that software hasn’t consumed yet. Beyond that floor, the same rounding rule from creation applies again: the actual new depth may still come back larger than requested, and has to be read back the same way.

What creation does not fix. Creating a CQ says nothing about which QPs will ever point at it – that association happens later, at each QP’s own creation or modification, entirely independent of this handshake. A CQ that will end up shared across dozens of QPs and one that will only ever serve a single QP go through the exact same creation call; nothing about creation itself distinguishes the two cases; Multi-QP Error Arbitration and this Part’s own arbitration chapter are where that distinction actually starts to matter. And destruction – what has to be true about a CQ’s outstanding entries and the QPs still pointing at it before it can safely go away – is this Part’s own closing chapter’s subject, not this one’s.

References

• No IETF or IBTA standard governs Completion Queue creation parameters, depth negotiation, or the event-channel/interrupt-steering-vector mechanism – these are software-interface and implementation-specific silicon design, not a wire-format contract any standards body defines. This chapter’s claims (the absent Protection Domain requirement, the depth-rounding behavior, the resize floor, the event-channel/vector split, the one-shot arm/re-arm requirement and its solicited-vs-any distinction) are grounded in the public ibv_create_cq(3) and ibv_resize_cq(3) manual pages and RDMAmojo’s own documentation of the same interface, not a standards document.