Queue Pair Management
Queue Pair Management
A Queue Pair is the connection a Reliable Connection RDMA operation runs on, and everything else in this book eventually answers to one: which QP a packet belongs to, what state that QP is in, and what happens when it faults. This part covers the QP end to end – its context field by field, the table that stores it, the on-chip cache that makes touching it cheap, the software handoff that installs one, the state machine hardware enforces around it, what happens when one QP faults alone, when many fault together, when the cache runs out of room, when a QP resets with an operation still in flight, and, closing it out, how many QPs a real chip can actually hold at once and what it costs in silicon to let them share this same structure safely.
Chapters
Two fields on every RC Queue Pair count outstanding RDMA reads and atomics – one for what this QP has sent, one for what it will accept – and confusing them is the single most common misreading of a QP’s context. Here’s the full field list, wire-format ties included, and which fields hardware updates on its own without a single software write.
A QP number isn’t a name, it’s an address. This chapter is about the structure that address indexes into, and why a page-table-shaped answer fits a problem the public record barely documents.
A NIC will sell you far more queue pairs than it can actually keep fast. This chapter is about the gap between those two numbers, and the on-chip cache that decides which QPs live on the right side of it.
A QP exists in hardware before it can send or receive anything at all. This chapter covers the two steps that get it there: allocating its table entry, and the one transition that turns an allocated-but-inert entry into something bound to a port with its access rules already fixed.
Every QP state transition is a contract: software supplies a fixed set of attributes, hardware refuses the transition if even one is missing. Reliable Connection’s contract is the strictest of the three transports it’s enforced against, and that’s not incidental – it’s every parameter the reliability engine needs, collected at the one moment hardware can still refuse to proceed without it.
A QP in ERROR doesn’t fail loudly and stop – it fails loudly and keeps going, flushing every queued work request through the completion queue with a status that says none of this actually happened. Here’s what that flush actually does, and why the QP sitting next to it never even notices.
One QP faulting is a contained event with a clean isolation boundary. A hundred QPs faulting in the same instant, from the same shared cause, all need to tell software about it through the same narrow channel – and no public source says how a real RNIC decides who goes first.
The QP Cache runs out of room the moment enough connections are active at once. Something has to leave to make space for the QP that just arrived, and which QP that is turns out to matter as much as the eviction itself.
Deciding which QP to evict is one question. Deciding what actually runs that decision – fixed silicon, a programmable engine, or a trap to software – is a separate one, and it’s where the cost of flexibility gets paid in latency, not complexity.
RESET clears a QP’s leftovers – that much is documented. What counts as a leftover when a multi-packet message is half-arrived, a replay-buffer entry is still waiting on an ACK, and a DMA write is mid-burst is not, and this chapter says so plainly before reasoning through it anyway.
Every earlier chapter in this part answered a question about one QP. This one asks how many QPs a real chip can actually hold at once, what really sets that number, and what it costs in silicon to let a slow control-plane write and a per-packet lookup safely touch the same structure.
Last updated on