Multi-QP RX Arbitration

Overview

A Send Queue chooses when to make a WQE ready to send; a Queue Pair’s receiving side has no equivalent control over when the wire delivers it a packet. Many Queue Pairs sharing one adapter can have packets land within the same narrow window, all needing the same shared parser, the same sequence-number check, the same placement hardware – and only one physical pipeline to run them through. This is the same policy shape Multi-QP TX Scheduling already worked through for the transmit side, and the same shape Eviction Policy worked through before that for the QP Cache – round-robin, weighted, priority – asked here of arrival order on the receiving side instead of departure order on the sending one.

Locator

Single-QP RX Stalls confirmed that one Queue Pair’s own wait does not touch a different Queue Pair’s state. This chapter asks the question that leaves open: when packets belonging to several different Queue Pairs are all ready for this part’s shared hardware at once, something has to decide whose gets serviced next.

Block diagram

Deep walkthrough

The candidate policies – the same shapes this book has already named twice. Round-robin services whichever Queue Pairs have a packet waiting in a fixed rotation, simple and starvation-free by construction, but blind to the fact that servicing one Queue Pair’s packet might cost this shared pipeline far more cycles than another’s – the same blind spot Eviction Policy already named for pure least-recently-used cache selection. Weighted round-robin fixes that blindness by giving some Queue Pairs a larger share of arbitration turns than others, at the same cost every weighted scheme this book has covered pays: something external has to own and set those weights. Priority arbitration lets some arrived traffic jump the queue entirely – at the same starvation risk Multi-QP Error Arbitration already named for priority-weighted fault reporting, a steady stream of high-priority arrivals can, in principle, indefinitely delay whatever is sharing the same gate at lower priority.

A real candidate for priority treatment, reasoned here rather than sourced. An arriving acknowledgment carries no payload to place and exists purely to let the sending side retire outstanding work and advance its own progress – unlike a data-carrying packet, servicing it quickly has a direct effect on a different Queue Pair’s transmit-side throughput, not just this one’s receive-side bookkeeping. This is a plausible, architecturally motivated reason a real implementation might prioritize acknowledgment traffic over ordinary data arrivals at this gate, offered as reasoned analysis rather than a documented fact – no source this book found confirms a real implementation actually does this, the same honest limit Single-QP RX Stalls already hit for placement-stage backpressure.

What is confirmed, and what is not. The Send Queue Ring already named the source: Kong et al. (NSDI 2023)’s own hardware diagram names a “TX Pipeline (Processing Unit)” and an “RX Pipeline (Processing Unit)” as separate blocks sharing common on-chip cache resources between them – a direct measurement of real adapter hardware, confirming independently of this book’s own reasoning that the receiving side has dedicated hardware distinct from the transmit side rather than a single undifferentiated pipeline handling both directions. What that measurement does not do is describe the receive-side unit’s own internal arbitration policy the way a separate, vendor-quoted admission confirmed an internal scheduling policy exists on the transmit side. This chapter states that gap honestly: the existence of dedicated receive-side hardware is confirmed; which policy governs arbitration within it is not.

Why this is not simply Multi-QP TX Scheduling’s argument restated. The transmit-side chapter’s central finding was that congestion control does not substitute for a real scheduling policy, because the transmit-side problem is internal to the adapter and exists even on an uncongested network. The equivalent claim on the receive side is different in kind, not just in direction: nothing about congestion control is even in scope here, because this chapter’s contention is over shared parsing and placement hardware for packets that have already crossed the wire successfully – congestion control governs whether and how fast a sender’s packets reach the wire in the first place, a question that is settled by the time this chapter’s arbitration even begins. This chapter’s problem starts precisely where a congestion-control mechanism’s job already ends.

What happens next. Whichever policy actually governs this gate, something has to run it – fixed silicon, a programmable engine, or a mechanism pushed out to software entirely. That is Hardware-Assisted RX Dispatch Engines’s subject, immediately next.

Hardware-Assisted RX Dispatch Engines is next: what actually executes whichever arbitration policy this chapter named.

References

• No IETF or IBTA standard governs internal multi-QP RX arbitration policy – it is implementation-specific silicon and firmware design, not a wire-format or software-interface contract any standards body defines. This chapter’s treatment is grounded in the sources cited in-text above, not a standards document.
• Kong, Zhang, et al., published at NSDI 2023, is the source for the measured TX/RX Processing Unit split cited above – the same paper The Send Queue Ring already names in full for its own doorbell-mechanism citation.