Multi-QP TX Scheduling

Overview

Many QPs sharing one adapter can all have traffic ready to send at the same moment, and only one shared TX pipeline to send it through. This is the same policy shape Eviction Policy already worked through for the QP Cache – LRU, priority-weighted, error-weighted – asked here of a different scarce resource: not which cached context survives, but whose WQE gets the pipeline’s next cycle. Unlike that chapter, this one isn’t limited to reasoned analysis alone – a real vendor’s own quoted response to a measured anomaly confirms that shipped RNICs do run some internal scheduling policy, and a separate, independent study confirms directly that congestion control alone does not fix the problem this chapter is about.

Locator

Single-QP Pipeline Stalls measured what happens with no scheduling policy actively protecting one QP from another’s traffic. This chapter is the policy question that measurement motivates: whichever QPs have a WQE ready to send at the same instant, something has to decide whose gets fetched and built next.

Block diagram

Deep walkthrough

A vendor’s own admission that internal scheduling exists – not this book’s inference. Kong et al. (NSDI 2023) quote NVIDIA’s own response, in the paper’s Appendix B, to the paper’s measured deregistration-versus-throughput finding: “the deregistration control verbs can cause drastic performance reduction mainly because of the NIC internal QoS scheduling policy. The deregistration control verbs have higher priority than other types of operations and will be scheduled first.” This is a specific, vendor-acknowledged confirmation that a shipped RNIC runs some internal QoS/priority mechanism arbitrating between different operation types – not a generic industry assumption this book is applying by analogy, a named vendor’s own explanation for a specific measured phenomenon. NVIDIA’s stated fix was a firmware-level tuning of that same policy, not a hardware redesign – indirect evidence that at least part of this mechanism is firmware-adjustable rather than fixed at tape-out, a detail Hardware-Assisted TX Scheduling Engines picks up directly next.

Candidate policies, the same shapes Eviction Policy already named for a different resource. Round-robin services every QP with ready work in a fixed rotation, regardless of how much each one has to send – simple, and starvation-free by construction, but blind to the fact that one QP’s WQE might be trivial to service and another’s might occupy the pipeline far longer, the same “recency isn’t the same as importance” blind spot Eviction Policy already named for pure LRU. Weighted round-robin fixes that blindness by giving each QP a share proportional to an assigned weight rather than a flat turn – Zhang et al. (Justitia) describe exactly this shape as a working software-layer mitigation, distributing rate tokens “among the active resource-hungry flows in a round-robin fashion,” with a stated straightforward extension to weights (“If a flow’s weight is w_i, Justitia can ensure it receives w_i-proportional tokens during each round”). Priority scheduling lets some traffic – NVIDIA’s own deregistration-verbs example above – jump ahead of ordinary traffic entirely, at the same starvation risk Multi-QP Error Arbitration already named for priority-weighted fault reporting: a steady stream of high-priority work can, in principle, indefinitely delay something lower-priority sharing the same gate.

A concrete worked example, tying the fence mechanism back to that starvation risk. Say QP1 posts a fenced Work Request and stalls waiting on an earlier read to complete (Single-QP Pipeline Stalls covers that stall on its own), while QP2 has a continuous stream of ordinary work ready to go. Under strict priority with QP2 favored, QP2 simply gets serviced every turn while QP1 has nothing eligible to offer – unremarkable so far, since QP1’s fence genuinely leaves it with nothing ready to send. The interesting case is what happens next: the earlier read QP1’s fence was waiting on completes, its Work Requests become eligible again, and the fence itself is gone – but if QP2’s priority advantage and continuous ready work haven’t changed, QP1 can stay starved indefinitely anyway. The fence explains why QP1 was blocked before; the scheduling policy, not the fence, is what explains it staying blocked after. Round-robin doesn’t share this failure mode – QP1 simply resumes its normal turn the instant it has eligible work, with no memory of how long it sat idle – which is exactly the starvation-free-by-construction property named above, and exactly what strict priority trades away in exchange for servicing genuinely urgent work predictably fast.

The two stalls back to back, from the prefetch pool to the gate:

All 9 steps

1. The prefetch pool. QP1 posts 100 Work Requests. The tenth is fenced: it may not start until an earlier READ, still outstanding, has its response. WQE Prefetch
2. The pool is empty and nothing limits one QP’s share, so QP1 is granted all 100 slots and its WQEs are prefetched on-chip. WQE Prefetch
3. WQEs 1 to 9 issue through the gate and leave on the wire, freeing their slots as they go. Multi-QP TX Scheduling
4. No. A Send Queue issues strictly in order, so everything behind WQE 10 waits with it: 90 slots resident, prefetched, and doing nothing. Single-QP Pipeline Stalls
5. QP2 arrives with work of its own and finds only the 9 slots QP1 freed. Nobody did anything wrong: the fence is legitimate, the request is legitimate, and the problem is how the pool was handed out. WQE Prefetch
6. At the gate. The gate favours QP2 and QP2 always has something ready, so QP2 is serviced every turn. Unremarkable so far: QP1 has nothing eligible to offer. Multi-QP TX Scheduling
7. The earlier READ’s response arrives and QP1’s fence clears. WQE 10 and the 90 behind it are eligible again. Single-QP Pipeline Stalls
8. Not under strict priority. QP2 is still ready and still favoured, so it wins every turn and QP1 stays idle. Same symptom as before, different cause: the fence explained QP1’s wait before, the policy explains it now. Multi-QP TX Scheduling
9. The gate is round-robin instead: Round-robin does not share this failure: QP1 gets the next turn and resumes, and QP2 is served on the turn after. Multi-QP TX Scheduling

A working example – built specifically because the hardware didn’t already do this. Justitia’s own scheduling mechanism is shipped, but it’s explicitly a software/driver-layer mitigation, not a description of on-chip scheduling hardware: its shapers are “implemented in the RDMA driver,” built as a workaround for RNICs that, at the time of that study, lacked a programmable on-chip packet scheduler at all – the paper states this directly: “Justitia works on existing RDMA NICs that do not have programmable packet schedulers.” It’s precisely what it looks like: working evidence for where at least some real deployments have historically landed on the fixed/programmable/software-driven spectrum Hardware-Assisted TX Scheduling Engines covers in full next, not proof that no RNIC has ever shipped a hardware scheduler.

Why this is not Congestion Control Engine’s question wearing a different label – confirmed directly, not asserted by this book. Justitia’s authors tested this directly: running their anomaly-reproducing benchmarks with “PFC enabled at both the NICs and switch ports, DCQCN… enabled at the NICs, and ECN markings enabled,” they still found “latency- and throughput-sensitive flows still suffer unpredictably.” Their own diagnosis, on a topology deliberately simple enough to rule out the network itself as the cause: “the network is not the source of anomalies in these experiments, and thus explains why DCQCN does not fix those anomalies. Rather, at the end hosts, RNICs’ immediately processing all ready-to-consume messages to achieve work conservation is very likely to cause head-of-line (HOL) blocking.” This is a direct, empirical confirmation – not this book’s own organizational convenience – that TX-pipeline scheduling and congestion-control rate-limiting are separate problems: DCQCN reacts to network-path congestion signals (ECN marks a switch set), while this chapter’s problem is entirely internal to the RNIC, present even when the network itself is uncongested and blameless.

Where this is checked, and what happens next. The scheduling decision this chapter describes happens at the same gate Single-QP Pipeline Stalls already named – wherever a real pipeline actually holds ready-but-not-yet-serviced WQEs from multiple QPs – consulting whatever policy (round-robin, weighted, priority) governs that gate. What actually runs that policy decision – fixed silicon, a programmable engine, or a firmware-adjustable knob like NVIDIA’s own QoS mechanism above – is Hardware-Assisted TX Scheduling Engines’s subject, immediately next.

Hardware-Assisted TX Scheduling Engines is next: what actually runs whichever policy this chapter names – fixed silicon, a programmable engine, or firmware.

References

• No IETF or IBTA standard governs internal multi-QP TX scheduling policy – it is implementation-specific silicon and firmware design, not a wire-format or software-interface contract any standards body defines. This chapter’s treatment is grounded in the sources cited in-text above, not a standards document.