Hardware-Assisted TX Scheduling Engines
Overview
A scheduling decision that has to run on every single arbitration between ready QPs can be hardwired at design time, built as a programmable engine a driver can retune without a hardware respin, or pushed out to software entirely. Hardware-Assisted Eviction Engines already worked through this exact spectrum for cache eviction, and that reasoning applies here without needing to be re-derived – what this chapter adds is dated, concrete evidence for where actual deployments have sat on it, not just the abstract tradeoff.
Locator
Multi-QP TX Scheduling named the real candidate policies – round-robin, weighted round-robin, priority. This chapter asks what actually executes whichever one a given design picks, the same fixed/programmable/software-driven question Hardware-Assisted Eviction Engines already asked of QP Cache eviction.
Block diagram
Deep walkthrough
The spectrum itself – not re-derived, cross-referenced. Hardware-Assisted Eviction Engines already worked through the full latency-versus-flexibility tradeoff between fixed-function hardware, a programmable hardware engine, and a software-driven trap, for QP Cache eviction. Every word of that reasoning – fixed silicon is fastest and least flexible; a programmable engine is the realistic middle ground most designs reach for; a software trap on every decision reintroduces exactly the round-trip RDMA hardware offload exists to remove – applies to TX scheduling without modification. This chapter doesn’t re-argue it a second time; it applies the same spectrum to a different decision (whose WQE gets the pipeline next, not which cache line gets evicted) and grounds where real evidence actually places it.
The middle of the spectrum – vendor-confirmed and firmware-adjustable. Multi-QP TX Scheduling already named NVIDIA’s own quoted admission that a shipped RNIC runs an internal QoS scheduling policy, and that the vendor’s fix for a measured anomaly was a firmware-level tuning of that same policy rather than a hardware redesign. That’s concrete evidence the mechanism sits closer to “programmable hardware engine” than “fixed-function” – a policy that can be retuned in firmware, without a new chip, is by definition not hardwired at tape-out the way pure fixed-function logic would be. What isn’t public: the exact shape of that programmability – whether it’s a small set of vendor-exposed knobs, a fuller programmable sequencer, or something narrower than either – so this chapter states the position on the spectrum (closer to programmable than fixed) without overclaiming the specific mechanism.
The software-driven end – dated, and explicitly a workaround, not a design choice by the RNIC vendor. Zhang et al. (Justitia) state directly that the RDMA NICs their study targeted did not have programmable packet schedulers at all: “Justitia works on existing RDMA NICs that do not have programmable packet schedulers.” Their own comparison names Titan and Loom as prior systems that “improve performance isolation on conventional NICs by programming on-NIC packet schedulers,” explicitly judged insufficient for RDMA because that approach “only schedules a single resource: the outgoing link,” not the other internal resources (caches, execution units) this book’s own caching chapters already established are real and contended. Justitia’s answer was to build the missing scheduling entirely in software – a driver-level daemon distributing tokens in a round-robin (or weighted-round-robin) fashion, enforced by shapers “implemented in the RDMA driver.” This is working, shipped-as-a-workaround evidence for the software-driven end of the spectrum – dated to a specific study (2019, later published 2022) and scoped to the specific RNICs it tested, not a claim that every RNIC lacks a programmable scheduler today or ever did.
The fixed-function end – included for spectrum completeness, not because it’s confirmed. No source this book found describes a real RNIC shipping pure fixed-function TX scheduling with zero programmability – unlike QP Cache eviction, where Hardware-Assisted Eviction Engines at least had the general silicon-design tradeoff to reason from without a specific counter-example. This chapter states that absence plainly rather than asserting a fixed-function RNIC exists somewhere unconfirmed.
A standards-level analog, one layer removed, named here so it isn’t confused with this chapter’s mechanism. IEEE 802.1Qaz’s Enhanced Transmission Selection (ETS) defines a public standard for a hierarchical scheduler combining static priority scheduling with a bandwidth-sharing algorithm such as weighted round robin or deficit round robin – the same shape of policy this chapter has been describing. But ETS governs switch and link-level scheduling of already-built Ethernet frames moving through a fabric, not an RNIC’s own internal arbitration between WQEs from different QPs before a packet is even built. [NOTE: summarized here from secondary sources, not the IEEE standard text itself -- the general shape (priority + weighted-share hybrid) is reliable, but this chapter doesn't cite specific numeric parameters from it.]
Gotcha
A reader who encounters “ETS” in a switch vendor’s documentation shouldn’t assume it describes what this chapter is about. ETS schedules already-built Ethernet frames moving through a fabric; this chapter is about an RNIC’s own internal arbitration between WQEs, before a packet even exists.
Deficit Round Robin – the general algorithmic family a real WRR-style scheduler would draw from, not an RDMA-specific mechanism. DRR (and its weighted variant) is a packet-based approximation of ideal generalized processor sharing, using a per-queue deficit counter that accumulates credit while a queue waits and spends it on transmission, handling variable-length packets fairly without per-packet floating-point computation. This is general network-scheduling literature, not something any source ties specifically to an RNIC’s internal TX arbiter – cited here as the algorithmic family a real weighted scheduler would plausibly draw from, not as a claim about what any specific vendor implements.
What none of this addresses: the cost of arbitrating among thousands of idle QPs. Every policy above is described as choosing among QPs that currently have ready work. A real adapter can be tracking many thousands of QP contexts, the overwhelming majority idle at any given instant – a scheduler that walks the full population every arbitration cycle, round-robin fashion, wastes cycles on QPs with nothing to send. The efficient answer is almost certainly some form of eligibility tracking – an active list a QP joins the moment it has a ready Work Request and leaves once it’s drained, so the scheduler only ever arbitrates among QPs actually contending for the gate – but no source this book found describes a real RNIC’s eligibility-tracking mechanism at this scale, and this chapter doesn’t invent one.
Food for Thought
A scheduler arbitrating among a handful of QPs and one arbitrating among thousands are not the same design problem, even running the identical policy. What structure would you reach for to track which of several thousand QPs currently has ready work, given this book’s caching chapters have already shown that anything resembling a per-QP table entry competes for the same scarce on-chip SRAM everything else does?
What happens next. This chapter closes the scheduling question this part opened with Single-QP Pipeline Stalls and continued through Multi-QP TX Scheduling. What’s left is a question orthogonal to all of it: what happens to a WQE caught mid-fetch, mid-build, or mid-schedule when the QP it belongs to resets underneath it.
Why
The same design tension Hardware-Assisted Eviction Engines already named applies here with concrete evidence behind it rather than abstract reasoning alone: a vendor building toward NVIDIA’s demonstrated position (a firmware-tunable internal policy) buys the ability to fix a measured fairness problem without a hardware respin, at the cost of building and validating that programmability in the first place. A vendor building toward Justitia’s demonstrated position – no on-chip programmable scheduler at all – ships simpler hardware and pushes the entire fairness problem to software, at the direct cost Single-QP Pipeline Stalls already established: a severe throughput collapse for a stalled flow, on hardware that hasn’t built a way to prevent it in silicon. Neither position is free, and the honest reading of the evidence above is that shipped RNICs have occupied both ends of this spectrum at different points, not converged on one answer.
Mid-Transaction TX Reset closes this part: what happens to a WQE caught mid-fetch, mid-build, or mid-schedule when its owning QP resets underneath it.
References
- No IETF or IBTA standard governs which hardware runs a TX scheduling policy – it is implementation-specific silicon and firmware design, not a wire-format or software-interface contract any standards body defines. IEEE 802.1Qaz (Enhanced Transmission Selection) is a real public standard for the switch/link-level analog named above, but does not govern this chapter’s mechanism. This chapter’s treatment is grounded in the sources cited in-text above, not a standards document.