Skip to content
TX Pipeline
The Send Queue Ring

The Send Queue Ring

Overview

A Send Queue’s WQEs live in a ring in host memory – a fixed-size, circular buffer software writes into and hardware reads from, tracked by a producer index (software’s next write position) and a consumer index (the oldest entry hardware hasn’t finished with). Hardware’s read position runs ahead of the consumer index: a fetched WQE keeps its slot until it completes, because a retransmission may still need it. In the Linux software RoCE driver, rxe, the requester’s wqe_index is the read position and rewinds to the consumer index on a retry, while the consumer index itself advances only when the completer finishes a WQE. Posting a Work Request writes a WQE into the ring at the producer index and advances it; it does not, by itself, move any data. A separate act – ringing the doorbell – is what actually tells hardware new work exists.

Locator

The master RNIC block diagram with Host Memory / Descriptor Rings and WQE Fetch / TX Scheduler both highlighted – this chapter covers the ring itself and the doorbell that tells the fetch stage it has new work.

The WQE and the Send Queue Descriptor covered what one entry contains. This chapter covers the structure holding every entry for one Send Queue, and how hardware learns a new one exists.

Block diagram

An 8-slot Send Queue ring wrapping back to slot 0, with a consumer index and producer index each pointing into the ring and the slots between them shaded amber as posted-but-unfetched, captioned with the N-1 usable-capacity rule; below it, two notification paths to WQE Fetch – an ordinary producer-index write through a 64-byte MMIO doorbell register, and a dashed BlueFlame path writing WQE bytes directly into a UAR buffer, bypassing the ring entirely for lower latency on small messages.

Three Work Requests go through an eight-slot Send Queue ring: posted, announced by the doorbell, fetched, sent, and retired by the completer. A NAK shows why a fetched WQE keeps its slot until it completes, and a second batch wraps the producer index and fills the ring to its N-1 limit.
A Send Queue ring: post, fetch, retire, wrap
All 14 steps
  1. Post and fetch. An empty eight-slot ring. Producer and consumer index are equal, which is what empty means. Nothing has been posted. The Send Queue Ring
  2. The application posts three Work Requests. Each becomes a WQE written into the next slot, and the producer index moves to 3. No byte has reached the adapter yet. The Send Queue Ring
  3. The application rings the doorbell: one MMIO write carrying the new producer index. This is the only part of posting the adapter sees directly. The Send Queue Ring
  4. WQE fetch compares the new producer index with its read position and learns that three entries are waiting. WQE Prefetch
  5. No. The read position moves to 3, but the consumer index stays at 0. Each WQE keeps its slot until it completes, because a retransmission rebuilds from it. WQE Prefetch
  6. The three operations leave as packets. Their slots are still held. TX Packet Build
  7. Retire and retry. An ACK covers WQE 0. The completer finishes it and advances the consumer index to 1; slot 0 is free again. Message Completion and the MSN
  8. A NAK names WQE 1’s first packet as missing. The completer sends WQE fetch back: its read position goes back to the consumer index, 1, so WQEs 1 and 2 are sent again. In rxe, the Linux software RoCE driver, this is the requester’s wqe_index being reset to the consumer index. Go-Back-N Retransmission
  9. No. The application could have posted new WQEs over them. Because the consumer index never passed them, both WQEs are still in place to be read again. The Replay Buffer
  10. The resent packets are acknowledged. The completer advances the consumer index to 3. The ring is empty again: producer equals consumer. Message Completion and the MSN
  11. A signaled Work Request also gets a CQE, which tells the application its WQE is done. In rxe the consumer index advances for unsignaled WQEs as well, and the application’s full check reads it directly; only the CQE is optional. CQE Generation
  12. Wrap and full. The application posts five more WQEs into slots 3 to 7. The producer index goes from 7 to (7 + 1) & 7 = 0 by masking, with no modulo. The Send Queue Ring
  13. Nothing has completed yet. Two more posts go into slots 0 and 1, and the producer index reaches 2. Seven entries are held. The Send Queue Ring
  14. No. The producer index would become 3, equal to the consumer index, and the ring would read as empty with eight WQEs in it. The full test is (PI + 1 - CI) & 7 = 0, so one slot always stays unused. The Send Queue Ring

Deep walkthrough

Ring structure – addressed by index, wrapped by masking, not modulo. A real implementation’s private tracking state for a ring like this needs, at minimum, a buffer pointer and size, an element size, and a way to wrap an index back to the start of the ring once it runs off the end. Constraining the ring’s slot count to a power of two lets that wrap happen as a bitwise AND against a fixed mask rather than a division or modulo operation – (index + 1) & index_mask – a real, checkable optimization technique available to any implementation willing to accept that one constraint on ring sizing, not a claim about how every RNIC’s ring is built.

The off-by-one every ring like this needs. A ring where the producer index equals the consumer index is defined as empty – which means a ring with N physical slots can only ever hold N-1 posted-but-unconsumed entries, because filling all N slots would make the producer index wrap around and equal the consumer index again, indistinguishable from empty. This is not a bug or a wasted slot; it is the standard resolution every fixed-size ring with this exact ambiguity needs.

Concept

A ring with N physical slots can only ever hold N-1 posted-but-unconsumed entries. Filling all N would make producer-equals-consumer mean both “full” and “empty” at once – the one slot given up is what keeps that ambiguity from ever arising.

Producer and consumer indices are shared, cross-privilege state, with synchronization consequences that follow directly. The producer index lives in memory a userspace process writes directly (kernel-bypass, the same fact What Is RDMA opened this book with), while the consumer index is hardware’s own, advanced at completion rather than at fetch. Reads and writes to this shared state need an explicit memory-ordering discipline between the two sides – acquire/release semantics on each side of the handoff, not a full lock, because a lock would reintroduce exactly the per-operation kernel involvement RDMA’s whole model exists to remove. This is a concrete instance of a general problem: any structure touched by both an untrusted, unsynchronized userspace process and a privileged intermediary or a hardware pipeline needs an explicit ordering discipline, not just “the data is correct because nothing else is running right now.”

Ringing the doorbell – a measured mechanism, not a black box. Kong et al. (NSDI 2023) measured the concrete shape of this step directly: “To post a work request, users need to ring the RNIC’s doorbell through memory-mapped IO (MMIO). Each MMIO has a fixed aligned size 64-byte.” The same paper traces the full path from there: the doorbell write reaches the adapter, which fetches the WQE the doorbell referenced from host DRAM, resolves whatever context it needs (this QP’s live state, an SGE’s lkey) against on-chip caches – falling back to a DRAM fetch on a miss, exactly the cost QP Caching and Key Caching already quantified – and only then proceeds to build a packet. The same paper’s own hardware diagram names a “TX Pipeline (Processing Unit)” and an “RX Pipeline (Processing Unit)” as separate blocks sharing common cache resources between them – an independent, measured source corroborating this book’s own master-diagram split between the two.

Aside: BlueFlame – a named, non-standard optimization

Mellanox’s BlueFlame mechanism lets software write a WQE’s bytes directly into a UAR (User Access Region – device memory mapped for direct userspace access) in the same operation as ringing the doorbell, rather than writing to the ring and separately notifying hardware. Copying those bytes safely takes real care: the copy has to use instructions that bypass the CPU’s own cache and preserve write ordering, because an ordinary memory copy can reorder the underlying writes in ways that don’t matter when nothing else is reading the destination, but matter a great deal when hardware may be reading these bytes as they arrive rather than after a guaranteed-complete copy. What’s public: the general concept (WQE bytes travel with the notification instead of a separate index update) and the existence of a doorbell record tracking a posted-WQE counter. What’s not public: current-generation register-level layout, and whether every vendor’s equivalent mechanism works identically. Treat BlueFlame as a real example of “notification plus payload in one write” going one step beyond a plain producer-index doorbell, not as the one universal mechanism every RNIC uses. Nor as a mechanism that scales to every WQE size: writing an entire WQE via non-temporal MMIO stores costs real CPU-side work per byte, and no source this book found states the specific size ceiling above which a real implementation stops routing a WQE through BlueFlame and falls back to the ordinary ring – only that the technique is real, and that “small” is doing unquantified work in that sentence.

What happens next. A doorbell ring (BlueFlame or ordinary) is where this chapter’s job ends and the fetch side begins: getting that referenced WQE, or several WQEs ahead of when they’re strictly needed, actually resident on-chip is WQE Prefetch’s subject next.

Why

A ring buffer addressed by masking rather than modulo, and an off-by-one capacity rule that never lets producer and consumer collide ambiguously, are both instances of the same design principle: pay a small, fixed, one-time cost (one wasted slot; a power-of-two size constraint) to make every subsequent operation cheap and branch-free, rather than handling the ambiguous case explicitly on every single push and pop. BlueFlame makes a related trade at a different layer: writing WQE bytes directly into a UAR alongside the doorbell costs more CPU-side work per post (a larger MMIO write, non-temporal stores to bypass cache) in exchange for skipping the ring’s own host-memory round trip on the hardware side for small, latency-sensitive messages – a trade that only pays off when the message is small enough that the extra CPU-side copy is cheaper than the PCIe fetch it avoids, which is exactly why it exists as an optional path rather than replacing the ordinary ring entirely.

WQE Prefetch is next: what happens between a doorbell ring and a WQE actually being on-chip and ready for the pipeline to build a packet from.

References

  • No IETF or IBTA standard governs Send Queue ring internals, doorbell register layout, or vendor-specific write-combining mechanisms like BlueFlame – these are implementation-specific silicon design, not a wire-format or software-interface contract any standards body defines. This chapter’s treatment is grounded in the sources cited in-text above, not a standards document.
Last updated on
The WQE and the Send Queue DescriptorWQE Prefetch