Skip to content
TX Pipeline

TX Pipeline

Every RDMA operation described so far in this book assumed a WQE somehow becomes bytes on the wire. This part opens up that assumption: a Work Queue Element sitting in a ring in host memory, fetched by hardware, prefetched ahead of when it’s actually needed, built into a real BTH/RETH/AETH packet with a valid ICRC, and arbitrated onto a pipeline shared by every other QP with traffic ready to send at the same instant – plus what happens when one QP’s send path stalls without taking any other QP down with it, and what happens when a QP resets with a WQE already mid-flight through one of these stages.

TX Pipeline has no state machine of its own – it runs under the gate the QP’s own state machine (QP State Machine) already provides. Installing a Send Queue’s ring happens as part of Installing a QP, not as a separate handoff. What this part covers instead: the WQE ring and its on-chip prefetch buffer, the scheduler that arbitrates many QPs onto one shared pipeline, and TX Packet Build – the stage that turns a fetched WQE into the wire format What Is RoCEv2 already described field by field.

One WQE, from post to wire – what the TX Pipeline does to one Work Queue Element, from posting through the ring, prefetch, and packet build, to the hazards that can interrupt it (single-QP stalls, multi-QP scheduling, mid-transaction reset), plus the responses the same packet-build stage produces for the other side. Each step names the chapter that covers it.

Chapters

The WQE and the Send Queue Descriptor
A WQE and a wire packet are not the same shape. Software builds one – a generic descriptor any transport can use, every field and flag named here. Hardware builds the other from it – opcode, BTH, RETH, payload, ICRC. This chapter is about the shape that exists in between the two: what a Send Queue actually holds before any of that translation happens.
The Send Queue Ring
Posting a Work Request doesn’t move a single byte – it writes a WQE into a ring in host memory and rings a doorbell. This chapter is the ring itself: how it wraps around without a modulo, how a shared index survives being touched by both an untrusted process and a privileged hardware pipeline, and the one optimization that skips the ring’s own memory entirely.
WQE Prefetch
Every caching chapter so far in this book resolves a key or a QP number that could arrive in any order. A Send Queue ring doesn’t work that way – it’s consumed strictly in order, front to back – and that one difference is why this chapter’s on-chip structure is a prefetch buffer, not a lookup cache, even though it’s solving the exact same PCIe-latency problem.
TX Packet Build
Every other chapter in this part is about getting a WQE on-chip. This one is about what happens the instant it is: the stage that actually turns a fetched descriptor into the BTH/RETH packet this book described field by field back in Overview, and closes it out with an ICRC covering exactly the fields that survive the trip unchanged.
Single-QP Pipeline Stalls
A large RDMA operation and a small, latency-sensitive one sharing the same TX pipeline sound like a scheduling question. A measured study says it’s actually simpler and worse than that: the pipeline serves whatever is ready to send, in the order it became ready, and a fast RDMA write behind a slow one measurably collapses – before any scheduling policy even gets a turn to decide otherwise.
Multi-QP TX Scheduling
Which QP’s WQE gets built next, when a dozen QPs all have one ready at once, is the same shape of question Queue Pair Management asked about QP Cache eviction – round-robin, weighted, priority – asked here of pipeline cycles instead of cache slots. A vendor’s own quoted admission, and a real study that measured DCQCN failing to fix the problem, both ground this chapter in more than reasoning alone.
Hardware-Assisted TX Scheduling Engines
Queue Pair Management asked whether an eviction policy runs as fixed silicon, a programmable engine, or a software trap. This chapter asks the same question of TX scheduling – and unlike Queue Pair Management’s version, it has dated, concrete evidence that at least some shipped RNICs sit at the software-driven end of that spectrum, not by choice so much as by not having built the alternative yet.
Mid-Transaction TX Reset
Mid-Transaction QP Reset closed Queue Pair Management asking what happens to a DMA write already mid-burst when RESET lands. This chapter asks the TX-pipeline-specific version of the same question – and unlike that chapter, it has a named, shipped RNIC on record as simply unable to answer it in hardware at all.
Response Generation
Every other chapter in this part follows a packet the local software asked for. This one covers the packets nobody posted a WQE for: the ACKs, NAKs, READ responses and ATOMIC acknowledgments a responder owes, what decides which one goes back, and when it is allowed to wait.
Last updated on