TX Packet Build

Overview

This stage is the last step of a fetch-produce-issue sequence, run for RDMA specifically: everything the previous three chapters covered – the WQE’s fields, the ring holding it, the on-chip prefetch getting it there ahead of need – exists to feed this one stage, turning a fetched WQE into the exact wire packet What Is RoCEv2 already described byte-for-byte. This is where a WQE’s opcode and scatter-gather list actually become a Base Transport Header, a RETH if the opcode needs one, and a closing ICRC – the concrete moment software’s shape and hardware’s wire shape finally converge.

Locator

WQE Prefetch covered getting a WQE on-chip ahead of need. This chapter is what happens the instant it’s needed: the stage that turns a fetched WQE into the wire packet What Is RoCEv2 already described field by field.

Block diagram

Deep walkthrough

What this stage actually assembles, field by field, already specified elsewhere. A Base Transport Header carries this WQE’s opcode, the destination-QP field this QP fixed once at INIT→RTR (QP Context) and never revisits per-packet, and a PSN drawn from this QP’s own live send-side PSN counter – advancing under hardware’s own control, no software write involved, exactly as QP Context already establishes. If this WQE’s opcode requires one – an RDMA Write or Read, on the first or only packet of the message – a RETH carries the remote virtual address and the rkey the WQE itself supplied. This side does not resolve that rkey against anything: it is an opaque value naming a Memory Region on the remote peer, forwarded as-is: the responder is who resolves it against its own Key Table on arrival, exactly the way The Key Table describes for the remote-side case – the mirror image of the local lkey lookup this side performs for its own local reads and writes. None of these fields’ shapes are new information this chapter introduces – What Is RoCEv2 already fixed every bit width and byte offset, and The Operation Catalog already fixed which opcodes carry which extended headers. This chapter’s actual job is narrower and more honest about it: naming that a real pipeline stage has to produce those already-specified shapes from a WQE’s fields, not re-deriving the shapes themselves.

Scope: a WQE-triggered request, not every packet this stage will ever build. Everything above describes a packet this QP’s own software posted a WQE for, drawing its own PSN forward under its own control. Two packet types this same stage builds don’t follow that rule. An ACK carries an AETH, not a RETH, and is built and sent the same way any other packet is (One Transaction, Start to Finish) – but it isn’t triggered by a WQE this QP posted, and it doesn’t draw a new PSN from this QP’s own counter at all. A RDMA_READ_RESPONSE is a sharper exception still: The Operation Catalog already established that its PSN is “the requester’s PSN space, continued from the original request, not a PSN of the responder’s own choosing” – the opposite of the WQE-triggered rule this chapter just described. Both are real packets this stage builds; neither is this chapter’s subject. Response Generation covers what triggers them, which PSN each carries, and how the AETH is built.

Where this is checked, and the honest limit of what’s public here. No source this book found describes the internal microarchitecture of this specific stage – whether BTH, RETH, and payload assembly happen as one combined operation or several internal pipeline sub-stages, whether a single hardware unit builds every opcode’s packet or different opcodes route through different logic, is not publicly documented for any real RNIC. This is the same honest limit QP Table and Key Table already hit describing their own internal addressing mechanism – a real design choice exists, and no vendor’s public documentation commits to one. What can be said with confidence, because it follows directly from what every opcode’s required fields already are: this stage cannot complete before the fields it needs – this QP’s live PSN, a resolved rkey if one’s required, the payload itself – are all actually available, which is exactly the dependency Single-QP Pipeline Stalls picks up next.

ICRC – computed last, and only over what actually survives the trip. What Is RoCEv2 already established which fields the invariant CRC covers – everything except the handful of fields a router or switch is allowed to modify in flight (Ethernet addresses, IP TTL/hop limit, and similar mutable framing). This stage is where that computation actually has to happen, and it has to happen after every covered field has its final value – a BTH/RETH built with a placeholder PSN or an unresolved address, then patched afterward, would need the ICRC recomputed against the final bytes, not the placeholder ones. This chapter doesn’t resolve whether a real pipeline computes ICRC incrementally as each covered field is finalized or in one pass at the end – another instance of the same internal-sequencing question the paragraph above already flagged as undocumented.

What happens next. A built, ICRC-closed packet is what actually leaves the port – the exact bytes What Is RoCEv2 described crossing the wire, now a real instance of that shape rather than a description of it. This chapter’s job ends here; what happens if this stage never gets to run at all – because the WQE feeding it stalled, or the QP feeding it reset mid-build – belongs to the next two chapters, not this one.

Single-QP Pipeline Stalls is next: what happens when one of this stage’s required inputs – a resolved rkey, an available PSN, the payload itself – isn’t ready yet, and whether that stall touches any other QP sharing this same pipeline.

References

• InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications).
• InfiniBand Trade Association, Supplement to InfiniBand Architecture Specification Volume 1, Release 1.2.1, Annex A17: RoCEv2 (2014).