The WQE and the Send Queue Descriptor

Overview

A Work Queue Element (WQE) is, architecturally, nothing more exotic than a command descriptor in a producer-consumer ring – this chapter uses “WQE” and “Work Request” interchangeably throughout, the same object considered as a ring entry versus as the thing software constructs to become one. What’s RDMA-specific is only the fields inside it, not the mechanism: what software actually writes into a Send Queue ring is a generic, transport-agnostic description of one operation – an opcode, a scatter-gather list (SGE list) naming local bytes, and whatever remote-addressing fields that opcode needs. It is not the wire packet What Is RoCEv2 already described field by field, and it is not necessarily what a specific RNIC’s own silicon stores on-chip either – three distinct shapes sit between “software posts a request” and “a BTH/RETH/AETH packet leaves the port,” and this chapter is about the first of them.

Locator

One Transaction, Start to Finish already walked a Work Request from posting to wire, and named the doorbell that tells hardware new work exists without saying what, exactly, that work looks like sitting in memory. This chapter opens that box: the WQE itself, before any pipeline stage has touched it.

Block diagram

Deep walkthrough

What the standard actually requires, versus what a software-facing shape makes concrete. IBTA-V1 specifies a Work Request’s required content at a semantic level per opcode – an RDMA Write names a remote address, an rkey, and a local data segment list; it does not specify a C struct, a field order, or a byte layout. A generic, transport-agnostic software descriptor is one concrete way of communicating that requirement to hardware, not the requirement itself. That distinction matters before going further: the standard sets what a Work Request must communicate; a software-facing descriptor shape is one way of communicating it; and, one layer further still below, a specific implementation’s on-chip format is a third, independent thing again – three different shapes this chapter is about to walk through, only one of which any standards body actually specifies.

The software-facing shape, and what it deliberately doesn’t commit to. A generic work-request descriptor, the kind any transport-agnostic software API exposes to an application, is deliberately thin: a chaining mechanism for posting several requests in one call, a request identifier for matching a future completion back to this specific request, a scatter-gather list naming local bytes – source bytes for SEND and WRITE, the destination for a READ’s or an atomic’s returned data – each entry an address, a length, and the local key The Memory Region Descriptor already covers field by field, an opcode from the catalog The Operation Catalog already enumerated, a remote address and rkey for whichever opcode needs one, a flags field, and either an immediate-data value or an invalidate-rkey value depending on which opcode this is. Nowhere in this shape is there a BTH, a PSN, or an ICRC – those are hardware’s job to produce, not software’s job to supply. This is the same abstraction-boundary gap this book has already named at the doorbell (One Transaction, Start to Finish) and The QP Table’s internal organization: a stable, checkable software-facing shape exists, and it is not a claim about what any specific RNIC stores internally.

A real on-chip shape, shown specifically to make that gap concrete rather than asserted abstractly. At least one shipped RNIC’s on-chip control segment, confirmed directly against openly published low-level source rather than assumed, packs a WQE’s control information into exactly four 32-bit fields: an opcode field, a combined field folding two pieces of information together (a send-queue identifier and how many data segments this WQE carries, the latter constrained to 6 bits, capping a single WQE at 63 data segments), a flags field, and a vendor-metadata field. Compare this to the software-facing shape’s separate, independent segment-count field: the same information (how many scatter-gather entries this operation touches) exists in both shapes, but real hardware packs it into a shared, width-constrained field alongside a send-queue identifier, while the software-facing shape keeps it as its own untyped value. Neither shape is wrong – they’re solving different problems. Software’s shape optimizes for a stable, transport-agnostic API surface across every vendor’s hardware; a real on-chip shape optimizes for what actually fits efficiently into a fixed-width control segment a pipeline stage can parse in hardware. This is one confirmed, checkable example, not a claim that every RNIC packs a WQE control segment identically.

Why “control segment” is the right name for it, not just a label this chapter chose. A separate documented real implementation splits its own packed WQE into two explicit pieces – a fixed-size control segment carrying exactly the fields above, followed by a variable-length data segment holding the SGE list or inline payload – and only the control segment ever crosses the doorbell write itself, not the WQE’s data segment. That split is the concrete reason a real WQE isn’t one flat structure: the control segment is sized to fit whatever the doorbell mechanism can carry in one write, and the data segment – SGEs or inlined bytes – trails behind it in the ring, fetched or read separately. This chapter treats it as a second, independently sourced data point alongside Kong et al.’s field count above, not a claim that the two describe the identical structure.

What the flags field actually carries. That field isn’t a placeholder – four independent, opcode-agnostic requests live there, each changing how this operation is scheduled, reported, or handled rather than what it does. An ordering request, commonly called a fence, holds this Work Request’s own issue until every earlier RDMA read and atomic operation posted to the same Send Queue has completed – RDMAmojo’s documentation states the guarantee directly: fencing “guarantees the processing order and Completion notification of Send Requests that were posted to the same Send Queue.” It’s a purely local scheduling constraint; nothing about it reaches the wire. A completion-request flag selects whether this specific operation generates a completion at all once it finishes, since an application doesn’t need a Completion Queue entry for every operation it posts, only the ones it actually intends to wait on – also purely local. A solicited-event request is the one flag here that does reach the wire: it’s what sets BTH’s SE bit (What Is RoCEv2), asking the responder to notify its application through an asynchronous event rather than leave it to find the completion by polling. An inline request asks hardware to copy the payload directly into the WQE itself rather than leave it in the source buffer for a later DMA read – covered in depth two paragraphs below, since it’s the one flag with a hardware-enforced size ceiling attached to it. Single-QP Pipeline Stalls picks the fence request back up as a real stall cause; this chapter’s job is just naming that all four exist as flags a Work Request carries.

Measured sizes – concrete numbers, not qualitative gestures. Kong et al. (NSDI 2023), measuring real ConnectX hardware directly, give this chapter its measured anchor: “A WQE for RC/UC is 36-byte, and 68-byte for UD.” These are base sizes – a control segment plus room for a minimal SGE fan-out, not a ceiling; the on-chip control segment’s own segment-count field allows up to 63 data segments per WQE, so a WQE with heavy scatter-gather fan-out runs far past either base figure. UD’s larger base size tracks directly with what QP Context already established – a UD QP names its destination per-message rather than once at connection setup, so its WQE has to carry addressing information RC/UC’s WQE doesn’t need, because RC/UC already fixed the destination QP number at INIT→RTR.

The inline/DMA fork – a software request, with a fixed threshold enforcing it. Inline placement isn’t something hardware decides unprompted by payload size alone – it’s a request software makes via the inline flag named above, and the same paper measures the ceiling that request is checked against: a “maximal inline size (28-byte).” A payload within that threshold, requested inline, gets copied directly into the WQE itself, avoiding a separate PCIe transaction to fetch it later; a payload above the threshold can’t be inlined regardless of the flag, and falls to the adapter issuing its own DMA read for the data instead, a mechanism The Send Queue Ring picks up from the fetch side and DMA Engine’s DMA Engine owns in full. Inline carries two real consequences beyond just skipping a memory transaction: the payload is never referenced by an SGE at all, so there’s no lkey to check against the Key Table for it, and the source buffer is safe for software to reuse the instant the post call returns, since hardware has already copied the bytes it needs rather than deferring a read against memory software might otherwise overwrite. It’s the first of several places in this part where a small, fixed-size number decides whether an operation costs one memory transaction or two – “the adapter fetches the WQE” isn’t the end of the story.

Where this is checked, and what happens next. Nothing about the WQE’s own fields is validated in isolation – the opcode has to be one this QP’s transport actually supports (The Operation Catalog, gated by the QP’s own transport type from QP Context), the lkey on every SGE has to resolve against a permission-matching Memory Region entry (The Key Table) – except an inlined payload, which carries no SGE and so has no lkey to check at all – and the QP itself has to be in RTS (QP State Machine) before any of this is allowed to leave host memory at all. This chapter’s job stops at naming what the WQE contains and how its shape differs from both the software struct above it and the wire packet below it; The Send Queue Ring is where a WQE actually gets fetched off the structure holding it.

What happens when one of those checks fails. These three checks aren’t all the same kind of check, and that difference matters for what “failure” actually looks like. An opcode that doesn’t match the QP’s transport type is knowable from the WQE and the QP’s own state alone – no pipeline stage, no table lookup, nothing beyond metadata already sitting in host memory, and the same is true of a QP that hasn’t reached RTS yet. An lkey that doesn’t resolve to a valid, permission-matching entry is a different shape of failure: nothing about the WQE or the QP context alone can catch it, since it depends on the live state of the Key Table at the moment this WQE is actually processed, not at the moment it was posted. Whether a real RNIC rejects the first two synchronously at post time versus discovering all three only once the pipeline reaches this WQE isn’t something this chapter’s sources pin down, and this chapter doesn’t invent a specific answer for either case. What’s architecturally certain regardless: none of the three is ever allowed to become a packet on the wire. Single-QP Error Handling owns what a failure actually does to the QP and its outstanding work once one is detected.

The Send Queue Ring is next: the structure that holds one of these WQEs per posted entry, and how hardware finds the next one to fetch.

Field reference

Every field a Work Request carries, and how much of each one IBTA-V1 actually pins down versus leaves to whoever designs the software-facing interface. “Architect’s choice” doesn’t mean arbitrary – it means the standard requires the capability to exist without mandating this particular shape for it, and every real interface converges on something close to what’s below because the underlying requirement is the same one.

What’s inside opcode. A Work Request’s opcode selects one operation family, not a specific wire packet – there’s no FIRST/MIDDLE/LAST/ONLY choice at this level, because hardware decides packet framing once it knows the payload’s length, not software at post time. The Operation Catalog’s wire-level table is what each of these expands into.

ACKNOWLEDGE and ATOMIC_ACKNOWLEDGE don’t appear above – hardware generates both automatically as part of the reliability protocol; software never posts a Work Request asking for one directly.

What’s inside flags. The flags field above contains the following four requests:

References

• InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications).
• The control-segment/data-segment split is sourced to a vendor-manual-citing technical analysis (Alibaba Cloud’s developer blog, itself citing NVIDIA’s own hardware manual for a real WQE’s two-part shape), not IBTA-V1 – consistent with, but independent of, Kong et al.’s field-count figures elsewhere in this chapter.
• This chapter’s fence, completion-signaling, and inline-request descriptions are grounded in verbs-level documentation (RDMAmojo and the public ibv_post_send manual pages) of the behavior, not a direct quotation of IBTA-V1’s own text – the underlying semantic requirement (a Work Request interface capable of expressing ordering, completion-selection, and inline-placement intent) is consistent with, but not verified word-for-word against, the standard’s own Work Request definition. The solicited-event request is the one of the four with a direct wire-visible consequence: it sets BTH’s SE bit, a field IBTA-V1 does define.
• The WQE-level opcode set (one entry per operation family, no FIRST/MIDDLE/LAST/ONLY choice at this level) is confirmed against the same public verbs-level enum documentation as the flags above, not IBTA-V1’s text directly – it matches The Operation Catalog’s own eight-family scope exactly, minus the two hardware-generated acknowledgement opcodes no Work Request ever requests directly.