The Gather Path

Overview

Gather is this engine’s TX-side job: reading local memory, one already-translated segment at a time, to assemble a payload before it leaves the port. Three opcodes trigger it – SEND, RDMA WRITE, and an RDMA READ response’s own construction at the responder – and none of them can proceed to TX Packet Build until this path finishes.

Locator

Block diagram

Deep walkthrough

The inline fork happens before this chapter ever starts. The WQE and the Send Queue Descriptor already covers this precisely: a payload within the inline threshold is copied directly into the WQE by software at post time, and never touches a scatter-gather entry or an lkey at all. This path exists only for what’s left – a payload that couldn’t or wasn’t requested to be inlined, and that this engine has to go fetch itself, “a mechanism The Send Queue Ring picks up from the fetch side and DMA Engine owns in full,” in that chapter’s own words.

Each segment is its own lookup. A scatter-gather entry carries its own address, length, and lkey independently – The WQE and the Send Queue Descriptor already established that a single operation can span up to 63 data segments, each potentially naming a different Memory Region entirely. In-Flight Transfer State’s scatter-gather cursor is what tracks which segment is active; Address Translation Handoff gets called once per segment, not once per operation, since nothing guarantees consecutive segments even resolve through the same Key Cache entry.

SEND and RDMA WRITE – reading this side’s own memory to build a message. Both gather from a source their own software named at post time. The bytes this path reads become the message body a SEND carries, or the payload a WRITE’s own RETH will tell the far end where to place – this engine doesn’t know or care which opcode it’s serving at the byte-movement level; the opcode only decided which SGE list to walk.

RDMA READ response – gathering from wherever the request pointed, not from anything this side posted. Target Selection already named this the one case where gather and wire-named addressing coincide: the responder services a READ request by gathering from the local memory the request’s own RETH named, not from a WQE this side’s software wrote. The result is a response message this engine still has to assemble the same way it assembles any other outgoing payload – gathered, one segment at a time, and hand it onward.

Where gathering sits relative to the rate limiter’s gate. Rate-Limiter Context already established that a rate-limited QP’s dispatch decision happens “before the adapter spends any further effort on a WQE,” and The Rate Limiter Table sharpens that to a specific point: “before a scheduled WQE is allowed to proceed toward becoming a packet” – the same reasoning argues gathering should wait for that same gate, not run ahead of it. Fetching payload bytes for a transfer the rate limiter is about to hold serves no purpose and burns memory bandwidth a still-throttled QP has no immediate use for. This book states that ordering as the sensible design, not a confirmed cycle-accurate pipeline fact from any specific implementation.

The walk below gathers one 7,680-byte SEND from three buffers at a 4,096-byte path MTU. The first packet ends partway through the second buffer, and the cursor picks up from there for the next one.

All 12 steps

1. The WQE. Software posts a SEND whose payload is spread over three buffers: 1 KiB and 512 B in MR 1, 6 KiB in MR 2. That is 7,680 bytes, far too many to copy into the WQE inline, so the DMA engine has to fetch them. The WQE and the Send Queue Descriptor
2. Two. Each packet takes the path MTU or whatever is left, whichever is smaller: 4,096 bytes now, 3,584 next. The first packet ends 3,072 bytes into SGE 1. The packet boundary and the buffer boundaries have nothing to do with each other. The Gather Path
3. Packet 1. The cursor is on SGE 0, so its lkey is translated first: does MR 1 exist, may this QP’s protection domain read it, and where is 1 KiB at that address physically? Address Translation Handoff
4. All of SGE 0 fits, so all 1,024 bytes are read. The cursor runs off the end of SGE 0 and moves to SGE 1, offset 0.
5. SGE 1 is a different region with a different lkey, so it needs its own lookup. Even SGE 2, which is in MR 1 again, will get its own: a translation belongs to a segment, not to the operation. Address Translation Handoff
6. At offset 3,072 inside SGE 1. Only what fits is read; the cursor keeps its place so the next packet can resume there. In-Flight Transfer State
7. The full payload goes to TX Packet Build, which puts the headers in front and the ICRC behind. TX Packet Build
8. Packet 2. The next packet starts where the cursor stopped. SGE 1’s lkey is looked up again for this packet; the Key Cache is what makes a repeat lookup cheap. Key Caching
9. The rest of SGE 1, 3,072 bytes, is read, and the cursor moves to SGE 2.
10. SGE 2 is translated and its 512 bytes read. Bytes left reaches zero exactly as the cursor reaches the end of the last SGE.
11. The last packet of the message carries AckReq, so the receiver will answer once both packets are in. Response Generation
12. Both packets leave. NIC B sees two packets and nothing of the three buffers they came from; how they land is up to NIC B’s own SGE list. The Scatter Path

Where this connects. A completed gather hands its assembled payload to TX Packet Build, which wraps it in whatever header the opcode requires and computes ICRC over the result. What happens if a segment’s own lookup fails partway through a multi-segment gather – some bytes already fetched, the rest never will be – is Single-Burst Fault Handling’s question, not this chapter’s.

Interface reference

References

• No IBTA or IETF standard names a “gather path” as its own mechanism – the opcode behavior above is derived from RETH and SGE-list semantics already covered field by field in The Operation Catalog and The WQE and the Send Queue Descriptor.
• Linux kernel soft-RoCE driver, drivers/infiniband/sw/rxe/ – rxe_req.c sizes each packet as the smaller of the path MTU and the bytes left in the message, and copy_data() in rxe_mr.c fills it by walking the WQE’s SGE list from a saved cursor (cur_sge, sge_offset, resid), looking up each SGE’s lkey as the cursor reaches it. The walk’s per-packet model follows this driver.