Skip to content
DMA Engine
The Scatter Path

The Scatter Path

Overview

Scatter is this engine’s RX-side job: writing an arriving payload into memory, one already-translated segment at a time. Every scatter this engine ever runs writes into memory this side can address directly – there’s no such thing as writing into memory that’s actually remote from the engine doing the writing. What varies, per Target Selection, is who chose the address: this side’s own software, ahead of time, or the packet that just arrived.

Concept

This is the mechanism PSN Check and Out-of-Order Arrival already named, without describing, as “DMA writeback” – the same handoff, given its own chapter here rather than left as a forward reference.

Locator

The master RNIC block diagram with DMA Engine highlighted and every other block dimmed – this chapter is the RX-side half of this engine’s two data-movement paths, fed by RX PSN Check’s own handoff.

Block diagram

Four trigger cases – RECEIVE, RDMA WRITE, RDMA WRITE with Immediate, an RDMA READ response’s arrival, and an Atomic’s return value – each writing into a destination resolved through Address Translation Handoff, software-named for RECEIVE and both requester-side cases, wire-named for WRITE; WRITE with Immediate drawn with two separate arrows, one into the RETH-named payload destination and a second into a consumed RQ WQE that never receives the payload itself.

Deep walkthrough

RECEIVE – writing where this side’s own software already said to. A SEND arriving at this QP is placed into whichever Receive Queue Element this side posted ahead of time, in FIFO order against however many are queued – software-named, resolved through that WQE’s own SGE list and lkey, exactly as Target Selection already established.

RDMA WRITE – writing where the packet itself said to. No Receive Queue Element is consulted at all. The destination comes from RETH, carried in the packet, resolved against the rkey it names – this side’s software had no role in choosing this address at the moment the packet arrived, even though the memory it lands in is registered, in advance, by that same software.

RDMA WRITE with Immediate – two separate destinations, easy to conflate. The payload itself scatters exactly like a plain WRITE, wire-named via RETH. Separately, this variant consumes one posted Receive Queue Element – not to place the payload, only to give the immediate value somewhere to attach to and to produce a receive completion the application can poll. Treat these as two independent writes with two independent destinations, not one write with two names.

Gotcha

Don’t assume a WRITE with Immediate’s payload ends up in the consumed RQ WQE’s own buffer. It doesn’t – that buffer receives nothing but the immediate value’s bookkeeping. The payload itself goes exactly where a plain WRITE’s would, at the RETH-named address, entirely independent of whichever RQ WQE happened to be next in line.

RDMA READ response – writing back into the request that asked for it. The requester’s own DMA engine scatters an arriving response payload into the local buffers its original request WQE named – software-named, the same SGE list Target Selection already pointed at. In-Flight Transfer State’s completion tag is what makes this possible across a real network round trip: the request’s own destination identity has to survive from the moment it was issued until whenever the response actually arrives, which could be many other packets later.

Atomics – one word, not a list. The pre-operation value an atomic’s acknowledgement carries back – The Operation Catalog already names this precisely, ATMACK’s “returns the pre-operation value,” not the result of the operation – lands in a single local buffer the original WQE named. No scatter-gather cursor applies; an atomic touches exactly one location on each side, never a multi-segment chain.

Where scattering sits relative to the sequence-number gate. PSN Check and Out-of-Order Arrival already establishes that a packet has to clear sequence-number ordering before its payload is safe to place at all – an out-of-order arrival is rejected and resent later, never scattered early and reconciled afterwards. This path never runs ahead of that gate; by the time this chapter’s mechanism starts, the packet it’s serving is already known to be the next one this QP’s receive side expects.

The walk below receives that same SEND into a Receive WQE with a 2 KiB and an 8 KiB buffer, so the first packet splits across both and the second resumes inside the larger one.

The same 7,680-byte SEND arriving as two packets at a receiver whose posted Receive WQE names a 2 KiB buffer and an 8 KiB buffer. The first packet splits across both; the second resumes where it stopped; the completion reports how much of the posted space was used.
Scattering one SEND into two buffers
All 9 steps
  1. Packet 1 arrives. SEND_FIRST, PSN 100, arrives with 4,096 bytes of payload. PSN Check and Out-of-Order Arrival
  2. PSN 100 is the expected PSN, so the payload may be placed. A packet that failed this check would never reach the scatter engine at all. The Scatter Path
  3. A SEND names no address. The first packet of the message takes the next Receive WQE that software posted, and that WQE’s SGE list decides where the bytes go: 2 KiB in MR 3, then 8 KiB in MR 4. The Receive Queue Descriptor and Ring
  4. SGE 0’s lkey is translated, with a local-write check this time. Address Translation Handoff
  5. They go on into SGE 1. First SGE 0 is filled, all 2,048 bytes of it.
  6. Then SGE 1 is translated and the second half of the packet lands at its start. One packet, two buffers in two regions.
  7. Packet 2 arrives. SEND_LAST, PSN 101, is next in sequence. It belongs to the message already in progress, so it uses the same Receive WQE and the cursor where packet 1 left it. Multi-Packet Reassembly
  8. All 3,584 bytes fit in what is left of SGE 1, starting at offset 2,048.
  9. Done. The receive CQE carries the byte count, 7,680. The last 2,560 bytes of SGE 1 are left untouched. The opposite case is not a truncation: a SEND larger than all the posted SGEs together is an error, and on RC the receiver NAKs it as an invalid request. CQE Generation

Where this connects. A completed scatter is what CQE Generation is waiting on before it can write a receive-side completion – unconditionally, per that chapter’s own asymmetric rule for receive-side events. What happens if a scatter can’t complete – the destination faults partway through a multi-segment write – is Single-Burst Fault Handling’s question, not this chapter’s.

Interface reference

Trigger opcodeDestinationAddressing (per Target Selection)
RECEIVE (SEND arriving)receiver’s posted RQ WQE SGE listsoftware-named
RDMA WRITERETH virtual address, rkeywire-named
RDMA WRITE w/ Immediatepayload: RETH address (wire-named); RQ WQE consumed only for immediate + completionmixed, see Gotcha above
RDMA READ responserequester’s original WQE SGE listsoftware-named
Atomic acknowledgementrequester’s original WQE buffer (single location, pre-operation value)software-named

References

  • InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 – governs RETH, ATMACK, and the receive-completion semantics this chapter’s scatter destinations are derived from, already cited in full in The Operation Catalog.
  • Linux kernel soft-RoCE driver, drivers/infiniband/sw/rxe/ – rxe_resp.c places a SEND’s payload with the same copy_data() cursor walk over the Receive WQE’s SGE list, checking local-write access per lkey; running out of SGEs returns -ENOSPC, which an RC QP answers with an invalid-request NAK rather than truncating.
Last updated on
The Gather PathOrdering and Outstanding-DMA Tracking