Skip to content
Overview
The Operation Catalog
RDMA WRITE

RDMA WRITE

OpcodeExtended headersNotes
RDMA_WRITE_FIRSTBTH + RETHopens a multi-packet WRITE, carries address/rkey/length
RDMA_WRITE_MIDDLEBTHcontinuation packet
RDMA_WRITE_LASTBTHcloses the message
RDMA_WRITE_LAST_WITH_IMMEDIATEBTH + IMMDTcloses the message, carries immediate data — no RETH
RDMA_WRITE_ONLYBTH + RETHsingle-packet message
RDMA_WRITE_ONLY_WITH_IMMEDIATEBTH + RETH + IMMDTsingle-packet message with immediate data

RETH appears only on FIRST and ONLY: the destination address, rkey, and length only need stating once per message, at its start, not repeated on every packet of it.

Gotcha

RDMA_WRITE_LAST_WITH_IMMEDIATE does not carry RETH — it matches plain RDMA_WRITE_LAST, not the ONLY variants. The address was already established back on FIRST and doesn’t need restating just because this packet happens to carry immediate data too.

RDMA WRITE is one-sided: the responder’s hardware performs the memory write directly against the address RETH named, with no Receive Request consumed and no completion generated on the responder side — the responder’s software may never even know a plain WRITE happened, which is precisely the point of a one-sided operation. Silent to software is not the same as silent on the wire, though: RC’s reliability contract still requires every WRITE to be acknowledged back to the requester so the reliability engine can retire its replay-buffer entry and disarm its retry timer (Reliability Engine) – the responder’s application may never see a plain WRITE, but the responder’s hardware still owes the requester’s hardware an ACK.

A message longer than the path MTU is where RETH-on-FIRST-only earns its keep. Play a 12 KiB WRITE leaving as three packets and watch what the responder has to remember between them:

A 12 KiB RDMA Write at a 4 KiB path MTU: WRITE_FIRST carries the RETH, WRITE_MIDDLE and WRITE_LAST carry only payload and land through the responder's reassembly context, and one ACK for the last PSN retires all three packets on the requester.
RDMA Write in three packets
All 10 steps
  1. The application posts one Work Request for 12 KiB. The path MTU is 4 KiB, so the message will leave as three packets. The WQE and the Send Queue Descriptor
  2. TX Packet Build gathers the source bytes under the WQE’s lkey and cuts them at the MTU. Only the first packet gets a RETH. TX Packet Build
  3. Each packet gets a Replay Buffer entry as it leaves. The retry timer watches only the oldest outstanding PSN, 1000. The Replay Buffer
  4. WRITE_FIRST carries the RETH: the target address, the rkey, and the length of the whole message, not just of this packet. RDMA WRITE
  5. The responder checks PSN 1000, then resolves the rkey. Because the RETH states the full length, the permission and bounds check covers all 12 KiB at once. Permission and Bounds Violations
  6. The first 4 KiB lands at the RETH address. For the rest of the message the responder keeps two numbers: where the next byte goes, and how many bytes are left. Multi-Packet Reassembly
  7. From the reassembly context. WRITE_MIDDLE is a BTH and payload, nothing else; its bytes go where the context points, and the context advances. Multi-Packet Reassembly
  8. WRITE_LAST places the final 4 KiB and brings the count to zero, closing the context. A plain WRITE consumes no Receive WQE and writes no CQE on this side. Multi-Packet Reassembly
  9. One. ACKs are cumulative, so ACK 1002 confirms 1000 and 1001 as well. All three Replay Buffer entries retire and the retry timer disarms. Message Completion and the MSN
  10. Only if the Work Request was signaled: MSN 1 says one whole message completed, so the WQE retires and a send completion is written. Message Completion and the MSN

Block by block

A single-packet WRITE across both hosts. Compare it with SEND’s walk: RNIC B places the payload at the address the packet carries, and its Receive Queue fetch, CQE generation and event blocks never light.

A single-packet RDMA WRITE across both hosts: Host A's doorbell, WQE read and payload read, the packet with its RETH on the wire, RNIC B's rkey check and payload write straight to the named address, and the ACK back. No Receive WQE, completion or event on Host B, until the last scene adds immediate data.
RDMA WRITE, block by block
All 13 steps
  1. On Host A (Requester). The application writes a WRITE WQE into the Send Queue ring, naming a local source buffer and a remote address and rkey, and rings the doorbell. The Send Queue Ring
  2. RNIC A reads the WQE out of host memory and looks up the QP’s context. WQE Prefetch
  3. The Key Cache checks the lkey on the source buffer, then the DMA engine reads the 4 KiB payload out of Host A’s memory. The Gather Path
  4. TX Packet Build puts a BTH and a RETH in front of the payload. The RETH carries the remote address, the rkey and the length: everything the responder needs to place the bytes. A Replay Buffer entry is recorded and the packet leaves. TX Packet Build
  5. On Host B (Responder). RX Parse checks the PSN, then the Key Cache resolves the rkey. It must grant remote write over all 4,096 bytes at that address. Permission and Bounds Violations
  6. No. The RETH already named the address, so the DMA engine writes the payload straight there. The Receive Queue is never read. The Scatter Path
  7. The ACK for PSN 1000 leaves through RNIC B’s TX path. That is all: no completion, no event. Host B’s software is not told a WRITE happened. Message Completion and the MSN
  8. The ACK returns (Requester). RX Parse on RNIC A matches the ACK against the Replay Buffer, retires the entry and disarms the retry timer. Retry and Timeout State
  9. Only if the Work Request was signaled: RNIC A writes the send completion into Host A’s Completion Queue. CQE Generation
  10. One: the memory it was written into. Host B’s application, its Receive Queue and its Completion Queue never took part. That is what one-sided means. Target Selection
  11. With Immediate (Responder). The payload still lands at the RETH address. But the immediate value has to reach software, so RNIC B now consumes a Receive WQE. It uses only the WQE’s wr_id, which tags the completion; the scatter list is ignored. The Receive Queue Descriptor and Ring
  12. After the payload write, RNIC B writes a receive CQE carrying the immediate value and the byte count. A one-sided WRITE has become one Host B’s software is told about. The CQE
  13. Only if the CQ is armed: RNIC B raises a completion event, so an application sleeping on the CQ wakes and reads the immediate value from the CQE. Event and Multi-CQ Arbitration

WRITE with Immediate

The WITH_IMMEDIATE variants break that quietly-one-sided property on purpose: immediate data has to surface to the application somehow, so those variants do consume a Receive Request and do generate a completion, turning an otherwise-invisible WRITE into something the responder’s software is explicitly notified about. The address still comes from RETH; the Receive WQE supplies only somewhere to report.

The last scene of the WRITE walk above plays this variant: the payload still lands at the RETH address, and a Receive WQE is consumed only to report the completion. One Transaction, Start to Finish follows a WRITE with Immediate through every stage of both adapters, packet by packet, and names the chapter that owns each one. One All-Reduce, Four Hosts shows the variant in bulk use: NCCL ends every chunk it moves with a WRITE with Immediate, so a single receive completion tells the receiver the whole chunk has landed.

References

  • InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications).
Last updated on
SENDRDMA READ