RDMA READ

RDMA READ is the one family where request and response are structurally distinct opcodes traveling in opposite directions, and that split does real work. A RDMA_READ_REQUEST arriving at the responder doesn’t just trigger a local memory access — it turns the responder into the initiator of a brand-new message, built entirely by its own hardware: pull the requested range out of the memory RETH named, and transmit it back as a RDMA_READ_RESPONSE sequence – carrying the requester’s PSN space, continued from the original request, not a PSN of the responder’s own choosing. Notice that FIRST, LAST and ONLY all carry AETH — the same extended header a plain acknowledgement uses — and only MIDDLE goes without. That’s not incidental: a read response doubles as its own acknowledgement. A WRITE needs a separate, trailing ACK packet to tell the requester its operation landed; a READ’s response payload arriving successfully is the acknowledgement, with no separate ACK message required on top of it. What AETH’s syndrome byte actually communicates, and how a NIC acts on it, belongs to Reliability Engine — the point to carry forward here is just that this family’s hardware cost includes acting as a full message-initiator, not only a memory responder, and that reusing the requester’s own PSN space is exactly what lets the requester’s reliability engine match an incoming response back to the specific outstanding request it belongs to.

Here is an 8 KiB READ, request and two-packet response, with the PSNs on every packet:

All 9 steps

1. The application posts a READ: the remote address and rkey to read from, and a local buffer with its lkey to read into. The WQE and the Send Queue Descriptor
2. The request carries no payload. It takes PSN 2000, and because 8 KiB needs two response packets at this MTU it reserves 2001 too. The next request on this QP starts at 2002. RDMA READ
3. The request goes into the Replay Buffer like any other, and the retry timer arms on PSN 2000. The Replay Buffer
4. The responder checks PSN 2000, then the rkey, this time for remote read permission over all 8 KiB. Permission and Bounds Violations
5. Reads and atomics in flight are capped per QP, at a ceiling both sides agreed when the connection was set up. This request takes one slot. QP Context
6. The responder’s hardware. Its DMA engine reads the 8 KiB, and for this message the responder becomes the sender. The Gather Path
7. READ_RESPONSE_FIRST carries PSN 2000, the request’s own PSN, and an AETH. The requester places it in the buffer its own WQE named; nothing on the wire says where. Multi-Packet Reassembly
8. READ_RESPONSE_LAST closes the message. The response itself is the acknowledgement: the Replay Buffer entry retires and the timer disarms. RDMA READ
9. Only if the Work Request was signaled: The requester writes its completion. By the time the application sees it, the data is already in local memory. CQE Generation

Block by block

The same READ across both hosts. Every arrow that carried data in a WRITE’s walk runs the other way here: RNIC B reads its host’s memory, RNIC B’s TX path builds the response, and RNIC A writes it into Host A’s buffer. RNIC B’s software never takes part.

All 9 steps

1. On Host A (Requester). The application writes a READ WQE, naming a remote address and rkey to read from and a local buffer to read into, and rings the doorbell. The Send Queue Ring
2. RNIC A reads the WQE out of host memory and looks up the QP’s context. WQE Prefetch
3. Nothing. The local buffer is where the data will land, not where it comes from: the Key Cache checks its lkey, and the request is a BTH and a RETH with no payload. It takes PSN 2000 and reserves 2001 for the second response packet. A Replay Buffer entry is recorded and the request leaves. TX Packet Build
4. On Host B (Responder). RX Parse checks the PSN, and the Key Cache checks the rkey for remote read over all 8 KiB. The request also takes one of the QP’s outstanding-read slots. Permission and Bounds Violations
5. RNIC B’s DMA engine. The arrow that carried a WRITE’s payload into Host B’s memory now runs the other way. The Gather Path
6. RNIC B’s TX Packet Build turns the data into a response, the job a requester’s TX path normally does. READ_RESPONSE_FIRST carries PSN 2000, the request’s own PSN, with an AETH and the first 4 KiB. TX Packet Build
7. The response returns (Requester). RNIC A matches PSN 2000 to its outstanding READ and writes the first 4 KiB into the buffer its own WQE named; nothing on the wire says where. READ_RESPONSE_LAST is already on its way behind it. Target Selection
8. The last 4 KiB lands behind the first. Its AETH is the acknowledgement: the Replay Buffer entry retires and the retry timer disarms, with no separate ACK. Message Completion and the MSN
9. Only if the Work Request was signaled: RNIC A writes the completion into Host A’s Completion Queue. By the time the application sees it, the data is already in its buffer. CQE Generation

References

• InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications).