SEND

The invalidate variants exist only on LAST and ONLY, never FIRST or MIDDLE — invalidation is something that happens once, at the point a message actually completes, not something meaningful to attach to a packet in the middle of one.

SEND is RDMA’s two-sided operation: every SEND message, in any variant, consumes a Receive Request the responder posted in advance — there is no such thing as a SEND arriving with nowhere to land, by construction, because the responder’s hardware is matching it against a receive queue it already provisioned. That consumption is the hardware fact that makes SEND fundamentally different from the one-sided families: it’s the one opcode family that requires the responder’s software to have done work ahead of time.

All 9 steps

1. Before anything is sent, the responder’s application posts a Receive WQE naming a buffer. That WQE is where an incoming SEND will land. The Receive Queue Descriptor and Ring
2. The requester posts a SEND. It names no remote address: where the bytes go was decided in advance, by the responder. SEND
3. The TX engine gathers the source buffer under the WQE’s lkey and stamps the next PSN. TX Packet Build
4. SEND_ONLY is a BTH and a payload, nothing more. The responder parses it and compares its PSN with the one it expects. PSN Check and Out-of-Order Arrival
5. Two things could stop it here. A PSN gap draws a sequence-error NAK; an empty Receive Queue draws an RNR NAK. This time neither happens. The Retry and RNR Timeout State Machines
6. Into the buffer named by the Receive WQE at the head of the RQ, taken strictly in order. The DMA engine places the payload there. The Receive Queue Descriptor and Ring
7. A receive completion is written to the responder’s CQ. A plain WRITE would leave the responder’s software unaware; a SEND always tells it. The ACK is already crossing back; it does not wait for the completion. CQE Generation
8. The ACK for PSN 1000 retires the requester’s Replay Buffer entry and disarms the retry timer. The Replay Buffer
9. Only if the Work Request was signaled: MSN 1 says one whole message completed, so the WQE retires and the send completion posts. Message Completion and the MSN

That pre-posting discipline plays out like this: the responder posts its Receive Request entirely out of band, before any packet arrives, and the only coordination at SEND-time is the RX engine popping whatever WQE is sitting at the head of the RQ ring, in strict order. Two failure branches fall directly out of that design — a PSN that arrives higher than the receiver’s expected value gets NAK’d as a sequence error rather than processed, and a SEND arriving against an empty RQ (nothing pre-posted to consume) gets an RNR NAK carrying a retry-delay hint instead of silently stalling. Both are named here rather than walked in full: the retry, timeout, and RNR-backoff machinery behind them belongs to Reliability Engine, not this catalog.

Block by block

The same SEND across both hosts. The blocks to watch are the ones a WRITE never lights on RNIC B: the Receive Queue fetch before placement, and the completion written for the responder’s software after it. The last scene adds immediate data, which changes what the completion carries and nothing else.

All 13 steps

1. Before anything is sent (Responder). Host B’s application has already posted a Receive WQE naming a buffer. A SEND carries no address, so that WQE is the only thing that will say where the bytes go. The Receive Queue Descriptor and Ring
2. On Host A (Requester). The application writes a SEND WQE into the Send Queue ring and rings the doorbell: one MMIO write across PCIe. The Send Queue Ring
3. RNIC A reads the WQE out of host memory and looks up the QP’s context. WQE Prefetch
4. The Key Cache checks the lkey on the source buffer, then the DMA engine reads the 4 KiB payload out of Host A’s memory. The Gather Path
5. TX Packet Build puts a BTH in front of the payload and nothing else: no RETH, because there is no remote address to state. A Replay Buffer entry is recorded and the packet leaves. TX Packet Build
6. On Host B (Responder). RX Parse checks the PSN. There is no rkey in the packet to check: whatever permission this SEND needs comes from Host B’s own Receive WQE. PSN Check and Out-of-Order Arrival
7. In the Receive WQE at the head of the Receive Queue, taken strictly in order. RNIC B reads it across PCIe, unless it already holds it on-chip. An empty queue here would draw an RNR NAK instead. The Receive Queue Descriptor and Ring
8. The Key Cache checks the lkey Host B’s software put in that WQE, then the DMA engine writes the payload into the buffer it named. The Scatter Path
9. CQE Generation writes a receive completion into Host B’s Completion Queue, and at the same moment the ACK for PSN 1000 leaves through RNIC B’s TX path. A SEND always tells the responder’s software; the ACK does not wait for that. CQE Generation
10. Only if software armed the CQ: A completion event tells Host B’s application to look. Event and Multi-CQ Arbitration
11. The ACK returns (Requester). RX Parse on RNIC A matches the ACK against the Replay Buffer, retires the entry and disarms the retry timer. Retry and Timeout State
12. Only if the Work Request was signaled: RNIC A writes the send completion into Host A’s Completion Queue. CQE Generation
13. With Immediate (Responder). None. The path is the same; only the receive CQE differs. RNIC B copies the ImmDt value into it, so Host B’s software reads the 32 bits from the completion without looking in the buffer. The CQE

SEND with Invalidate

A SEND with Invalidate carries one extra header, the IETH, holding an rkey that belongs to the responder. When the message completes, the responder invalidates that key, and its receive completion names the key it revoked. The usual case: the responder lent the requester a buffer under that rkey, the requester has finished with it, and the same SEND that says so also takes the access away. Without it, the responder’s software would have to post its own local invalidate before reusing the buffer.

Not every key can be revoked this way. Linux’s soft-RoCE driver accepts a memory window, or a memory region allocated for fast registration, and refuses an ordinary registration or one with windows still bound to it. What the revocation does to the Key Table and to an operation already in flight against that key is Memory Region / Key Table’s subject.

All 10 steps

1. The responder exposed a buffer under rkey 0x2A04, registered so it can be revoked remotely, and the requester has been using it. Now the requester is done, and its application posts a SEND with Invalidate that names that key. Registering a Memory Region
2. The payload is gathered under the WQE’s lkey. The packet gets an IETH: four bytes, the rkey to revoke. TX Packet Build
3. PSN 4000 is the one expected. Like any SEND, the packet carries no address, so the responder takes the Receive WQE at the head of its queue. The Receive Queue Descriptor and Ring
4. The payload lands in the buffer the Receive WQE named, not in the buffer 0x2A04 covers. The Scatter Path
5. Invalidates it, once the message is complete. That is why only LAST and ONLY packets can carry an IETH. From here on, no QP can use the key. Mid-Transaction Deregistration
6. The receive completion says an invalidation happened and names the key, so the responder’s software learns the buffer is no longer exposed. The ACK is already on its way back. The CQE
7. The ACK retires PSN 4000 at the requester. Retry and Timeout State
8. Only if the Work Request was signaled: The send completion posts. CQE Generation
9. It leaves as usual: nothing at the requester knows the key is gone. Permission and Bounds Violations
10. The rkey check fails and nothing is written. The NAK carries a remote access error, and both QPs move to ERROR, as for any bad rkey. Permission and Bounds Violations

All 11 steps

1. Before the SEND (Responder). Host B lent Host A a buffer under rkey 0x2A04, a key registered so it can be revoked this way, and Host A has been writing into it. Host B has also posted a Receive WQE for the message that will say Host A is done. Registering a Memory Region
2. On Host A (Requester). The application writes a SEND with Invalidate WQE naming rkey 0x2A04, and rings the doorbell. The WQE and the Send Queue Descriptor
3. RNIC A reads the WQE, checks the lkey on the source buffer, and reads the payload. The Gather Path
4. The packet is a BTH, an IETH holding the rkey to revoke, and the payload. Like any SEND, it names no address. TX Packet Build
5. On Host B (Responder). RX Parse checks the PSN, and RNIC B reads the Receive WQE at the head of the Receive Queue, as for any SEND. The Receive Queue Descriptor and Ring
6. The payload goes into the buffer that WQE named. The lent buffer under 0x2A04 is not where it lands. The Scatter Path
7. RNIC B invalidates it. From here on no QP can use that key: any lookup fails the way an unknown key would. The memory itself is untouched; Host B can register it again under a new key. Mid-Transaction Deregistration
8. The receive completion goes to Host B’s Completion Queue, flagged as carrying an invalidation and naming 0x2A04, so Host B’s software knows the key is gone without asking. The ACK leaves at the same moment. The CQE
9. The ACK returns (Requester). RNIC A retires the Replay Buffer entry and, if the Work Request was signaled, writes the send completion. Retry and Timeout State
10. A stale WRITE (Requester). Suppose Host A still had a WRITE queued that uses rkey 0x2A04. It leaves as usual; nothing on Host A knows the key is gone. TX Packet Build
11. Refused (Responder). The Key Cache lookup fails, nothing is written, and RNIC B answers with a remote access error NAK. As with any bad rkey, both QPs then move to ERROR. The revocation took effect without Host B’s software doing anything. Permission and Bounds Violations

References

• InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications).
• Linux kernel soft-RoCE driver, drivers/infiniband/sw/rxe/ – rxe_resp.c runs the invalidation after the payload is placed and before the completion, and reports the revoked key in the receive completion; rxe_mr.c (rxe_invalidate_mr) refuses a key that isn’t a fast-registration MR or that still has memory windows bound.