Response Generation

Overview

A responder’s TX side builds packets no Work Request asked for. Every request that arrives on a Reliable Connection eventually earns an answer: a positive acknowledgment, a negative one, an RDMA READ’s data, or an ATOMIC’s original value. Response Generation is the logic that decides which of those is owed, which PSN and Message Sequence Number it carries, and whether it has to leave now or may wait and cover several requests at once. It consumes RX PSN Check’s verdict on each request and hands a fully specified response to TX Packet Build, which builds it on the same wire as the QP’s own requests.

Locator

TX Packet Build named the gap this chapter closes: an ACK or a READ response is built by the same stage as any request, but nothing this part had covered decides when one is due or what goes in its AETH. The “response due” arrow is that decision.

Block diagram

Deep walkthrough

What a response is. IBTA-V1 §9.7.5.1.7 allows exactly three forms: an acknowledge packet for a SEND or RDMA WRITE, an RDMA READ response, and an ATOMIC acknowledgment. All three carry the same four pieces of information: a syndrome saying success or failure, a PSN the requester uses to find the request it answers, a Message Sequence Number, and optionally end-to-end credits. The READ response and the ATOMIC acknowledgment add a payload; the plain acknowledgment never does. A NAK is not a fourth form. It is an acknowledge packet whose AETH syndrome carries a NAK code instead of an ACK (C9-113.a1), even when it replaces a READ response packet.

The AETH, field by field. Four bytes: an 8-bit syndrome and a 24-bit MSN (§9.7.5.2). The syndrome’s top bit is reserved and zero. Bits 6:5 select the kind: 00 ACK, 01 RNR NAK, 11 NAK, 10 reserved. Bits 4:0 then mean something different for each kind:

The MSN is valid in every one of them, NAKs included (C9-149). Message Completion and the MSN covers what it counts and how the requester uses it; this chapter only needs the generation rule: one increment per completed new request message, never for a duplicate.

Which PSN goes in the BTH. The rule depends on what is being answered, and getting it wrong breaks the requester’s matching:

• An ACK for SENDs and WRITEs carries the PSN of the most recent new request it covers (C9-95). Because of coalescing, consecutive ACKs need not carry consecutive PSNs.
• A READ response’s packets carry the READ request’s own PSN, then count up one per packet (C9-96). The requester reserved those PSNs when it sent the request.
• An ATOMIC acknowledgment carries the request’s PSN (o9-58); an ATOMIC is a single packet.
• A NAK for a PSN sequence error or an invalid request carries the responder’s expected PSN (C9-111, C9-121, C9-125), so the requester knows where to resume. A remote access error carries the PSN of the offending request (C9-124), and an RNR NAK the PSN of the request it refuses (C9-113).

When a response is due: AckReq and coalescing. For SENDs and WRITEs the spec does not schedule responses at all. It lets one acknowledgment cover several requests (§9.7.5.1.2) and leaves the timing to the responder, bounded by one rule: a valid request with the BTH’s AckReq bit set must have a response scheduled, carrying that request’s PSN or a later one (C9-116). Without AckReq the responder “may even wait an unbounded time for additional requests.” That is why the spec advises a requester to set AckReq on the last packet of whatever it has posted, or risk waiting on an acknowledgment that never comes. The Linux soft-RoCE driver shows both halves of this contract. Its requester sets AckReq on the last packet of every message and on every 64th packet in between (rxe_req.c, RXE_MAX_PKT_PER_ACK). Its responder answers a SEND or WRITE only when AckReq is set (acknowledge() in rxe_resp.c), so its coalescing comes entirely from multi-packet messages.

A READ response acknowledges more than the READ. The first packet of a READ response (or its only packet) must carry an AETH, and so must the last (C9-97, C9-98). The two do different jobs. The first one implicitly acknowledges every earlier SEND or WRITE still outstanding at the requester, as a coalesced ACK would. The last one acknowledges the READ itself. Middle packets carry no AETH. When the MSN goes up for the READ is the responder’s choice: right after validating it, so the first packet already carries the new value, or once the data has been sent, so only the last does (C9-148). The walk below takes the second option.

Ordering is the subtle part. Because READs are only loosely ordered against later requests, a responder may start executing SENDs and WRITEs that arrived after a READ. But it may not acknowledge them until the READ’s response has gone out (C9-101). The reason is on the requester’s side: an acknowledgment carrying a PSN later than an outstanding READ implies a NAK for that READ (§9.7.5.1.2, rule 1). An ACK that overtook the READ response would tell the requester to resend a READ that was succeeding.

Duplicates are answered, not re-executed. RX PSN Check classifies a request with a PSN behind the expected one as a duplicate (PSN Check and Out-of-Order Arrival). Response Generation then owes it a response, and the expected PSN does not move while it is produced (C9-103):

• A duplicate SEND or WRITE is not executed again. It gets an ACK carrying the PSN of the most recently completed new request (C9-105), which tells the requester the responder has already seen everything up to there.
• A duplicate READ is executed again: the data is read from memory a second time, and the response restarts at the duplicate’s PSN (C9-107). That only works if the responder still holds the original request’s PSN range, which is what the READ resources in the block diagram are for.
• A duplicate ATOMIC is never executed again. If its PSN matches one the responder saved, the saved original value goes back (o9-66). If not, the request is dropped (o9-67). Atomics walks this case.

Duplicates are answered in PSN order, oldest first (C9-104, C9-110), and an error while answering one is never reported with a NAK; the duplicate is dropped instead (C9-106, C9-115). A NAK about an old request could be mistaken for a NAK about a new one.

NAKs, and the latch after a sequence error. A PSN ahead of the expected one means something was lost. The responder sends one NAK with code 0 and its expected PSN, consumes no Receive WQE, and then ignores every new request until one arrives with exactly that PSN (C9-114, C9-122). It sends no further NAKs in the meantime, while still answering duplicates. The “NAK latch” in the block diagram is that rule as one bit of state. It is set by the NAK and cleared by the matching request. Without it, every packet still in flight behind the lost one would trigger its own NAK, and the requester would rewind once per NAK. Go-Back-N Retransmission covers what the requester does with the one NAK it does get. The other NAK codes follow request errors that RX checks detect: Permission and Bounds Violations covers the remote access error, and an RNR NAK follows a SEND that found no Receive WQE (The Retry and RNR Timeout State Machines).

A READ that fails partway through has its own rule. If the responder hits an error after some response packets have gone out, it sends none of the bad data. The packet that would have carried it goes out as an Acknowledge opcode with the NAK code instead of a READ response, and its PSN names the response packet being refused (C9-100, C9-112).

Credits: the Receive Queue’s say in the AETH. When the syndrome is an ACK, bits 4:0 carry encoded end-to-end credits: how many more request messages the Receive Queue can take, counted in posted Receive WQEs (§9.7.7.2). An HCA must generate them for any QP without a shared receive queue (C9-150.2.1). A QP on an SRQ cannot know how many WQEs its share of the pool holds, so it must send 11111, “not valid”, instead (o9-95.2.2). The responder can also send credits without anything to acknowledge, by re-sending its most recent positive ACK with an updated credit field. This is called an unsolicited acknowledgment. It can never be made from a NAK, since a duplicate NAK is forbidden. So the Receive Queue feeds this stage, which is the credit-count input in the block diagram.

Play the whole sequence, from two unacknowledged requests through a READ, a lost packet and the resend:

All 17 steps

1. SEND, no AckReq (Requester). A SEND arrives with PSN 300 and the AckReq bit clear. The requester chose not to ask for a response yet. Response Generation
2. Nothing owed yet (Responder). The SEND is valid and complete, so the MSN goes to 1 and the expected PSN to 301. With AckReq clear the responder may hold its answer for as long as it likes, so it sends nothing. Message Completion and the MSN
3. WRITE, no AckReq (Requester). A WRITE follows with PSN 301, again without AckReq.
4. Still nothing owed (Responder). Placed and complete: MSN 2, expected PSN 302. Two requests are now done and neither has been answered.
5. The READ (Requester). A READ for 8 KiB at a 4 KiB path MTU. Its response will need two packets, so the requester has set aside PSNs 302 and 303 for them. RDMA READ
6. READ responses (Responder). Yes. AckReq only matters for SENDs and WRITEs; a READ or ATOMIC always gets an explicit response. The responder validates the RETH, takes an outstanding-read resource that remembers PSNs 302 and 303, and starts the DMA read. The expected PSN jumps past both. The Gather Path
7. The first READ response packet. It must carry an AETH, and that AETH implicitly acknowledges every SEND or WRITE with an earlier PSN. MSN 2 tells the requester two messages are complete. This responder bumps the MSN for the READ when it finishes; bumping it at the start is equally allowed. Message Completion and the MSN
8. The last packet carries the second AETH, which acknowledges only the READ itself. MSN 3. The resource is kept, so a duplicate READ could be answered again.
9. A lost packet (Requester). The SEND with PSN 304 is lost on the wire. The next one, PSN 305, arrives with AckReq set. Go-Back-N Retransmission
10. Sequence error (Responder). 305 is ahead of the expected 304: a PSN sequence error. The packet is not executed, no Receive WQE is used, and the MSN stays at 3. PSN Check and Out-of-Order Arrival
11. 304, the responder’s expected PSN, so the requester knows where to resume. A NAK is a packet with the Acknowledge opcode and a NAK code in the AETH syndrome. Its MSN is still valid: 3. Go-Back-N Retransmission
12. Both sides act on it at once. The requester backs up to 304. The responder’s latch makes it ignore every new request until 304 arrives, and it sends no second NAK in the meantime.
13. The resend (Requester). The resent 304 arrives.
14. Back in sequence (Responder). It matches the expected PSN, so the latch clears and the SEND runs: MSN 4.
15. AckReq again (Requester). 305 follows, with AckReq set as before.
16. The ACK (Responder). MSN 5. AckReq means a response must now be scheduled, with a PSN of at least 305.
17. One ACK carries PSN 305 and MSN 5. On the requester it retires both 304 and 305, because an ACK covers every earlier SEND or WRITE. The Replay Buffer

This chapter closes TX Pipeline. RX Pipeline runs a related set of questions against the receiving side, including the PSN check whose verdicts this chapter turned into responses.

Interface reference

References

• InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1, Release 1.2.1 – §9.7.5.1.1 (response PSNs, C9-95, C9-96, o9-58), §9.7.5.1.2 (coalesced acknowledgments and the requester’s interpretation rules), §9.7.5.1.3 (READ responses, C9-97 to C9-101), §9.7.5.1.4 (duplicates, C9-102 to C9-110, o9-66, o9-67), §9.7.5.1.5 (NAKs, C9-111 to C9-115), §9.7.5.1.6 (AckReq scheduling, C9-116), §9.7.5.1.7 (response formats), §9.7.5.2 (AETH syndrome, Tables 43 and 44, C9-119 to C9-126), §9.7.7.1 (MSN generation, C9-148, C9-149), §9.7.7.2 (end-to-end credits, C9-150.2.1, o9-95.2.2, unsolicited acknowledgments). Read directly at these section numbers.
• InfiniBand Trade Association, Supplement to InfiniBand Architecture Specification Volume 1, Release 1.2.1, Annex A17: RoCEv2 (2014) – carries these transport rules over unchanged.
• Linux kernel soft-RoCE driver, drivers/infiniband/sw/rxe/ – rxe_req.c sets AckReq on the last packet of every message and every RXE_MAX_PKT_PER_ACK (64) packets (rxe_param.h); rxe_resp.c’s acknowledge() answers a SEND or WRITE only when AckReq is set, and every positive AETH it builds uses AETH_ACK_UNLIMITED (0x1f, the “credits not valid” encoding, rxe_hdr.h); read_reply() chooses First, Middle, Last or Only by the remaining length against the path MTU.
• No standard governs how a responder holds its pending-ACK state, how large its READ and ATOMIC resources are, or how long it waits before sending an ACK that no AckReq forced; those are stated here as reasoned inference from the rules above.