Go-Back-N Retransmission

Overview

RX Packet Parse and PSN Check and Out-of-Order Arrival already covered the responder’s half of this exchange: a gap in arriving sequence numbers is rejected outright, not buffered, with a signal naming the sequence number the responder is actually waiting for. This chapter picks up from the requester’s side of that same event – what happens once a NAK naming a resume point arrives, or once a timer expires with nothing having arrived at all.

Locator

The Retry and RNR Timeout State Machines named the two conditions that trigger a retry. This chapter covers what a retry – specifically, the ordinary timeout branch’s retry – actually resends.

Block diagram

Deep walkthrough

What “go back N” actually means. When the ordinary timeout branch fires, or when a NAK arrives naming a specific sequence number as the point the responder is waiting from, the requester’s remedy is the same either way: resend every outstanding packet from that point forward, in order, not just the one packet that’s suspected lost. If a NAK names PSN N+2 as the resume point, PSNs N+2 through whatever was most recently sent all get rebuilt and retransmitted – including any of them that, unknown to the requester, actually arrived correctly the first time and are only being resent because something after them didn’t.

Why hardware doesn’t need to know exactly what was lost. PSN Check and Out-of-Order Arrival already established that Reliable Connection’s responder rejects an out-of-order arrival outright rather than buffering it – which means the requester never receives a fine-grained picture of exactly which packets among a range did and didn’t make it, only a single resume point. Go-back-N is the retransmission strategy that fits the information actually available: it doesn’t ask the requester to reconstruct a precise loss pattern it was never told, only to trust the resume point named and resend everything after it.

What gets rebuilt, not what gets re-decided. Retransmission is not a new decision about what to send – every field a resent packet carries was already determined the first time it was sent, and The Replay Buffer held exactly what’s needed to reproduce it: the header fields, and a reference back to the original Work Queue Element’s source data. Rebuilding a retransmitted packet reuses TX Packet Build’s own mechanism – the same header-assembly stage, working from held state instead of a freshly-fetched WQE, not a separate retransmission-specific datapath duplicating what that stage already does.

Here is the NAK-triggered case end to end: one packet lost in the middle of a four-packet WRITE, and everything after it sent twice.

All 11 steps

1. A 16 KiB WRITE leaves as PSN 1000 to 1003. All four sit in the Replay Buffer, and the retry timer watches the oldest, 1000. The Replay Buffer
2. 1000 is the PSN the responder expects. It is accepted, and the expected PSN moves to 1001. PSN Check and Out-of-Order Arrival
3. PSN 1001 is dropped somewhere in the fabric. Nobody is told; it simply never arrives. Go-Back-N Retransmission
4. No. A PSN higher than expected means a gap, and RC rejects the packet rather than buffering it. Keeping out-of-order packets would need a reorder buffer for every QP. PSN Check and Out-of-Order Arrival
5. The responder NAKs, naming 1001, the PSN it is waiting for. That one number is all the requester will learn about the loss. It crosses PSN 1003 on the way. PSN Check and Out-of-Order Arrival
6. 1003 was already on its way. It is out of order too, and is dropped. PSN Check and Out-of-Order Arrival
7. The NAK reaches the requester. A sequence-error NAK draws on the same retry count a timeout does. Everything before 1001 counts as received. The Retry and RNR Timeout State Machines
8. Every one from 1001 on. It knows only the resume point, not what made it across, so it rebuilds 1001, 1002 and 1003 from the Replay Buffer through the ordinary TX Packet Build path. Go-Back-N Retransmission
9. The three resends leave back to back and are in flight together. They are the same packets: same PSNs, same headers, same source bytes. TX Packet Build
10. This time they arrive in order. Each is placed through the reassembly context, which had stayed where 1000 left it. Multi-Packet Reassembly
11. One ACK for 1003 retires what was left and disarms the timer. 1002 and 1003 crossed the fabric twice because the responder would not keep them the first time. Go-Back-N Retransmission

The pure-timeout case, without a NAK. Not every retry is triggered by an explicit rejection. A timer can expire with nothing having arrived at all – no acknowledgment, and no NAK either, because the packet or every subsequent acknowledgment was lost outright rather than rejected. The requester’s own resume point in this case is whatever Retry and Timeout State’s armed timer was tracking as the oldest unacknowledged PSN – the requester doesn’t need the responder to tell it where to resume, because it already knows the oldest thing it’s still waiting to hear about. The retransmission itself proceeds identically either way, from the oldest unconfirmed PSN forward, whether that resume point came from an explicit NAK or from the requester’s own bookkeeping.

The timeout case at its smallest: the data lands, the ACK is lost, and the responder’s duplicate check makes the resend harmless.

All 7 steps

1. A single-packet WRITE, PSN 1000. Its Replay Buffer entry is written and the retry timer arms. Retry and Timeout State
2. It arrives in order and the payload is placed. The responder now expects 1001. PSN Check and Out-of-Order Arrival
3. The ACK is dropped on the way back. The data is in place, and the requester has no way to know it. Retry and Timeout State
4. The ACK timeout expires. Nothing came back at all, so the requester resumes from the oldest PSN it is still timing: 1000. Go-Back-N Retransmission
5. The retry count drops by one, and PSN 1000 is rebuilt from the Replay Buffer. The Retry and RNR Timeout State Machines
6. 1000 is lower than expected, so it is a duplicate. The payload is not placed again; the ACK is sent again, so the requester stops retrying. PSN Check and Out-of-Order Arrival
7. This ACK gets through. The entry retires and the timer disarms. Had the retry count reached zero first, the QP would have moved to ERROR. The Retry and RNR Timeout State Machines

What this model covers, and the two cases it doesn’t. Everything above describes a requester rebuilding its own outstanding sent requests – SEND, WRITE, and the READ_REQUEST/Atomic request packets themselves, all of which the requester’s own Replay Buffer genuinely holds. It does not cover a lost RDMA_READ_RESPONSE packet: that data is generated by the responder, re-reading its own memory, and the requester’s Replay Buffer never held a copy of it to rebuild from – PSN Check and Out-of-Order Arrival already named this as an open exception this book doesn’t resolve, and this chapter’s requester-side model doesn’t resolve it either. Nor does it address what a responder does if a duplicate READ or Atomic request arrives after the original already executed: re-running a WRITE is harmless, but re-running COMPARE_SWAP/FETCH_ADD a second time is not, since both are non-idempotent reads-modify-writes – a real implementation needs some form of duplicate-request suppression at the responder for these two families specifically, and no chapter in this book states what that mechanism is.

What happens next. A single QP’s retransmission is a self-contained event once it’s triggered. What happens when several QPs need retransmission service from the same shared TX pipeline at the same time – whether a retransmission gets any different priority than fresh traffic competing for the same resource – is Multi-QP Retry Arbitration’s subject next.

References

• InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications) – defines the cumulative-acknowledgment and sequence-error NAK semantics this chapter’s go-back-N behavior follows from.