The Retry and RNR Timeout State Machines

Overview

Two different conditions can leave a requester without an acknowledgment for something it sent: the packet or its acknowledgment was lost somewhere between the two endpoints, or the packet arrived intact but the responder had nowhere to put it and rejected it instead of processing it. Reliable Connection does not fold these into one generic “didn’t get acknowledged” state – they run as two structurally separate state machines, each armed by a different trigger, each counted against its own independent budget, converging only at the very end on the same terminal outcome: a Queue Pair driven to ERROR once its budget for one of them runs out.

Locator

Retry and Timeout State named the live counters this chapter’s state machines actually drive. This chapter is the machinery that reads them.

Block diagram

All 11 steps

1. Loss. A requester sends PSN 1000 and arms its retry timer for the ACK timeout. This QP was brought up with a retry count of 2, small so the walk runs out quickly. Retry and Timeout State
2. Nothing comes back: no ACK and no NAK. The timer expires. The Retry and RNR Timeout State Machines
3. Go-back-N from the oldest unacknowledged PSN, and one retry spent. The timer is armed again for the resent packets. Go-Back-N Retransmission
4. Round the loop again, discovered a different way: a PSN Sequence Error NAK arrives instead of silence. It draws on the same counter, because it is the same condition from the requester’s side. Another go-back-N, and the count reaches zero. The Retry and RNR Timeout State Machines
5. No. The count is exhausted, the WQE completes with a transport retry counter exceeded error, and the QP goes to ERROR. Its RNR retry count is still 3: loss never touched it. Single-QP Error Handling
6. Receiver not ready. A different QP, with the same budgets. Its SEND reaches a responder whose Receive Queue is empty, and comes back as an RNR NAK carrying a delay. That arms the RNR timer, not the ordinary one. The Retry and RNR Timeout State Machines
7. The delay runs out. It is never shorter than the RNR timer floor the responder is allowed to advertise. The Retry and RNR Timeout State Machines
8. The same PSN is sent again and one RNR retry is spent. The ordinary retry count stays at 2: an empty receive queue costs nothing from the budget kept for loss. The Retry and RNR Timeout State Machines
9. The Receive Queue stays empty: Each further RNR NAK spends one more. When the RNR count reaches zero the WQE completes with an RNR retry counter exceeded error and the QP goes to the same ERROR, with its ordinary count still at 2. Single-QP Error Handling
10. Not retryable. A third QP’s WRITE comes back with a NAK for a Remote Access Error. Only sequence-error and RNR NAKs are retried; this one is reported to the client on its first occurrence. Permission and Bounds Violations
11. Neither. No waiting and no second attempt: the QP goes to ERROR immediately with both budgets untouched. Single-QP Error Handling

Deep walkthrough

The ordinary branch: armed, expired-or-rejected, retried, or disarmed. A packet leaving the wire arms the timer Retry and Timeout State already introduced, set to fire after the configured ACK timeout duration if nothing acknowledges it first. An acknowledgment arriving before expiry disarms the timer and, if it’s the last outstanding request, leaves nothing armed at all; an acknowledgment arriving for a later request while this one is still the oldest outstanding re-arms against whatever is now oldest, per The Replay Buffer’s cumulative-retirement logic. Two distinct events drain the same counter here, not one: silent expiry with nothing having arrived at all, and an explicit PSN Sequence Error NAK arriving instead of an ACK – IBTA-V1 groups both under one shared retry counter (a lost packet and a NAK that says “I saw a gap” are the same underlying condition from the requester’s side, just discovered two different ways). Either one triggers Go-Back-N Retransmission, covered next in full, and decrements the same live retry count. This loop continues for as long as the live retry count stays above zero.

The RNR branch: a different trigger, a different remedy, a separate budget. A Receiver-Not-Ready condition isn’t silence – it’s an explicit signal, a NAK carrying a backoff delay the responder itself is advertising, sent because the responder lacked a resource the arriving message needed to process it: an empty receive-side buffer for an incoming SEND, most concretely. This is not a lost-packet condition at all: the responder saw the packet, correctly identified that it had nowhere to place it, and said so. The requester’s response is correspondingly different from the ordinary branch’s go-back-N: rather than resending a range of packets, it waits out the advertised delay – never shorter than the RNR-NAK timer floor the responder is permitted to advertise – and then resends the same single packet that triggered the rejection, no more. The RNR-retry count decrements on each such attempt, counted entirely independently of the ordinary retry count next to it.

Play the RNR branch with a single SEND against an empty Receive Queue:

All 7 steps

1. A single-packet SEND, PSN 1000. A SEND names no remote address: it depends on the responder having posted a Receive WQE. SEND
2. The PSN is right, but the responder’s Receive Queue is empty. The packet arrived intact and has nowhere to go. The Receive Queue Descriptor and Ring
3. No, and the responder says so outright: an RNR NAK, whose AETH carries how long to wait before trying again. The Retry and RNR Timeout State Machines
4. The requester waits out the delay. This draws on the RNR retry count, a budget separate from the one loss uses, so an empty receive queue never eats into the tolerance for loss. Retry and Timeout State
5. Meanwhile the responder’s application posts a Receive WQE. Nothing tells the requester; it tries again when the delay is up. The Receive Queue Descriptor and Ring
6. The same packet arrives again. This time a WQE sits at the head of the RQ: it is consumed, the payload placed, and a receive completion written. CQE Generation
7. The ACK retires the send. Had the RQ stayed empty until the RNR retry count ran out, the QP would have moved to ERROR. The Retry and RNR Timeout State Machines

Why one shared counter would be the wrong design. A QP whose peer’s receive queue runs empty a few times under a bursty workload, and separately loses a few packets on a lossy path, is facing two unrelated problems. Charging both against one shared budget would let either condition’s bad luck exhaust the tolerance for the other – a QP that happened to hit several RNR conditions early in a connection’s life would have less headroom left to tolerate ordinary loss later, for no reason connected to the path’s actual loss characteristics. Two independent counters mean a QP’s tolerance for one failure mode is never a hidden function of how much of the other failure mode it happened to experience first.

The one door out, shared by both branches. Both branches feed the same single exit: per QP State Machine, the QP is driven to ERROR once its counter runs out. What ERROR actually does to a QP’s other outstanding work from that point is Single-QP Error Handling’s subject, not re-derived here – this chapter’s job stops at naming that exhaustion is the trigger, not at re-explaining what firing that trigger does.

Most NAK codes never reach this state machine at all. IBTA-V1 draws a hard line between two families of remotely-detected fault, and this chapter’s two branches only cover one of them: “Of the possible NAK codes, two (NAK-Sequence Error and NAK-RNR) indicate operations that should be retried automatically by the requester. The NAK codes other than NAK-Sequence Error and NAK-RNR indicate failures that must be reported to the requester’s client immediately and cannot be retried.” A Remote Access Error, a Remote Operational Error, or an Invalid Request (an unsupported opcode, a missing FIRST/LAST in a multi-packet message, a length error, or – per the Gotcha above – exceeding the negotiated outstanding RDMA Read/Atomic ceiling) never touches either retry counter above – there’s no waiting, no decrementing, no second attempt. The requester reports the failure to its client on the very first occurrence and the QP is driven to error immediately. Single-QP Error Handling already covers this second family under its own “access-violation error” and “invalid-request error” names; this chapter’s retry/RNR machinery is not the mechanism behind either of them.

What happens next. This chapter named what triggers a retry and what happens when retries run out. It deliberately didn’t cover what retransmission actually does to the packets themselves – Go-Back-N Retransmission is that chapter, immediately next. Many QPs timing out at the same moment from one shared cause is a separate problem, taken up in Multi-QP Retry Arbitration.

References

• InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications) – defines the RNR NAK mechanism, the minimum RNR timer floor, and the independent retry-count semantics this chapter describes. §9.9.2.2 (Requester Side Error Detection – Remotely Detected Errors) is the direct source for the retryable-vs-immediately-reported split this chapter’s Gotcha above draws; Table 56/57’s Requester Fault Behavior Classes are the source for which specific NAK codes fall on each side of that split. Table 58 (Responder Error Behavior Summary) is the direct source classifying “too many outstanding RDMA READ or ATOMIC Requests” as Invalid Request rather than RNR NAK – the correction the ordinary/RNR-branch Gotcha above makes.