PSN Check and Out-of-Order Arrival

Overview

A packet sequence number exists to solve one problem: let a hardware receiver, with no notion of wall-clock time and no reliable way to know what a sender has already retransmitted, decide with certainty whether an arriving packet is the one it expects, one it has already seen, or one that implies something was lost. This is a transaction-tag problem, not a timing problem. A Queue Pair’s receiving side holds exactly one number that matters for this decision: the sequence number it expects next.

Locator

RX Packet Parse confirmed a packet’s framing, identified its destination Queue Pair, and validated its integrity – but deliberately stopped short of deciding whether the packet is actually allowed to proceed. This chapter is that decision.

Block diagram

All 11 steps

1. A 16 KiB WRITE leaves as PSN 1000 to 1003. All four sit in the Replay Buffer, and the retry timer watches the oldest, 1000. The Replay Buffer
2. 1000 is the PSN the responder expects. It is accepted, and the expected PSN moves to 1001. PSN Check and Out-of-Order Arrival
3. PSN 1001 is dropped somewhere in the fabric. Nobody is told; it simply never arrives. Go-Back-N Retransmission
4. No. A PSN higher than expected means a gap, and RC rejects the packet rather than buffering it. Keeping out-of-order packets would need a reorder buffer for every QP. PSN Check and Out-of-Order Arrival
5. The responder NAKs, naming 1001, the PSN it is waiting for. That one number is all the requester will learn about the loss. It crosses PSN 1003 on the way. PSN Check and Out-of-Order Arrival
6. 1003 was already on its way. It is out of order too, and is dropped. PSN Check and Out-of-Order Arrival
7. The NAK reaches the requester. A sequence-error NAK draws on the same retry count a timeout does. Everything before 1001 counts as received. The Retry and RNR Timeout State Machines
8. Every one from 1001 on. It knows only the resume point, not what made it across, so it rebuilds 1001, 1002 and 1003 from the Replay Buffer through the ordinary TX Packet Build path. Go-Back-N Retransmission
9. The three resends leave back to back and are in flight together. They are the same packets: same PSNs, same headers, same source bytes. TX Packet Build
10. This time they arrive in order. Each is placed through the reassembly context, which had stayed where 1000 left it. Multi-Packet Reassembly
11. One ACK for 1003 retires what was left and disarms the timer. 1002 and 1003 crossed the fabric twice because the responder would not keep them the first time. Go-Back-N Retransmission

All 7 steps

1. A single-packet WRITE, PSN 1000. Its Replay Buffer entry is written and the retry timer arms. Retry and Timeout State
2. It arrives in order and the payload is placed. The responder now expects 1001. PSN Check and Out-of-Order Arrival
3. The ACK is dropped on the way back. The data is in place, and the requester has no way to know it. Retry and Timeout State
4. The ACK timeout expires. Nothing came back at all, so the requester resumes from the oldest PSN it is still timing: 1000. Go-Back-N Retransmission
5. The retry count drops by one, and PSN 1000 is rebuilt from the Replay Buffer. The Retry and RNR Timeout State Machines
6. 1000 is lower than expected, so it is a duplicate. The payload is not placed again; the ACK is sent again, so the requester stops retrying. PSN Check and Out-of-Order Arrival
7. This ACK gets through. The entry retires and the timer disarms. Had the retry count reached zero first, the QP would have moved to ERROR. The Retry and RNR Timeout State Machines

Deep walkthrough

The comparison, and the three outcomes it produces. A Reliable Connection Queue Pair tracks the next sequence number it expects to receive as part of its live context – the same field this book has already named without opening. Every packet that survives parsing gets its own sequence number compared against that expectation, and the comparison has exactly three possible outcomes. A match means the packet is exactly the one expected, and processing continues. A sequence number lower than expected means this packet is a duplicate – almost certainly a retransmission of something already received, most likely because the sender’s earlier acknowledgment never arrived back. A sequence number higher than expected means a gap: something between what was last received and what just arrived is missing, whether lost in transit or never sent yet in the sender’s own ordering.

A duplicate does not get reprocessed, and does not touch the Receive Queue – for the families where a bare re-acknowledgment is actually a complete response. The correct response to a lower-than-expected sequence number is not to run the packet through the rest of the pipeline a second time – its effects, whatever they were, already happened once. Re-executing a SEND’s Receive Queue consumption a second time would silently corrupt the ordering The Receive Queue Descriptor and Ring established as this structure’s entire reason for existing; re-executing a memory placement a second time risks writing stale data over whatever has landed there since. For SEND and WRITE, the architecturally correct response is narrower: acknowledge the sequence number again, so the sender’s own retry logic sees confirmation and stops retransmitting, without repeating any of the work the first, correctly-ordered arrival already did.

That narrower response doesn’t cover every opcode family, though. The Operation Catalog already established that RDMA READ has no separate bare-acknowledgment packet at all – its response payload doubles as its own ACK – so a duplicate RDMA_READ_REQUEST can’t be resolved by “acknowledging the sequence number again”; there is no such packet for READ to send. What a real responder does instead – regenerate and resend the actual read-response data, or something else – is not something this book resolves here. The same gap applies to Atomics: ATOMIC_ACKNOWLEDGE carries the pre-operation value the requester still needs, which a bare re-ack can’t supply. Both are named as an open exception to this chapter’s general rule, not folded silently into it.

A gap is not buffered – it is rejected, with enough information to resume. This is the point at which Reliable Connection’s receiving side makes a real architectural choice, not an accident of implementation: a packet arriving out of order is not held in some per-Queue-Pair reorder structure waiting for the missing one to show up later. It is rejected outright, with a signal back to the sender naming the sequence number the receiver is actually waiting for, so the sender knows exactly where to resume rather than guessing. This is a deliberate rejection of the alternative – buffering out-of-order arrivals and resequencing them once the gap fills – and the alternative is not free: it would require a per-Queue-Pair reorder buffer sized to however far ahead a sender might legitimately race, real on-chip storage spent on a condition every well-behaved connection experiences rarely. What actually happens to the sender’s own outstanding, unacknowledged work once this rejection reaches it belongs to the not-yet-written reliability engine, not this chapter – this chapter’s job stops at naming that the receiver rejects rather than buffers, and why that is a load-bearing design choice rather than an oversight.

Once a packet matches, one more gate stands before memory, and this chapter names it without re-deriving it. A packet whose sequence number matches is not yet cleared to touch host memory. Whatever address or key it carries – a remote address and its authorizing key for a one-sided operation, or nothing at all for a two-sided one consuming a Receive Queue entry instead – still has to resolve against the Key Table and pass whatever permission and bounds checking that resolution requires. That check, and everything it can reject, belongs entirely to Permission and Bounds Violations; this chapter does not repeat it.

The handoff: what has to be true before a packet reaches memory, not how it gets placed. A packet that clears the sequence-number gate and clears permission and bounds checking is, at that instant, ready to have its payload placed in host memory. The mechanics of that placement – resolving a virtual address, walking a scatter-gather descriptor chain, the ordering guarantees a real implementation provides – belong entirely to the not-yet-written DMA Engine. This chapter’s job ends at naming the handoff itself: a resolved, authorized operation, carrying a validated address and length, crossing from this part’s territory into that one’s. Nothing about how the DMA engine actually moves those bytes is this chapter’s concern.

What happens next, for the one case this chapter has deliberately deferred. Everything above describes a single packet’s fate. A message spanning more than one packet – opened by a FIRST, continued by however many MIDDLE packets, closed by a LAST – needs state that persists across every packet in that sequence, not just a single comparison against the next one. Multi-Packet Reassembly is that state, immediately next.

Multi-Packet Reassembly is next: the state a message spanning more than one packet needs, and this chapter’s single-packet gate does not provide.

References

• InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications) – defines the packet sequence number field this chapter’s comparison operates on and the duplicate/gap semantics this chapter describes.