Replay Buffer Capacity and Backpressure

Overview

A QP Cache entry evicted under pressure costs a future re-fetch from the QP Table, nothing more – the evicted context was still valid, still correct, just no longer resident. A Key Cache entry evicted under pressure costs the same kind of re-fetch. Neither eviction loses anything a workload actually needed; both just move a cost from now to later. A Replay Buffer entry is not like either of them. This resource cannot answer a capacity question with a victim-selection policy the way the other two do.

Locator

Eviction Policy and Key Cache Eviction Policy both answered the same question for two other cached resources: which entry gets sacrificed when capacity runs out. This chapter is where that question breaks, and what replaces it.

Block diagram

All 8 steps

1. Eight WQEs are ready in the Send Queue. This QP’s Replay Buffer allocation is 4 entries, and its responder is a long round trip away. The gate below is this book’s reasoning about where the check has to sit, not a documented design. Replay Buffer Capacity and Backpressure
2. The gate admits WQE 1: it allocates an entry to resend from, and the packet goes out in the same step. Replay Buffer Capacity and Backpressure
3. WQEs 2 to 4 follow the same way. No ACK has come back yet: the round trip is longer than it takes to send four. Replay Buffer Capacity and Backpressure
4. No. All four entries hold unacknowledged packets, and WQE 5 would have nothing to be resent from. The gate holds it. This is a stall, not a fault. Replay Buffer Capacity and Backpressure
5. No. Entry 1 is the only way to resend WQE 1 if it was lost. Unlike a cache line, it has no backing copy to fetch again, so evicting it would break the reliability contract rather than cost latency. Replay Buffer Capacity and Backpressure
6. One cumulative ACK covers WQEs 1 and 2. Their entries are retired. Replay Buffer Capacity and Backpressure
7. The gate opens for WQEs 5 and 6, which take the freed entries, then closes again. Replay Buffer Capacity and Backpressure
8. From here, this QP sends only as fast as ACKs free entries: at most 4 entries’ worth of packets per round trip. A longer round trip or a smaller allocation lowers that ceiling, however fast the link is. Replay Buffer Capacity and Backpressure

Deep walkthrough

Why eviction is off the table here, restated precisely. Eviction Policy exists because a QP Cache entry has a backing copy in the QP Table that eviction merely stops keeping resident – the data isn’t lost, only relocated conceptually to a slower structure. A Replay Buffer entry has no equivalent backing copy of its own retransmission-readiness: The Replay Buffer already established that what it holds is a reference back to the original source buffer plus header state, not a duplicate that could be regenerated from some other authoritative copy if discarded. Discarding an unacknowledged entry doesn’t create a cache miss recoverable by re-fetching; it creates a hole in what a requester can prove it sent, with no path back to recovering it if the peer never received it at all. LRU, priority-weighted, and error-weighted eviction all presuppose that sacrificing the chosen victim is safe. None of them are safe here.

What governs capacity instead: never let the problem occur. If eviction under pressure isn’t survivable, the only remaining answer is preventing pressure from exceeding capacity in the first place – admission control rather than victim selection. A Queue Pair has to be prevented from having more unacknowledged, outstanding requests than its Replay Buffer allocation can actually hold, which means the constraint has to be enforced before a new request is transmitted, not discovered after the fact when there’s nowhere left to put its replay state. This is a fundamentally different kind of gate than anything else in this book’s caching chapters: not “evict something to make room,” but “don’t create the condition that would require it.”

Where this gate actually sits, reasoned rather than sourced. The Replay Buffer already named that outstanding-request count is naturally bounded by round-trip time and a Send Queue’s own depth, but a Send Queue provisioned deeper than a specific QP’s Replay Buffer allocation can afford would let software post more work than hardware can safely track for retransmission. The architecturally sound answer is a credit or windowing mechanism gating transmission, not posting – a QP can have work queued and ready in its Send Queue while still being held back from actually going out, exactly the way Single-QP Pipeline Stalls already covers stall conditions gating one QP’s send path without touching any other QP. This chapter adds a new stall cause to that list: a QP that has reached its Replay Buffer allocation’s limit stalls at the transmit gate until an acknowledgment retires an entry and frees room for the next one. This is a different resource than QP Context’s outstanding-read/atomic ceilings, though the shape of the gate rhymes: that pair bounds a count of outstanding read/atomic operations specifically, sized independently of packet volume or path latency, while this gate bounds outstanding bytes/packets across every opcode type, sized by round-trip time per Replay Buffer Caching. A QP could exhaust one without touching the other – ordinary WRITE traffic alone can fill a Replay Buffer allocation without a single read or atomic in flight. No public source confirms this is the exact mechanism a real implementation uses – this book states it as the shape that’s structurally required, not a documented fact.

What this means for a workload that keeps hitting the limit. A QP whose replay-buffer allocation is consistently the binding constraint isn’t failing – it’s a QP whose peer’s acknowledgments aren’t returning fast enough to keep pace with how much new work the application wants outstanding at once, the same round-trip-time-driven pressure Replay Buffer Caching already named as this resource’s defining variable. The gate isn’t a fault condition; it’s ordinary flow control doing exactly what it’s supposed to do, holding a fast sender back from outrunning what the reliability contract can actually promise to recover.

What runs this check: a question this chapter can answer without a separate one. Whether the admission check above is a simple fixed counter comparison or something a firmware-adjustable threshold tunes is architecturally a much smaller question than the eviction-engine question Hardware-Assisted Eviction Engines and its Hardware-Assisted TX Scheduling Engines and Hardware-Assisted RX Dispatch Engines analogs asked of a real victim-selection policy. A counter-against-a-limit check has none of an eviction engine’s complexity – no candidates to rank, no policy to run, just a comparison gating a transmit decision. The likely shape is closer to fixed logic than a programmable engine, precisely because there’s no meaningful policy space to make programmable: the only real design choice is what the limit itself is set to, a provisioning decision, not a runtime policy one.

What happens next. This chapter closed the last open capacity question this part’s two resources raise under ordinary operation. What happens when a QP carrying armed timers and outstanding Replay Buffer entries resets mid-transaction – the one scenario this chapter and every other in this part have deliberately deferred – is Mid-Transaction Reliability Reset’s subject, closing this part.

References

• No IETF or IBTA standard governs Replay Buffer provisioning or the specific mechanism gating transmission against it – it is implementation-specific silicon design, not a wire-format or software-interface contract any standards body defines. This chapter’s treatment is reasoned architectural inference, stated as such, not a citation.