QP Caching
Overview
A QP’s live context faces a familiar caching problem: every packet that touches a QP — building one on transmit, parsing one on receive — needs that QP’s live context: its state, its PSN counters, its negotiated path. Fetching that context from the QP Table on every single packet would mean a DRAM or PCIe round trip on the critical path of every packet the adapter handles, which is not compatible with line-rate operation. The QP Cache exists to make the common case — a QP that’s actively carrying traffic — resolve on-chip, and to make the uncommon case — a QP that isn’t — cost a measurable, bounded penalty instead of an unbounded one.
Analogy
A CPU never re-walks a page table on every memory access it can avoid – a TLB caches the translations actually in use, because paying DRAM/page-walk latency on every single access would be unworkable at core clock speeds. The QP Cache is that same idea applied to per-QP context instead of address translations.
Locator
The QP Cache is the on-chip working set of QP contexts sitting in front of the QP Table’s DRAM/HBM-backed storage — this chapter, in the master diagram above.
Block diagram
Deep walkthrough
What the cache holds. A working set of QP Contexts (QPCs) — the same per-QP fields QP Context covers field by field, populated from the QP Table entry the last time that QP was touched. The on-chip SRAM holding this working set is scarce and expensive by construction, the same constraint that governs any on-chip cache – this book doesn’t have a citable source pinning down an exact QPC byte size or exact SRAM budget for any specific chip, and doesn’t state one as fact. What is real and citable: “Communication-Semantic-Aware RDMA Loss Recovery for QP-scalable Hyperscale AI Training” (arXiv 2606.20582) frames the identical problem this chapter opens with – a real RNIC can only keep a bounded working set of QP contexts resident on-chip at once, nowhere near the full range of QPs the adapter can address, which is exactly the gap this chapter’s “hit” and “miss” cases exist to describe.
Where this is checked. Every pipeline stage that needs a QP’s context checks the cache first: the TX Scheduler, which consults it to decide which QP to service next, and the RX Parser, which consults it to resolve the destination QP number it just read out of an incoming BTH (What Is RoCEv2). Neither stage has any path to the wire or to host memory that skips this lookup.
When a lookup misses. A miss means the requested QPC isn’t currently resident in the on-chip working set, and the same paper treats this as a routine condition at scale, not a hypothetical corner case: a shipped RNIC’s cache-bounded QP working set means traffic touching enough QPs beyond that bound pays a PCIe round trip back to host memory to retrieve QP metadata, on tested hardware. That gap between what a real adapter can address and what its cache can actually hold resident is not a defect in one generation of silicon; it’s the direct, structural consequence of on-chip SRAM being small and expensive, against a QP count that scales with however many connections an application layer decides to open. Whether that SRAM is dedicated to QP context alone or shared with other on-chip structures (address translation, per Key Caching, is one resource confirmed to share its own on-chip pool with other state) isn’t something this specific paper documents for QP context – this book doesn’t extend that MR-side finding to the QP cache without its own source.
What happens after a miss. A miss doesn’t drop or corrupt the packet waiting on it — it stalls that packet’s processing behind a fetch, at PCIe latency, against an on-chip hit’s much cheaper cost. Enough concurrently active QPs beyond the cacheable working set, and enough of the traffic touching QPs outside that set, and the adapter spends more time fetching context than building packets — a structural consequence of a bounded working set, not a hypothetical one. What decides which QPC gets evicted to make room for an incoming one, and whether that decision is fixed in hardware or programmable, is Eviction Policy and Hardware-Assisted Eviction Engines’s entire subject; this chapter’s job is establishing that the decision has to be made at all, and that getting it wrong carries a real cost.
The walk below plays both cases on a four-line cache with five busy QPs, including the dirty write-back a miss can force before its fetch, and the cost of evicting a QP that is needed again a moment later.
All 14 steps
- A hit. A SEND arrives for QP 0x12. The cache is full: four lines, four QPs. Three of them are dirty, meaning a packet has moved their PSN counters since they were fetched, so the QP Table’s copy is out of date. QP Caching
- The RX Parser reads DestQP out of the BTH and asks the cache for QP 0x12. Line 0 holds it.
- A hit. The context goes to PSN Check with no trip off the chip, and QP 0x12 stays at the front of the LRU order. PSN Check and Out-of-Order Arrival
- A miss. The next SEND is for QP 0x77, a connection that has been quiet for a while.
- The lookup compares 0x77 against all four lines. None of them holds it.
- No. A miss is routine, not an error. The packet is held until its context arrives; nothing about it is discarded and the sender never finds out. QP Caching
- Making room. Every line is taken, so one QP has to go. This walk uses LRU, which picks the QP that has gone longest without a packet: 0x40, in line 3. Eviction Policy
- Not while it is dirty. QP 0x40 has handled packets since it was fetched, so its PSN counters on the chip are newer than the table’s copy. Line 3 is written back over PCIe first; only then is it free. Eviction Policy
- The fetch. The cache reads QP 0x77’s entry from the QP Table, a second PCIe round trip, and fills line 3 with it. A freshly fetched line is clean. The QP Table
- The held packet gets its context and goes on to PSN Check, exactly as a hit would have. It arrives later than the hit did by one PCIe write and one PCIe read. PSN Check moves QP 0x77’s expected PSN on, so line 3 is dirty again.
- The QP that was pushed out. None. It is the QP that was evicted to make room for 0x77. The transmit side misses too, and LRU now picks QP 0x31 in line 2. QP Caching
- QP 0x31 is dirty, so it is written back first.
- QP 0x40 comes back from the QP Table into line 2, carrying the counters it was written back with.
- TX Packet Build gets the context and can start. Five busy QPs and four lines: the first miss pushed out a QP that was needed moments later, and bringing it back pushed out another. That is the working-set problem this chapter is about, and the reason Eviction Policy asks whether recency alone is the right test. Eviction Policy
Why
An RNIC could, in principle, skip caching entirely and hit the QP Table on every packet — simpler hardware, no working-set question to answer, no eviction policy to design. The cost of that choice is a DRAM or PCIe transaction sitting on the critical path of every single packet, at every port, all the time, rather than only on the packets touching a QP outside the current working set – which is exactly the cost caching exists to avoid. The tradeoff a real implementation actually makes isn’t “cache or don’t” — it’s how much scarce, expensive on-chip SRAM to spend on QP context specifically, against every other structure (address translation, buffering) competing for the same die area, knowing that whatever amount gets chosen, some real deployment will eventually run more concurrently-active QPs than it can hold.
Installing a QP covers what happens when a QP doesn’t yet exist in either the cache or the table at all — the software-to-hardware handoff that creates one.
References
- No IETF or IBTA standard governs on-chip QP context caching and working-set sizing – it is implementation-specific silicon design, not a wire-format or software-interface contract any standards body defines. This chapter’s treatment is grounded in the sources cited in-text above, not a standards document.