CNP Generation

Overview

Nothing this Part has covered so far explains what actually triggers a rate cut. Rate-Limiter Context named the reaction factor that responds to a Congestion Notification Packet; this chapter is where that packet actually comes from. It’s the one mechanism in this Part that runs on the receive side of a connection, triggered by something the RX pipeline sees, not anything the sending QP does.

Locator

Every other chapter in this Part is scoped to the sending side of a connection. This block is different: it sits on the receiving side, and what it produces travels back out to the peer that caused it.

Block diagram

Deep walkthrough

What actually triggers this. RX Pipeline already covers parsing an arriving packet’s IP header; this chapter picks up from there with one additional fact that pipeline didn’t need to mention until now: that IP header can carry an ECN mark, stamped by a switch somewhere along the path under load, exactly as RoCEv1 → RoCEv2 already established. Seeing that mark on an arriving packet is what starts everything in this chapter – nothing else about the packet changes because of it; it’s still parsed, still checked, still placed the way RX Pipeline already describes.

The rate-limited notification rule – sourced directly, not this book’s own design. A NIC doesn’t send a CNP for every single marked packet it sees; that would flood the reverse path under sustained congestion. The rule, quoted directly from the algorithm this book sources throughout this Part: “If a marked packet arrives for a flow, and no CNP has been sent for the flow in last N microseconds, a CNP is sent immediately. Then, at most one CNP packet every N microseconds for the flow, if any packet that arrives within that time window was marked.” The cited deployment uses N = 50µs; a second, independent implementation documented elsewhere in this Part uses a different default for its own related timer, further evidence this specific number is a tuned parameter, not a fixed constant.

Play it with two senders sharing one switch port toward a receiver: the mark, the CNP, and the rate cut on each sender. A ring collective is the opposite pattern: in One All-Reduce, Four Hosts every rank sends to exactly one neighbour, so on a single switch no output port has two senders aimed at it.

All 11 steps

1. Congestion builds. RNIC A is streaming a large Write to Host B at full rate. Its packets carry an ECN-capable code point in the IP header. RoCEv1 → RoCEv2
2. RNIC C starts sending to Host B as well, while A keeps streaming. Both flows need the same switch port at the same time. Congestion Control Engine
3. Two links’ worth of traffic arrives for one link’s worth of drain, so the egress queue toward B grows. Past a threshold, the switch starts setting the ECN bits on packets it forwards. RoCEv1 → RoCEv2
4. The receiver notifies. A marked packet reaches RNIC B. It is parsed, checked and placed as usual; the mark changes nothing about delivery. CNP Generation
5. The mark wakes CNP generation. No CNP has gone to this flow recently, so one goes at once; further marks inside the window (50 µs in one deployment) are folded into it. CNP Generation
6. The CNP is a BTH, a reserved field and the ICRC: opcode 0x81, DestQP naming RNIC A’s QP, PSN forced to 0. It should leave ahead of ordinary traffic. RNIC C’s flow was marked too, so a second CNP, naming C’s QP, follows it. CNP Generation
7. The switch forwards each CNP to its sender, A’s and C’s together. The switch only marked; it is the receiver that tells each sender. CNP Generation
8. The senders react. The rate limiter cuts this QP’s rate. The rate it had been sending at becomes the target to climb back toward, and both recovery counters reset to zero. The Rate Control State Machine
9. RNIC C cuts its own QP’s rate at the same moment, from its own CNP. Each sender reacts only for its own flow. Multi-Flow CNP Arbitration
10. With both senders slower, the queue drains below the threshold and the switch stops marking. Congestion Control Engine
11. With no new CNP, each rate limiter climbs back. Fast recovery halves the distance to the target on each timer or byte-counter expiry, then additive increase raises the target itself. A new mark at any point starts the cycle again. The Rate Control State Machine

The packet itself is standard-defined, not this book’s invention. RoCEv2’s own annex specifies a real CNP wire format: a BTH with Opcode set to 0b10000001 (0x81), DestQP set to the flow being throttled, PSN forced to 0, SE and M forced to 0, and P_KEY copied from the marked packet’s own BTH. A reserved field and the ICRC follow the BTH; no other transport header does. The WQE and the Send Queue Descriptor’s opcode table never lists this value, for the same reason The CQE’s completion-opcode vocabulary needed values no Work Request ever requests: nothing about posting a Work Request ever asks for a CNP to be sent. It’s generated entirely by this mechanism, not requested by software.

A sent CNP still has to be scheduled, the same as any other outgoing packet. Building a CNP doesn’t put it on the wire by itself – it consumes the same egress path TX Packet Build already covers, competing for the same resources any other outgoing packet does, just triggered from the receive side instead of a scheduled WQE. DCQCN’s own documentation is explicit that this competition matters: CNPs should go out “with high priority, to avoid missing the CNP deadline, and to enable faster convergence” – language that names exactly the kind of priority-scheduling question Multi-QP TX Scheduling already covers for control-plane traffic generally. This chapter doesn’t re-derive that scheduling story; a CNP needing to jump the queue ahead of ordinary WQE-sourced packets is this Part’s own instance of a question that chapter already owns.

Why generating CNPs promptly matters enough to be worth its own hardware budget, beyond scheduling priority. A real, named implementation’s own generation capability is a bounded, finite resource: one shipped generation supports the required CNP rate for roughly 10-20 simultaneously congested flows, a later generation from the same vendor over 200. This isn’t a hypothetical capacity question – Multi-Flow CNP Arbitration, later in this Part, covers what happens when more flows need a CNP at once than generation can keep up with.

Where this connects. The CNP this chapter builds travels back across the wire to the QP named in its DestQP field – the sender whose traffic caused the congestion in the first place. What that sender’s rate limiter actually does upon receiving it – the rate cut, the reaction-factor update – is The Rate Control State Machine, next. This chapter’s own job ends the instant the CNP is handed off to the same egress path every other outgoing packet uses.

Interface reference

References

• InfiniBand Trade Association, Supplement to InfiniBand Architecture Specification Volume 1, Release 1.2.1, Annex A17: RoCEv2 (2014) – the CNP wire format in full, cross-corroborated via a second independent secondary source (a named engineer’s public whitepaper, itself footnoted directly to this same annex) rather than read from the annex text itself.
• Zhu, Eran, Firestone, Guo, Lipshteyn, Liron, Padhye, Raindel, Haj Yahia, Zhang, “Congestion Control for Large-Scale RDMA Deployments,” SIGCOMM 2015 – the rate-limited notification rule (quoted directly), the CNP-priority rationale, and the ConnectX-3/ConnectX-4 generation-capacity figures.