Skip to content

A hardware-level guide to RDMA NICs

RDMA NIC architecture, block by block

Most writing on RDMA stops at “the hardware handles it.” This book opens the RNIC: every table, cache, state machine and pipeline stage between a Work Request and the wire.

Start reading → 10 parts · 86 chapters · 41 step-through walks

Contents

In reading order, from what RDMA asks of a NIC to the last address translation

  1. 01 Overview What RDMA asks of a NIC, the RoCEv2 packet, the four operation families, and one transaction end to end. 10 chapters · 6 with walks
  2. 02 Queue Pair Management The QP Table and context, the QP Cache, the state machine, and what happens when a QP faults. 11 chapters · 3 with walks
  3. 03 Memory Region / Key Table How an lkey or rkey becomes memory: registration, the Key Cache, and permission checks. 10 chapters · 4 with walks
  4. 04 TX Pipeline The Send Queue ring, WQE prefetch, packet build, scheduling across QPs, and the responses nobody posted. 9 chapters · 4 with walks
  5. 05 RX Pipeline Parsing, the PSN check, multi-packet reassembly, and the Receive Queue. 8 chapters · 2 with walks
  6. 06 Reliability Engine The Replay Buffer, retry and RNR timers, go-back-N, and the MSN. 9 chapters · 4 with walks
  7. 07 Congestion Control Engine DCQCN: ECN marks, CNPs, and the per-QP rate limiter that cuts and recovers. 10 chapters · 2 with walks
  8. 08 Completion Queue / Event Management CQE generation, overflow, notification, and arbitration across many CQs. 7 chapters · 3 with walks
  9. 09 DMA Engine Target selection, the gather and scatter paths, ordering, and FLUSH. 9 chapters · 4 with walks
  10. 10 System IOMMU / SMMU The second translation past PCIe, IOTLB misses the NIC can't see, and ATS. 3 chapters · 2 with walks

The whole adapter

Every block is a chapter. Click one to open it.

rnic.io -- RNIC Architecture, Master Block DiagramEvery block this book covers. Each chapter opens with this same drawing, one block highlighted, the rest dimmed.Host MemoryDescriptor RingsWQE Fetch /TX SchedulerTX Packet BuildBTH+RETH/AETH, ICRCwireoutQP CacheQP TableMR / Key TableKey CacheReplay Buffer /Retry TimerRate LimiterCNP Generationreceive side, on ECN markfour instances of the sametable + working-set + eviction shapeDMA EngineAddress translation,scatter-gather walkSystem IOMMU / SMMUoutside the RNIC, if presentwireinRX Parser /ReassemblyRX PSN CheckCQE GenCQ StorageHost:Event / Interruptwhich QPrate checkgateslkey/rkeyresolved heregathered payloadunacked copy heldsend completion (if signaled)dest QPfrom BTHack matchingaccepted -> DMA writeresponse duereceive completionECN mark seen

Step-through walks

Each packet's headers, each block's state, and a question to answer before the next step

Overview One RDMA Write, start to finish A WRITE with Immediate through both adapters, both hosts and a switch. ▶ 12 steps Overview A ring all-reduce Four hosts summing their buffers, hop by hop, in the RDMA operations NCCL issues. ▶ 10 steps Reliability Engine A lost packet One packet dropped, a NAK, and go-back-N resending three. ▶ 11 steps Queue Pair Management A QP Cache miss A miss, a dirty write-back, and the evicted QP missing again moments later. ▶ 14 steps Congestion Control Engine Two senders, one port ECN marks, CNPs and two rate cuts at a congested switch port. ▶ 11 steps DMA Engine Gather and scatter One SEND read from three buffers and written into two. ▶ 12 steps