Skip to content
Overview

Overview

A conventional NIC moves frames between a wire and a ring buffer and stops there. An RDMA-capable NIC takes a virtual address and a remote key and moves bytes into another machine’s memory directly – no syscall, no remote CPU, no unauthorized byte moved. This part names that shift, lays out the RoCEv2 wire format that carries it, and walks one operation through the whole machine. Every stage that operation touches gets its own part later in the book.

The whole chip, in one picture

Below is the same diagram every chapter in this book opens with – normally with just one box lit up to show where that chapter fits. Here it’s shown whole, before anything gets zoomed in on.

The dashed boxes on the left and right sit in the host machine, not on the RNIC chip itself – that’s where an operation starts and where it eventually gets reported back. The amber boxes store or look something up (a connection’s state, an address translation, an unacknowledged packet waiting to be resent). The blue boxes are where a packet actually gets built, parsed, or moved. Everything else is just the wiring between them.

Every box is clickable – click one to go straight to its chapter. Once you’ve read One Transaction, Start to Finish, play that transaction across this diagram, block by block.

rnic.io -- RNIC Architecture, Master Block DiagramEvery block this book covers. Each chapter opens with this same drawing, one block highlighted, the rest dimmed.Host MemoryDescriptor RingsWQE Fetch /TX SchedulerTX Packet BuildBTH+RETH/AETH, ICRCwireoutQP CacheQP TableMR / Key TableKey CacheReplay Buffer /Retry TimerRate LimiterCNP Generationreceive side, on ECN markfour instances of the sametable + working-set + eviction shapeDMA EngineAddress translation,scatter-gather walkSystem IOMMU / SMMUoutside the RNIC, if presentwireinRX Parser /ReassemblyRX PSN CheckCQE GenCQ StorageHost:Event / Interruptwhich QPrate checkgateslkey/rkeyresolved heregathered payloadunacked copy heldsend completion (if signaled)dest QPfrom BTHack matchingaccepted -> DMA writeresponse duereceive completionECN mark seen

Chapters

What Is RDMA
A normal NIC hands a frame to software and stops there. An RDMA-capable NIC is handed a virtual address and a remote key and is expected to move bytes into another machine’s memory without a syscall, without the remote CPU, and without a single unverified byte – this book is about what building that actually takes in hardware.
What Is RoCEv2
A RoCEv2 packet looks like an ordinary UDP datagram to everything sitting between two endpoints, and nothing like one to either end running verbs – what’s actually packed inside is where this book’s build-by-block depth begins.
RoCEv1 → RoCEv2
RoCEv1 could only run inside one Ethernet broadcast domain. RoCEv2 fixed that by wrapping the exact same transport headers in a real IP/UDP packet — and that one change put a header-construction datapath in the NIC that RoCEv1 hardware never needed.
The Operation Catalog
The four Reliable Connection operation families, the one thing each asks new of the hardware, and every operation played twice: packet by packet between requester and responder, and block by block across both hosts and both adapters.
One Transaction, Start to Finish
Every mechanism in this book gets its own chapter, in isolation. This is the one place they run together — one RDMA Write, start to finish, with a map of exactly which later chapter owns each stage it passes through.
One All-Reduce, Four Hosts
Distributed training spends much of its network time in one collective: every GPU ends up with the sum of every GPU’s buffer. Here is that all-reduce done the common way, as a ring over four hosts and a switch, with each hop broken down into the verbs an RNIC actually executes.
How to Read This Guide
Every chapter from Queue Pair Management onward follows the same template and draws on the same five color-coded callouts. Both are named here once, before Queue Pair Management starts.
Last updated on