Skip to content
Overview
The Operation Catalog
Atomics

Atomics

OpcodeExtended headersNotes
COMPARE_SWAPBTH + ATMETHconditional swap against a 64-bit remote word
FETCH_ADDBTH + ATMETHadd to a 64-bit remote word, return the prior value
ATOMIC_ACKNOWLEDGEBTH + AETH + ATMACKatomic-specific ack, returns the pre-operation value

ATOMIC_ACKNOWLEDGE is the atomic family’s own response. The plain ACKNOWLEDGE (BTH + AETH), the ACK or NAK for any SEND or RDMA WRITE message, belongs to no family: it is the reliability layer’s response to SEND and WRITE alike.

Atomics have no FIRST/MIDDLE/LAST variants at all — both operations act on a single fixed 64-bit value, so there is no such thing as a multi-packet atomic message to frame.

Atomics are the one family that costs more at the responder than a plain DMA write does. COMPARE_SWAP and FETCH_ADD both require an in-hardware read-modify-write at the responder’s memory: read the current 64-bit value, then either conditionally replace it (compare against ATMETH’s comparand, swap only on match) or add to it, atomically with respect to any other operation touching that same word — and then hand the pre-operation value back to the requester in ATOMIC_ACKNOWLEDGE’s ATMACK field, so the requester knows what was actually there before its atomic executed. A WRITE places data; an Atomic reads, computes, and conditionally replaces, all as one indivisible responder-side action — a memory-side compute operation, not just a placement, and meaningfully more hardware than the rest of this catalog needs.

Compare and Swap

A compare-and-swap that succeeds, with the original value coming back:

A COMPARE_SWAP on a remote 64-bit word: the request carries an ATMETH with the address, rkey, swap and compare values; the responder performs one indivisible read-modify-write and returns the original value in ATOMIC_ACKNOWLEDGE, which the requester writes to its reply buffer.
Atomic Compare and Swap
All 6 steps
  1. The application asks for a compare and swap on a remote 64-bit word: if it holds 5, make it 9. The WQE and the Send Queue Descriptor
  2. The request is one packet with an ATMETH: target address, rkey, swap value and compare value. The target must be 8-byte aligned. A local buffer, under an lkey, waits for the reply. Atomics
  3. PSN check, then the rkey, which this time must carry remote atomic permission. Permission and Bounds Violations
  4. It reads 5, finds it equal to the compare value, and writes 9, as one indivisible read-modify-write. A WRITE only places bytes; this computes on them. Atomics
  5. It sends back the value it found, 5, not the one it wrote. ATOMIC_ACKNOWLEDGE carries it in the ATMACK, and the requester writes it to the reply buffer. Atomics
  6. Only if the Work Request was signaled: The completion posts. Software sees 5 come back, equal to its compare value, and knows the swap happened. Had the word held 7, it would get 7 back and nothing would have changed. CQE Generation

The same operation across both hosts. The atomic unit on RNIC B is the one block no other family lights, and the read, the compare and the write happen there as a single step.

A COMPARE_SWAP across both hosts: the operands travel inside the request, RNIC B's atomic unit reads the remote word, compares and writes the swap value as one indivisible step, and the original value comes back in ATOMIC_ACKNOWLEDGE to be written into Host A's reply buffer.
Compare and Swap, block by block
All 9 steps
  1. On Host A (Requester). The application writes a COMPARE_SWAP WQE: a remote 64-bit word and its rkey, compare value 5, swap value 9, and a local buffer for the reply. It rings the doorbell. The Send Queue Ring
  2. RNIC A reads the WQE out of host memory and looks up the QP’s context. WQE Prefetch
  3. Both operands are in the WQE itself, so no payload is read from Host A’s memory. The Key Cache checks the lkey on the reply buffer, and TX Packet Build sends one packet: a BTH and an ATMETH. The Gather Path
  4. On Host B (Responder). RX Parse checks the PSN, and the Key Cache checks the rkey. This time the region must allow remote atomics, and the address must be 8-byte aligned. Permission and Bounds Violations
  5. The atomic unit reads the target word through the DMA engine. It holds 5. Target Selection
  6. 9 is written, and nothing else touches the word between the read and the write: the two are one indivisible step. How hardware makes them indivisible is the adapter’s choice. Linux’s soft-RoCE driver holds a lock across both; PCIe also defines atomic operations a device can ask the host to perform. Target Selection
  7. RNIC B’s TX path builds ATOMIC_ACKNOWLEDGE. Its ATMACK carries the value the word held before, 5, not the 9 just written. Atomics
  8. The reply returns (Requester). RNIC A matches the acknowledgement, retires the Replay Buffer entry, and writes the 8-byte original value into the reply buffer. The Scatter Path
  9. Only if the Work Request was signaled: The completion posts. The application sees 5, equal to its compare value, and knows the swap happened. Had the word held 7, it would read 7 back, and RNIC B would have skipped the write. CQE Generation

Fetch and Add

Fetch and Add always writes: it adds its operand to the word and returns what was there before. That makes a resent request dangerous in a way a resent WRITE never is. A WRITE placed twice leaves the same bytes; an add run twice counts twice. So the responder keeps the original value of each recent atomic, and answers a duplicate from what it kept instead of running it again.

A FETCH_ADD of 1 on a remote counter holding 41: the responder writes 42 and keeps 41. The ATOMIC_ACKNOWLEDGE is lost, the requester resends PSN 3100, and the responder answers the duplicate with the saved 41 instead of adding again.
Atomic Fetch and Add, and a duplicate that must not add twice
All 11 steps
  1. The application asks to add 1 to a remote 64-bit counter and get back what it held before. The WQE and the Send Queue Descriptor
  2. The request is one packet: a BTH and an ATMETH carrying the address, the rkey and the add value. It gets a Replay Buffer entry and the retry timer arms, like any request. The Replay Buffer
  3. PSN check, then the rkey, which must allow remote atomics on an 8-byte-aligned address. Permission and Bounds Violations
  4. The responder reads 41 and writes 42 as one indivisible read-modify-write. Fetch and Add always writes; there is no compare to fail. Atomics
  5. It keeps 41 alongside PSN 3100. Nothing needs it yet. QP Context
  6. The ATOMIC_ACKNOWLEDGE carrying 41 is lost in the fabric. Go-Back-N Retransmission
  7. No acknowledgement arrives, the ACK timeout expires, and the requester rebuilds PSN 3100 from the Replay Buffer. Retry and Timeout State
  8. Not add again. PSN 3100 is below the expected 3101, so it is a duplicate. The responder finds the 41 it kept for that PSN and sends it back without touching the counter. If the saved value were already gone, Linux’s soft-RoCE driver would drop the request rather than run it. PSN Check and Out-of-Order Arrival
  9. This acknowledgement gets through, with the original value in its ATMACK. Atomics
  10. The requester writes 41 to the reply buffer and retires the entry. The counter holds 42: one add, though the request crossed the wire twice. The Scatter Path
  11. Only if the Work Request was signaled: The send completion posts with 41 in the reply buffer. CQE Generation
A FETCH_ADD across both hosts: RNIC B reads 41, writes 42 and keeps 41. The acknowledgement is lost, RNIC A resends the request from its Replay Buffer, and RNIC B recognises the duplicate and answers with the saved 41 without touching Host B's memory again.
Fetch and Add, block by block, with a duplicate
All 12 steps
  1. On Host A (Requester). The application writes a FETCH_ADD WQE: a remote 64-bit counter and its rkey, an add value of 1, and a local buffer for the reply. It rings the doorbell. The Send Queue Ring
  2. RNIC A reads the WQE out of host memory and looks up the QP’s context. WQE Prefetch
  3. After the lkey check on the reply buffer, TX Packet Build sends a BTH and an ATMETH carrying the add value; the compare field goes unused. A Replay Buffer entry is recorded and the retry timer arms. The Replay Buffer
  4. On Host B (Responder). RX Parse checks the PSN, and the Key Cache checks the rkey for remote atomic permission on an 8-byte-aligned address. Permission and Bounds Violations
  5. The atomic unit reads the counter. It holds 41. Target Selection
  6. It writes 42, as part of the same indivisible step. Unlike Compare and Swap, Fetch and Add always writes. Target Selection
  7. RNIC B keeps the original value, 41, with the request’s PSN, then sends ATOMIC_ACKNOWLEDGE. The acknowledgement is lost in the fabric and never reaches RNIC A. Go-Back-N Retransmission
  8. The resend (Requester). RNIC A’s retry timer expires. It rebuilds FETCH_ADD PSN 3100 from the Replay Buffer and sends it again, unchanged. Retry and Timeout State
  9. The duplicate (Responder). No. RNIC B expects 3101, so 3100 is a duplicate. It is not executed again: RNIC B finds the 41 it kept for that PSN, and Host B’s memory is not touched. PSN Check and Out-of-Order Arrival
  10. The acknowledgement goes again, with the same original value, 41. Atomics
  11. The reply returns (Requester). RNIC A writes 41 into the reply buffer and retires the Replay Buffer entry. The counter on Host B holds 42: the add ran once, though the request crossed the wire twice. The Scatter Path
  12. Only if the Work Request was signaled: The completion posts to Host A’s Completion Queue. CQE Generation

References

  • InfiniBand Trade Association, InfiniBand™ Architecture Specification, Volume 1 (General Specifications).
  • Linux kernel soft-RoCE driver, drivers/infiniband/sw/rxe/ – rxe_mr.c (rxe_mr_do_atomic_op) rejects an unaligned target and performs the read, compare or add, and write under one lock; rxe_resp.c (atomic_reply, duplicate_request) saves the original value with the request’s PSN and replays it for a duplicate instead of executing again.
Last updated on
RDMA READOne Transaction, Start to Finish