ALL_REDUCE

This page is the reimplementer's reference for the ALL_REDUCE collective on the Vision-Q7 / Trainium TPB: the opcode encoding, the 64-byte argument struct, the reduce-operation mapping (ADD/MAX/MIN/FMA), and the full lowering path by which a single logical all-reduce becomes device traffic — BIR InstCollective → the host NRT SELECT → COMPOSE → EMIT rewrite → the per-step SB2SB (0xBF) leg + the in-SDMA CCE reduce → the EVT_SEM completion.

The one fact to anchor everything: an ALL_REDUCE pseudo-op never executes on hardware. It is a compiler placeholder that the host runtime (libnrt.so) lowers, at model-load time, into a program for the TOP_SP sequencer plus DMA descriptor rings. The device GPSIMD/POOL ucode only ever sees the lowered legs — there is no TriggerAllReduce in the device decode set (§8). This page documents the host-side machine; for the device walk and the firmware state machine see A Collective, End to End, the NCFW Ring + Kangaring, Mesh, and Hierarchical pages.

All claims are tagged [CONFIDENCE × PROVENANCE]: HIGH/MED/LOW × OBSERVED (read directly from a binary/header/JSON/DWARF), INFERRED, or CARRIED (established on a sibling page).

Provenance. ISA headers from aws-neuronx-gpsimd-customop-lib 0.21.2.0 (Cayman = NC-v3 tree, compile-verified with gcc sizeof/offsetof); host runtime from aws-neuronx-runtime-lib 2.31.24.0-0b044f4ce — libnrt.so.2.31.24.0 (BuildID 8bb57aba…, x86-64, not stripped, with debug_info) and libnrtucode_extisa.so. In libnrt.so .text/.rodata/.data are all VMA == file-offset (readelf -SW: .data VMA 0xc07e00 == offset 0xc07e00); in libnrtucode_extisa.so .data carries a +0x1000 delta (VMA 0x9350a0, offset 0x9340a0). C++ vtable slots are measured from vptr = _ZTV + 0x10; the ucode .so uses C fn-ptr tables.

1 · The three carriers — ALL_REDUCE is both an opcode and a ctype

ALL_REDUCE is expressible in the TPB ISA in three distinct ways, all of which libnrt recognises (both PseudoTriggerAll and PseudoTriggerCol appear in the runtime's 16-char opcode-name table — §3):

So "is ALL_REDUCE a distinct opcode, or TRIGGER_COLLECTIVE with comm_type = ALL_REDUCE?" — it is both: a distinct opcode (0xC7) and a ctype value (0x1) on 0xC8 and 0xD9/0xDA. [HIGH/OBSERVED — OPCODE enum lines 0xC7@269 / 0xC8@270 / 0xD9@287 / 0xDA@288 in aws_neuron_isa_tpb_common.h; instruction_mapping.jsonl.181 binds…_ALL_REDUCE_STRUCT → …_OPCODE_PSEUDO_TRIGGER_ALL_REDUCE.]

The pseudo-op class rule

0xC7 is a pseudo / control op, never a datapath op. The decisive rule is in the header verbatim (common.h:263):

// NRT relies on the fact that all pseudo instructions have upper three bits of the
// opcode equal to 0b110. Pseudo instructions are generated by compiler and
// translated into non-pseudo HW instructions by NRT.

0xC7 = 0b1100_0111 — upper three bits 0b110 ✓ — so it never runs on a HW engine; NRT rewrites it. The static ISA classifiers agree: PSEUDO_TRIGGER_ALL_REDUCE is a member of is_pseudo_op and is_control_instruction (aws_neuron_isa_tpb_instr_assert.h lines 602, 636, 841, 1029, 1166, 1211). [HIGH/OBSERVED]

GOTCHA — no static validation body. There is no is_valid_* assert body for the 0xC7 struct. aws_neuron_isa_tpb_assert.h:13571 carries only an empty section marker pseudo_trigger_all_reduce_assert.h (same pattern as the base 0xC8 @13574 and 0xD9/0xDA @13577/13580). The pseudo-op is validated and lowered by the runtime, not by the static ISA assert layer. Do not look for a field-range checker in the headers — there isn't one. [HIGH/OBSERVED]

NOTE — enums.h pretty-printer. The human-readable name "ALL_REDUCE" is emitted by a strncpy(buffer,"ALL_REDUCE",buflen-1) at aws_neuron_isa_tpb_enums.h:2828 (that is a line number, not a hex opcode), with the opcode/ctype membership tests at lines 311 (== …PSEUDO_TRIGGER_ALL_REDUCE) and 491 (== …COLLECTIVE_TYPE_ALL_REDUCE). [HIGH/OBSERVED — corrects a "0x2828 opcode" mis-reading.]

QUIRK — byte-identical across arches. The 0xC7 struct body is byte-identical across all four shipped arches (Cayman v3 / Mariana v4 / Maverick v5 / Sunda v2); the only diff is the ISA header for NC-vN banner comment. The lowering differs per arch (the CSR offsets in §7 are per-arch), but the encoding does not. [HIGH/OBSERVED — header diff.]

2 · The argument struct — 0xC7 PSEUDO_TRIGGER_ALL_REDUCE, 64 bytes

The dedicated form is the simplest. Not __packed__ at top level → natural C alignment; sizeof == 64, every offset compile-confirmed (gcc offsetof, this pass):

// aws_neuron_isa_tpb_pseudo_trigger_all_reduce.h
typedef struct NEURON_ISA_TPB_PSEUDO_TRIGGER_ALL_REDUCE_STRUCT {
    NEURON_ISA_TPB_HEADER header;            //  4   ( 0 -  3)  opcode==0xC7 in header.opcode
    NEURON_ISA_TPB_EVENTS events;            //  8   ( 4 - 11)  sem/event wait+update block
    NEURON_ISA_TPB_ALU_OP op;                //  1   (12     )  THE REDUCE OP (ADD/MAX/MIN…), §4
    NEURON_ISA_TPB_DTYPE  dtype;             //  1   (13     )  element data type, §5
    uint8_t               reserved0[2];      //  2   (14 - 15)  pad, treat 0
    uint32_t              input_tensor_id;   //  4   (16 - 19)  SRC compiler tensor handle
    uint32_t              output_tensor_id;  //  4   (20 - 23)  DST compiler tensor handle
    uint64_t              num_elements;      //  8   (24 - 31)  elements reduced/transferred
    uint8_t               reserved1[32];     // 32   (32 - 63)  pad to 64, treat 0
} NEURON_ISA_TPB_PSEUDO_TRIGGER_ALL_REDUCE_STRUCT;            // sizeof == 64

Member sizes (compile-verified): HEADER=4, EVENTS=8, ALU_OP=1, DTYPE=1. The struct is bound into the instruction union at aws_neuron_isa_tpb_util.h:47 (… pseudo_trigger_all_reduce;). [HIGH/OBSERVED — /tmp/probe_ar.cprintssizeof=64, offsets 0/4/12/13/14/16/20/24/32.]

The header documents the intent verbatim:

"TriggerAllReduce expands to a series of instructions executed on TOP SPs, which uses DMA to perform an all reduce operation. It then triggers the operation. TriggerAllReduce is a pseudo-instruction generated by compiler. It is replaced by TOP SP instructions by the runtime." [HIGH/OBSERVED]

input_tensor_id/output_tensor_id are compiler tensor handles, resolved to a NeuronAddr at model load; num_elements is 64-bit, so a single all-reduce can span very large buffers. [layout HIGH/OBSERVED; the handle→NeuronAddr runtime resolution is MED/INFERRED — not in the headers.]

0xC7 vs 0xC8 vs 0xD9/0xDA — the carrier diff

The dedicated form is the compact, single-replica-group ALL_REDUCE; the richer knobs live on the generic and v2 carriers:

[All rows HIGH/OBSERVED — the three Cayman headers pseudo_trigger_collective{,2,2_ext}.h: 0xC8hasgroup_id@14, ctype@32, src/dst_offset_elems@48/@56; 0xD9hasctype@14, cc_dim@15, src0@16; 0xDAext hasgroup_id@2, channel_id@4, stream_id@6, dst@8, cc_buf_start_addr@32.] See TRIGGER_COLLECTIVE (0xC8) and TRIGGER_COLLECTIVE2 / EXT (0xD9/0xDA).

3 · BIR InstCollective → host recognition

At model load libnrt decodes the NEFF instruction stream. The two trigger mnemonics sit, 16 ASCII characters each, in the runtime's opcode-name table — recoverable directly from the binary as a packed run:

…PseudoSemaphoreSPseudoLoadActFunPseudoTriggerAllPseudoTriggerColPseudoReadVarAdd…
   └ 0xC5         └ 0xC6          └ 0xC7          └ 0xC8          └ 0xC9

[HIGH/OBSERVED — strings libnrt.so; each field is 16-char padded, indexed by opcode.] The 0xD9 path additionally asserts the back-to-back pair: the assert string ptc2_ins->header.opcode == NEURON_ISA_TPB_OPCODE_PSEUDO_TRIGGER_COLLECTIVE2 and the first_ptc2_ins symbol are both present. [HIGH/OBSERVED]

From the decoded fields NRT builds an enc_operation / cc_op record, mapping the ISA vocabularies onto the runtime's:

• op kind → enc_op_type = ENC_ALLREDUCE (1)
• reduce op → SDMA_CCETYPE (§4)
• dtype → SDMA_DTYPE (verbatim, §5)

enc_op_type is read straight from DWARF (IDA _enums.json): ENC_ALLGATHER=0, ENC_ALLREDUCE=1, ENC_BROADCAST=2, ENC_REDUCE=3, ENC_REDUCE_SCATTER=4, ENC_SEND=5, ENC_RECV=6, ENC_ALLTOALL=7, ENC_PERMUTE=8, ENC_PERMUTE_REDUCE=9, ENC_PERMUTE_IMPLICIT=10, ENC_PERMUTE_REDUCE_IMPLICIT=11, ENC_ALLTOALL_V=12 (ENC_OP_INVALID = ENC_OP_N = 13). ⇒ ALL_REDUCE → ENC_ALLREDUCE = 1. [HIGH/OBSERVED]

ALL_REDUCE is not a single enc_comm_type value. enc_comm_type = {H_COMM_INTRA_ID=0, H_COMM_INTER_ID=1, H_COMM_MAX_ID=2}; a hierarchical all-reduce runs both an INTRA leg and an INTER leg (§6). [HIGH/OBSERVED for the enum; the "spans both scopes" reading is OBSERVED from the hier-decomposition builder names, §6.2.]

4 · The reduce operation — ADD / MAX / MIN / FMA

The reduction op rides the 1-byte op (ALU_OP @ 12) field and is carried through four vocabularies that all align on ADD/FMA/MAX/MIN, end to end.

4a · ISA ALU_OP (op @ 12) — common.h

ADD = 0x04, MAX = 0x08, MIN = 0x09 (int-engine forms ADD_INT = 0xC4, MAX_INT = 0xCF, MIN_INT = 0xD0). No FMA in the float ALU_OP set — FMA appears as a CCE/runtime op (4b–4c). [HIGH/OBSERVED — lines 944/948/949/970/981/982.]

4b · ISA CCE_OP (the hardware SDMA reduce primitive) — common.h:1003

NEURON_ISA_TPB_CCE_OP_ADD = 0x00,    // Same encoding as SDMA CCE op encoding
NEURON_ISA_TPB_CCE_OP_MAX = 0x02,    // Multiply    = 0x01,
NEURON_ISA_TPB_CCE_OP_MIN = 0x03,

This is THE hardware reduce primitive the collective DMA realises, and the header says so verbatim — "Same encoding as SDMA CCE op encoding". [HIGH/OBSERVED]

NOTE — not DGE_COMPUTE_OP. Do not confuse this with DGE_COMPUTE_OP (common.h:837: NONE=0, ADD=1, MULTIPLY=2, MAX=3, MIN=4), the DMA-generation- engine compute-on-copy. The collective reduce uses the SDMA CCE, not the DGE. The two enums even disagree numerically (CCE_MAX=2 vs DGE_MAX=3). [HIGH/OBSERVED — both enum blocks.]

4c · host SDMA_CCETYPE (the runtime's reduce-op carrier) — DWARF

SDMA_CCETYPE_ADD=0  SDMA_CCETYPE_FMA=1  SDMA_CCETYPE_MAX=2
SDMA_CCETYPE_MIN=3  SDMA_CCETYPE_EXT=4  SDMA_CCETYPE_GCE=5

SDMA_CCETYPE is the argument type of every ring/mesh reduce step primitive — the mangled symbols prove the 2nd argument is SDMA_CCETYPE:

enc_primitive::recv_reduce_send (enc_half_chunk_index, SDMA_CCETYPE, reduction_type_t, bool, bool)  @0x16ad70
enc_primitive::recv_reduce_copy (enc_half_chunk_index, SDMA_CCETYPE, reduction_type_t, vector<encd_neigh>, bool) @0x16b030
enc_primitive::recv_reduce_copy_send (…, SDMA_CCETYPE, …)                                            @0x16aed0
enc_primitive::direct_reduce_send_kangaring (enc_half_chunk_index, SDMA_CCETYPE)                     @0x158120
enc_primitive::direct_reduce_send_permute (SDMA_CCETYPE)                                             @0x16c950

[HIGH/OBSERVED — nm -C; the mangled …20enc_half_chunk_index12SDMA_CCETYPE…confirms the 2nd arg isSDMA_CCETYPE.]

The ISA CCE_OP {ADD=0, MAX=2, MIN=3} is a byte-exact subset of SDMA_CCETYPE {ADD=0, MAX=2, MIN=3}; SDMA_CCETYPE merely adds FMA=1 (the slot the CCE_OP comment reserves as Multiply=0x01) plus EXT=4/GCE=5. So the runtime's reduce-op carrier is the hardware SDMA CCE op encoding. [HIGH/OBSERVED — both enum tables.]

NOTE — reduction_type_t is a ring pattern, not the op. The 3rd argument to recv_reduce_* is enc_primitive::reduction_type_t = {RING_2R1W=0, RING_2R2W=1, KANGARING_NR1W=2} (DWARF) — the read/write rendezvous shape of the ring step (2-read/1-write, 2-read/2-write, N-read/1-write kangaring), not the ADD/MAX/MIN op. The op is the 2nd arg (SDMA_CCETYPE). [HIGH/OBSERVED — the enumerators ARE isolated in the IDA _enums.json.]

4d · host nrt_op_type (the public-API reduce-op enum) — DWARF

NRT_OP_ADD=0  NRT_OP_FMA=1  NRT_OP_MAX=2  NRT_OP_MIN=3  NRT_OP_INVALID=15

This is the op enum taken by nrt_cc_prepare @0x7f610 (the lower-level RT collective API). It orders identically to SDMA_CCETYPE (ADD0/FMA1/MAX2/MIN3), and translate_op_type(nrt_op_type) → SDMA_CCETYPE @0x7f5b0 is an identity map for ADD/FMA/MAX/MIN. The dispatch is observable in the disassembly of nrt_cc_prepare:

; nrt_cc_prepare @0x7f610, op_type in %ebp
7f667:  cmp  $0x1,%ebp        ; FMA → special scale-operand path
7f66a:  je   7f920
7f673:  cmp  $0x2,%ebp        ; MAX
7f676:  je   7f740
7f67c:  test %ebp,%ebp        ; ADD == 0 falls through
7f67e:  jne  7fbcc            ; else → error

⇒ {ADD, FMA, MAX, MIN} are the legal reduce ops, and FMA carries an extra operand (the cmp $0x1 branch). [HIGH/OBSERVED — disasm.]

4e · end-to-end reduce-op chain

ISA op (ALU_OP ADD0x04 / MAX0x08 / MIN0x09;  or API NRT_OP_*)
  → runtime maps to SDMA_CCETYPE {ADD0, FMA1, MAX2, MIN3}   (== ISA CCE_OP encoding)
  → SDMA_CCETYPE is the arg to recv_reduce_* / direct_reduce_* step primitives
  → those build an SDMA descriptor PACKET with the CCE reduce field set:
       add_dma_packet_cce @0x2307d0  →  vring_add_dma_packet_cce @0x3134e0
       al_sdma_m2s_build_cce_ext_meta_ctrl @0x451dc0   (CCE ext meta ctrl)
       al_sdma_m2s_build_fma_descriptor    @0x451c20   (FMA arm; 3 call sites)
       aws_sdma_get_cce_params_cayman      @0x471760   (writes 0x2000 / 0x800 = CCE max buf/elem)
  → the SDMA engine performs the element-wise reduce DURING the transfer
    (recv-and-reduce) — exactly the collective's `op`.

[each named hop HIGH/OBSERVED; the exact per-value ALU_OP 0x04/0x08/0x09 → SDMA_CCETYPE 0/2/3numeric remap is MED/INFERRED — the value-preservingCCE_OP ≡ SDMA_CCETYPE ≡ nrt_op_typealignment plus the clearADD/MAX/MINcorrespondence make the mapping unambiguous; theADD/MAX/MIN/FMA correspondence itself is HIGH.]

The CCE descriptor / reduce mechanics live on CCE (Compute-DMA) In-Transfer Compute — cce_info is the 140-byte kbin_dma_desc_cce_info_t (structures.json); the FMA descriptor is al_sdma_m2s_build_fma_descriptor @0x451c20. [both CARRIED/OBSERVED — re-verified this pass: struct size=140, symbol present, called from 3 sites.]

5 · dtype handling — which dtypes reduce

The dtype rides the 1-byte dtype (DTYPE @ 13) field and passes through verbatim to the SDMA descriptor: ISA DTYPE ≡ SDMA_DTYPE ≡ nrt_dtype, identical numeric encodings, no re-encoding on the lowering path. [HIGH/OBSERVED — ISA common.h DTYPE block (BFLOAT16=6, FP16=7, FP32=0xA, FP32R=0xB, FP8_EXP3..5=0xD..0xF) vs the DWARF SDMA_DTYPE enum, byte-identical.]

QUIRK — FP32R aliases the "reserved" slot. In SDMA_DTYPE both SDMA_DTYPE_RESERVED and SDMA_DTYPE_FP32R carry value 11 (0xB) — FP32R is the "rounded/replicated FP32" collective-accumulation tag occupying the reserved slot. [HIGH/OBSERVED — DWARF enum.]

Which dtypes actually reduce (are seeded into the CCE-reduce descriptor path): the rodata table cce_dtypes.4 @0x9b9f40 (0x60 bytes = 6 × {u64 dtype, u64 group}), consumed by dma_ring_set_seed_set_descriptors (loop +0x10, 6 iters):

[table bytes HIGH/OBSERVED — objdump -s @0x9b9f40reads06→2, 07→2, 0B→4, 0D→1, 0E→1, 0F→1.] The 2/2/4/1/1/1 column is the CCE element-grouping/packing factor (BF16/FP16 packed 2-wide, FP32R 4-wide, FP8 1-wide).

GOTCHA — plain FP32 (0x0A) is absent; FP32R is the FP32-class entry. The CCE reduce path seeds FP32R (0x0B), not FP32. This dovetails with the in-transfer page: sdma_data_type_size[fp32r = idx 11]@0x9be198 = 0 (read byte-exact this pass: 00 00 00 00 00 00 00 00; adjacent fp32@idx10 = 4, int64@idx12 = 8). FP32R rides the accumulation nibble, never the I/O byte-length nibble — a CCE reduce that asks for fp32r byte-sizing traps. Integers are likewise absent — all-reduce on this accelerator is predominantly a float (BF16/FP16/FP32R/FP8) operation. [table HIGH/OBSERVED; "exhaustive reducible set" MED/INFERRED — it is the descriptor-seed set, strongly implying the eligible dtypes; the fp32r → 0 size cross-check is OBSERVED, see [cce-in-transfer §1](../../dma/cce-in-transfer.md).]

The ctype values for completeness (COLLECTIVE_TYPE, common.h:793): INVALID=0, ALL_REDUCE=0x1, REDUCE_SCATTER=0x2, ALL_GATHER=0x3, ALL_TO_ALL=0x4, PERMUTE=0x5, PERMUTE_REDUCE=0x6, PERMUTE_IMPLICIT=0x7, PERMUTE_REDUCE_IMPLICIT=0x8, ALL_TO_ALL_V=0x9. ⇒ ALL_REDUCE = ctype 0x1. [HIGH/OBSERVED] See Collective Enum Reference.

6 · The host rewrite — SELECT → COMPOSE → EMIT

The runtime turns one ENC_ALLREDUCE record into device traffic in three phases.

6.1 · SELECT — __select_algorithms

ALL_REDUCE is not one algorithm; it is a choice the selector makes per communication leg. enc_hier_primitive::__select_algorithms @0x14c620 probes capability predicates and assigns an enc_alg_type per leg. The predicate call census inside __select_algorithms:

[HIGH/OBSERVED — nm -C signatures + the in-function call census.]

The selector draws from enc_alg_type (DWARF): RING=0, HIER=1, MESH=2, KANGARING=3, SINGLE_CYCLE_RING=4, INTRA_RDH=5, SINGLE_STEP_MESH=6, INTER_RDH=7, TWO_STEP_POD_MESH=8, LATENCY_OPT_MESH=9, BW_OPT_MESH=10, INVALID=11 (mesh sub-type enc_alg_mesh_type = {FULL_MESH=0, GROUPED_MESH=1, MESH_TRN2=2, MESH_SWITCH=3, INVALID=4}). [HIGH/OBSERVED]

Per-leg legality (verbatim libnrt assertion strings, the ALL_REDUCE rows; allr = all-reduce, redsct/allg = its sub-legs):

alg_intra_redsct == ENC_ALG_RING || alg_intra_redsct == ENC_ALG_KANGARING
alg_intra_allg   == ENC_ALG_RING || alg_intra_allg   == ENC_ALG_KANGARING
alg_inter_allr   == ENC_ALG_RING || alg_inter_allr   == ENC_ALG_INTER_RDH
                                  || alg_inter_allr   == ENC_ALG_SINGLE_CYCLE_RING
(alg_inter_allr  == ENC_ALG_MESH) || inter_metaring          // mesh alternative

i.e. intra reduce-scatter / all-gather → RING or KANGARING; inter all-reduce → RING, INTER_RDH, SINGLE_CYCLE_RING, or MESH. [HIGH/OBSERVED — strings libnrt.so.]

NOTE — SINGLE_CYCLE_RING is ALL_REDUCE-only. Proven by the op_type guard inside enc_can_post_single_cycle_ring @0xfaa80:

faa99:  mov  %ecx,%ebp        ; %ebp = op_type
faadd:  cmp  $0x1,%ebp        ; op_type == 1 == ENC_ALLREDUCE ?
faae0:  jne  fab50            ; no → reject

Strings confirm: Single Cycle ALLR, NEURON_RT_DBG_SINGLE_CYCLE_RING_ALLR_CC, dbg_single_cycle_ring_allr_mode. It is the latency-optimised single-pass ring all-reduce (reduce-scatter + all-gather fused into one cycle). [HIGH/OBSERVED — disasm + strings.]

A hierarchical all-reduce carries five per-leg algorithm choices, exposed by the runtime log (verbatim):

[nec_dev %d] Hier algorithm selections - alg_intra_allg %d alg_intra_redsct %d
              alg_inter_allr %d alg_inter_allg %d alg_inter_redsct %d

[HIGH/OBSERVED] Which numeric enc_alg_type a given (world-size, topology, dtype) resolves to depends on the can_post thresholds, not fully enumerated — [MED].

6.2 · COMPOSE — __compose_allreduce*

The chosen algorithm's composer emits the concrete step primitives. Dispatch enters through enc_{hier,mesh,metaring}_primitive::compose_operation (0x1a8d20 / 0x1a0d00 / 0x178170), which routes to the per-algorithm composer.

(a) RING — enc_metaring_primitive::__compose_allreduce_channel @0x171600. Its call census (disasm, exact targets) splits cleanly into the two phases:

; reduce-scatter phase — each rank recv-CCE-reduces a chunk into its own, forwards it
171d62:  call 16ad70 <enc_primitive::recv_reduce_send(…, SDMA_CCETYPE, …)>
171d7c:  call 16ad70 <…recv_reduce_send…>
171e09:  call 16aed0 <enc_primitive::recv_reduce_copy_send(…, SDMA_CCETYPE, …)>
171e26:  call 16aed0 <…recv_reduce_copy_send…>
; all-gather phase — fully-reduced chunks rotate WITHOUT reduction
171e7d:  call 16f820 <enc_primitive::direct_recv_send(…)>
171e9e:  call 16f820 <…direct_recv_send…>
171ee8:  call 158540 <enc_primitive::direct_recv(…)>
171eff:  call 158540 <…direct_recv…>

This is the textbook (N−1 reducing + N−1 copy) ring all-reduce: a reduce-scatter that takes SDMA_CCETYPE on every step, followed by an all-gather that does not. [HIGH/OBSERVED — the composer disasm.]

(b) HIERARCHICAL — enc_hier_primitive::__compose_allreduce @0x1a34c0 (and __compose_pipeline_allreduce @0x17b3f0). The page-table builders name the decomposition exactly:

__build_allreduce_pgt_intra_rdsc @0x1558f0   // intra REDUCE-SCATTER
__build_allreduce_pgt_inter_allr @0x14ff80   // inter ALL-REDUCE
__build_allreduce_pgt_intra_allg @0x155260   // intra ALL-GATHER

i.e. HIER ALL_REDUCE = intra reduce-scatter → inter all-reduce → intra all-gather, the five per-leg algos of §6.1. This matches the NCFW phased schedule (PHASE 1 local ring reduce-scatter → PHASE 2 inter-die mesh all-reduce → PHASE 3 ring all-gather broadcast-back). [decomposition HIGH/OBSERVED — builder symbol names; per-leg-algo-at-world-size MED.] See NCFW Hierarchical.

(c) MESH — composer family: __compose_allreduce_full_mesh @0x19ce00 (+ _t0 @0x1595f0 / _t1 @0x185890 two-step), __compose_allreduce_grouped_mesh @0x199ea0, __compose_rdh_allreduce @0x162260 (+_vnc1 @0x163980), __compose_allreduce_trn2 @0x190820, compose_single_step_all_reduce @0x2030e0, compose_one_rank_per_chip_all_reduce @0x2028b0, compose_single_hop_latency_opt_all_reduce @0x202020, compose_single_hop_bandwidth_opt_all_reduce @0x201730; two-step pod-mesh proxy __build_allreduce_redsct_two_step_pod_mesh_proxy @0x112a30. Mesh uses reduce/copy/broadcast handshakes (alg_full_mesh_reduce_cpy, __alg_full_mesh_broadcast, alg_grped_mesh_reduce). [HIGH/OBSERVED — symbols.] See NCFW Mesh.

(d) KANGARING / SINGLE_CYCLE_RING — __compose_allreduce_channel_kangaring @0x1726f0 (direct_reduce_send_kangaring(SDMA_CCETYPE) @0x158120) and __compose_single_cycle_allreduce_channel @0x16cdc0 (all-reduce-only). [HIGH/OBSERVED.] See NCFW Ring + Kangaring.

The per-iteration runtime log exposes the decomposition directly — each all-reduce is split into pipeline slices, each with a copy_slice (all-gather bytes) and a reduce_slice (reduce-scatter bytes), pipelined over channel_n channels:

[nec_dev %d] allreduce: comm(%p) alg(%d) dtype(%d), optype(%d), channel_n(%d)
  pipeline slice_n(%d) priority_class:%d copy_slice_sz:0x%lx reduce_slice_sz:0x%lx

[HIGH/OBSERVED — strings libnrt.so.]

6.3 · EMIT — the cc_op SPAD program + the DMA descriptor packets

The composed steps are emitted into two artifacts:

1. SDMA descriptor packets with the CCE reduce field set — add_dma_packet_cce @0x2307d0 (→ vring_add_dma_packet_cce @0x3134e0) for the reduce steps; al_sdma_m2s_build_cce_ext_meta_ctrl @0x451dc0 / al_sdma_m2s_build_fma_descriptor @0x451c20 for the CCE/FMA arm; CCE buffer/element limits from aws_sdma_get_cce_params_cayman @0x471760 (0x2000 buf / 0x800 elem) and the user CSR offset from aws_hal_get_sdma_cce_user_offset_{cayman,mariana,sunda}. [HIGH/OBSERVED.]

2. A TOP_SP cc_op SPAD command table — create_spad_ctrl_entry @0x232cd0 builds each entry with byte0 bit0 = .cc_op = 1 (the active-cc-op flag), algo_type/algo_sub_type nibbles, and trigger_next. libnrt asserts spad_ctrl->header.cc_op == 1 for an active cc-op (spad_ctrl_cc_op_entry_t size = 7, spad_ctrl_entry_header_t size = 1, structures.json). The per-algorithm config builders (encd_populate_metaring_topsp_config @0x240bf0, encd_populate_mesh_topsp_config @0x240330) chain entries, so a hierarchical all-reduce becomes multiple chained cc_op entries — one per leg. [HIGH/OBSERVED.] See TOP_SP Collective Lowering.

A per-collective record cc_op_info {posted_status, cc_op, op_type, alg, alg_name} is serialised — for ALL_REDUCE: op_type = ENC_ALLREDUCE, cc_op = 1, alg = enc_alg_type — carried via ntff::collectives_op_info (a protobuf into the model profile / NCFW context; the ntff::collectives_op_info::_table_ and parse/merge methods are in the symbol table). [HIGH/OBSERVED — DWARF members + protobuf symbols.]

7 · Triggering the operation — the TOP_SP host-trigger CSR

The lowered DMA leg is triggered by writing the TOP_SP host-trigger CSR. The concrete per-arch offsets are immediate-return register-offset functions:

aws_reg_cayman_get_top_sp_nx_local_reg_host_trigger_offset @0x47b080:  mov $0x615a0,%eax ; ret
aws_reg_mariana_..._host_trigger_offset                    @0x477990:  mov $0x615a0,%eax ; ret
aws_reg_sunda_..._host_trigger_offset                      @0x479120:  mov $0x60848,%eax ; ret

⇒ host-trigger = 0x615a0 (Cayman/Mariana) / 0x60848 (Sunda), companion stop_signal = 0x615c0, basic_block_switch = 0x615e0 (Cayman, the adjacent immediate returns). These are dispatched per-arch through the kaena_khal HAL vtable: aws_hal_sp_topsp_get_host_trigger_reg_offset @0x457ba0 does call al_hal_tpb_get_arch_type ; mov 0x710(%rax),%rax (vtable slot +0x710 is the offset getter; the adjacent aws_hal_sp_topsp_set_host_trigger performs the write through slot +0x708, @0x457b44: mov 0x708(%rax),%rax — see the authoritative split in TOP_SP lowering §3a). A one-shot value-1 write to this CSR is the literal "It then triggers the operation" of the 0xC7 header comment. [HIGH/OBSERVED — immediate-return offsets + the vtable dispatch disasm. This resolves the LOW CSR gap of the 0xC8/0xD9 pages.] See TOP_SP Collective Lowering.

8 · The device leg — SB2SB (0xBF) + the EVT_SEM completion

There are two trigger surfaces: the one-shot NX-program start doorbell (§7), and a per-step DMA-tail doorbell baked into the descriptor ring. encd_get_trigger_addr @0x23f150 resolves the per-step address — it calls encd_arch_get_sp_base_addr (the sp_base 0x8280000000 movabs is at 0x257040) and adds the EVT_SEM SEMAPHORE_INC base (+0x1800) + idx*4. A tail-pointer write hits that address; the TOP_SP polls it (WAIT_FOR_SEM_GE) to advance to the next step. [encd_get_trigger_addr/sp_baseHIGH/OBSERVED; the+0x1800 EVT_SEMinc base CARRIED from the orientation trace, where it is OBSERVED viaaws_hal_get_top_sp_evt_sem_inc_base.]

The concrete intra-node byte movement is the SB2SB collective, SB2SB_COLLECTIVE = 0xBF (common.h:262) — a real (non-pseudo) hardware op run by the POOL/Q7 iDMA kernel. The device ucode decoder decode_sb2sb_collective (in libnrtucode_extisa.so) does the SBUF→SBUF copy with a Pool/Q7 pre-sync handshake — OBSERVED strings:

P%i: Decode : SB2SB_Collective
P%i: SB2SB_Collective : total_src_nelem = %zu, total_dst_nelem = %zu, dtype=%d, dst_nc=%zu, src_nc=%zu, mask=0x%x
P%i: SB2SB_Collective: num_chans = %0d, tpb_idx = %u
P%i: SB2SB Pre-sync: remote_pool_xt_addr=0x%x, remote_q7_xt_addr=0x%x, sb2sb_ready_to_receive_remote=0x%x

[HIGH/OBSERVED.] See S3D3 Collective (SB2SB, 0xBF) and the firmware SB2SB Remote Copy.

NOTE — no trigger pseudo-op on the device. The device decode set is exactly {ExtendedInstCopy, ExtendedInstCptc, ExtendedInstRandGetState, ExtendedInstRandSetState, ExtendedInstTensorTensorArith, GetSequenceBounds, SB2SB_Collective, Sbuf2Sbuf} — no TriggerAllReduce / TriggerCollective among them (P%i: Decode : … strings, libnrtucode_extisa.so). This proves the pseudo ALL_REDUCE is lowered host-side before any device decode; the device only ever sees the lowered SB2SB/DMA legs. [HIGH/OBSERVED.]

9 · The full lowering path — one table

CARRIED — host enc_alg_type ≡ firmware 4-bit algo_type. Stage C reads the host enc_alg_type low nibble as the firmware algo_type; this identity is MED/INFERRED — no byte-level host→device write was traced (the device walk is in the scalar-LX NCFW images, which carry no shipped disassembler config — CAYMAN = NC-v3). [MED/INFERRED; the NCFW dispatch table and the scalar-LX wall are documented on [NCFW Main Dispatch Loop](../ncfw/main-dispatch-loop.md).]

The compiler-side recognition/relocation is documented on The Collective NRT-Load-Time Rewrite; the whole end-to-end thread is on A Collective, End to End.

Reimplementer's checklist

• Encode the dedicated form as a 64 B word: header.opcode = 0xC7, op = ALU_OP reduce, dtype = element type, input_tensor_id/output_tensor_id handles, num_elements (u64). No ctype — the kind is fixed. Use 0xC8 for a group_id + offsets, 0xD9/0xDA for SBUF operands + channel/stream/cc-buffer.
• Map the reduce op onto SDMA_CCETYPE {ADD0, FMA1, MAX2, MIN3} (== ISA CCE_OP); it is the 2nd arg to every recv_reduce_* step. FMA carries an extra scale operand.
• Restrict CCE reduce to {BF16, FP16, FP32R, FP8_E3/E4/E5}; never FP32 (0x0A) or integers — FP32R (0x0B) is the FP32-class entry and has SDMA byte size 0 (it rides the accumulation nibble only).
• Select an algorithm per leg from {RING, KANGARING} (intra reduce-scatter / all-gather) and {RING, INTER_RDH, SINGLE_CYCLE_RING, MESH} (inter all-reduce); SINGLE_CYCLE_RING is all-reduce-only.
• Compose RING as recv_reduce_send/recv_reduce_copy_send (reduce-scatter) + direct_recv_send/direct_recv (all-gather); HIER as intra_rdsc → inter_allr → intra_allg.
• Emit SDMA CCE descriptor packets + a cc_op SPAD table (byte0 .cc_op=1), trigger via host_trigger (0x615a0/0x60848), and let the device run SB2SB (0xBF) legs with the CCE reduce, completing through EVT_SEM (+0x1800).