How to Read This Guide
This is a reimplementation reference, not a tutorial. The intended reader is a senior C++ engineer — comfortable with ELF, LLVM, and DSP/VLIW microarchitecture — who wants to rebuild a Vision-Q7-compatible GPSIMD engine (or a toolchain, runtime, or simulator that targets one). Every page is written so that, after reading it, you could write the C++ that reproduces the described behaviour.
Everything here was recovered by static analysis of shipped, redistributable
binaries, headers, config files, and TIE artifacts — 13 host x86-64 libraries,
29 embedded ELF32-Xtensa device blobs (e_machine = 94), and the cleartext
per-generation arch-ISA headers / core config / TIE database. No runtime trace,
no debugger session, no vendor source. That single fact shapes every convention
below: because the device firmware is symbol-stripped and compiled -fno-rtti
(verified: _ZTS/_ZTI/_ZTV/__cxxabiv1 byte-count = 0 across all 29 device
ELFs), nothing in this guide rests on a recovered class hierarchy. It rests on
anchors you can re-derive yourself — section names, dispatch tables, enum bytes,
config tokens — and every claim is tagged with how strongly the bytes support it.
Read this page once. The four things it gives you — the 16-Part map, the three reading paths, the page anatomy, and the in-text conventions — are the operating manual for every leaf page that follows (Part 0's reference-apparatus pages plus the 401 subsystem pages of Parts 1–16).
1. The 16-Part structure
The body is organised in journey order: a reimplementer reads top to bottom
to rebuild the engine, each Part assuming the ones before it. The full tree lives
in SUMMARY.md; the shape is:
| Part | Title | What it lets you build |
|---|---|---|
| 0 | Reference Apparatus | (this Part) — how to read, trust, and re-verify everything else |
| 1 | Orientation | The one-screen mental model: what GPSIMD is, the seven "faces" of the one core, the facts people get wrong |
| 2 | Q7 Core & ISA Foundations | The Cairo / ncore2gp core identity, the FLIX encoding, the eight register files, the libisa decode model |
| 3 | Per-Instruction ISA Reference | All 1534 Vision-Q7 opcodes in 30 batches + the formal operational semantics |
| 4 | Microarchitecture & Timing | Pipeline, co-issue matrix, register-file ports, boot/reset, the scalar-LX management core |
| 5 | Device Firmware & Kernel Catalog | The SEQ front-end, POOL dispatch, DGE, and the 140-opcode kernel catalog |
| 6 | Firmware Images & Generations | The per-(generation × engine) device images and how they differ across Sunda → Maverick |
| 7 | Custom-Op ABI | The at::Tensor chain, the customop_* marshalling, the build/link/strip flow |
| 8 | Host Runtime | libnrt / nrtucode — bring-up, dispatch, the 8-core SPMD model, the prelinker |
| 9 | DMA / Descriptors / Memory | Descriptor model, gather/scatter, RDMA, SBUF/PSUM banks, the al_udma engine |
| 10 | Collectives & NCFW | The collective opcodes and the NX-CoreFirmware management core that orchestrates them |
| 11 | NEFF Container Format | The on-disk container, carved byte-by-byte, and its relationship to ELF |
| 12 | Compiler Seam | The neuronx-cc → opcode lowering, the BIR instruction roster, the MX path |
| 13 | Control Plane | CSR / address map / interrupt / security — the SoC plumbing around the core |
| 14 | ISS as Executable Oracle | libcas / libfiss — the cycle-accurate and fast simulators, used as a value oracle |
| 15 | Validation & Verification | The four-oracle bit-exact differential method and per-family pass/fail ledger |
| 16 | Appendices | Struct census, the master opcode↔kernel↔engine matrix, the coverage ledger, the checklist |
Two structural rules are worth internalising before you start:
SUMMARY.mdis the contract. Parts 1–16 are committed as a roadmap; a line becomes a live link the moment its page ships. If a path you expect is still in the roadmap comment block, the page is authored but not yet wired — checkSUMMARY.mdfor the canonical path.- The core is gen-invariant; the firmware is not. For the four shipped
generations (Sunda / Cayman / Mariana / Mariana+) there is exactly one Xtensa
core config — one
core-isa.h, oneConfigName Xm_ncore2gp, oneuarchName "Cairo". Parts 2–4 describe that one core. What changes per generation is the SoC, the NCFW firmware, and the per-engine TPB tensor-ISA headers — which is why Parts 5, 6, and 13 are generation-aware and Parts 2–4 are not. Maverick (v5) is the one exception and is flagged everywhere it matters (see Part 6 and the Codename Cross-Walk).
2. Three reading paths
Path A — the reimplementer (front to back)
Read in journey order. Part 0 → 1 → 2 → … → 16. Each Part hands the next its vocabulary: Part 2's FLIX encoding is what Part 3's per-instruction pages decode; Part 5's kernel catalog is what Part 7's ABI dispatches into; Part 12's compiler seam is what produces the NEFF that Part 11 describes. If you read nothing else first, read Part 1 · What GPSIMD Is and Part 1 · Keystone Facts Reimplementers Get Wrong — they inoculate you against the mistakes a reimplementer makes most often.
Path B — "I just need opcode X" (lookup)
You have an opcode byte (or a kernel name, or a NEURON_ISA_TPB_* enum) and want
its behaviour. Start at one of the two master indexes, then jump:
- Appendix · Opcode ↔ Kernel ↔ Engine Matrix — the one-row-per-opcode table: byte → kernel page → engine → struct.
- From the matrix row, jump to the Part 5 kernel page (e.g. opcode
0x68GATHER→firmware/kernels/indirection-gather.md) for the device handler, and to the matching Part 3 batch page (e.g.isa/ref/b19-scatter-gather.md) for the underlying Vision-Q7 instructions. - For the semantics of the underlying instruction, the Part 14 ISS pages are the executable oracle — they are the ground truth a differential test runs against.
- For encoding, the Master ISA Encoding Appendix and Part 2 · The FLIX VLIW Encoding.
The opcode catalog ledger — firmware/kernels/opcode-catalog-ledger.md
— is the authoritative list of the 140 real hardware opcodes (the 172-value enum
union minus 31 compiler-pseudo opcodes and 1 invalid).
Path C — per-generation (silicon-targeted)
You are targeting one specific silicon and want only what differs. Anchor on the Codename ↔ Generation Cross-Walk to fix your coordinates, then read the per-generation slice:
- the generation's image pages in Part 6 (e.g. for Trn2 →
images/cayman-pool.mdand siblings), - the Master Per-Generation Capability Matrix and the Cross-Gen Opcode-Table Diff,
- and, for the control plane, the gen-scaled CSR/address pages in Part 13.
The product coordinates, pinned to binary evidence, are: Sunda = NC-v2 = Trn1 / Inf2; Cayman = NC-v3 = Trn2; Mariana = NC-v4 = Trn3; Mariana+ = NC-v4+ = Trn3-pre; Maverick = NC-v5 (firmware-internal-only, unshipped on the public runtime path). Do not invent a Trn4 for Maverick — that binding is open. The cross-walk page carries the full coretype/arch_id stride table and the caveats.
3. Page anatomy
Most subsystem pages follow the same skeleton, so you can skim predictably:
- Opening prose — what the subsystem does, why it exists, where it sits in the pipeline.
- Key-facts table — the addresses, sizes, struct/enum names, and knobs a reimplementer copies first.
- Algorithm — mixed prose + C-style pseudocode for the core logic.
- Data-structure layouts — byte offsets, field types, sizes (the field-exact struct census lives in Part 16).
- Configuration / knobs — defaults and effects.
- Diagnostic strings — what the binary prints, and when (these are recovery anchors as much as documentation).
- Function / address map — identity table for the key routines.
- Cross-references — to the related pages.
Not every page needs all eight; a 30-instruction batch page is mostly (2)+(3)+(4), an image page is mostly (2)+(7)+(8). The constant is that prose carries the why and reasoning; tables and code carry the what.
4. In-text conventions
These conventions are used on every page. Learn them once.
4.1 Confidence tags — <level> / <evidence>
Every non-obvious claim carries two orthogonal tags, written together as
HIGH/OBSERVED, MED/INFERRED, and so on.
Confidence level — how strongly the bytes support the claim:
| Level | Meaning |
|---|---|
| HIGH | Multi-source or byte-exact. Re-derivable from the shipped artifacts. |
| MED | Strong inference from solid evidence, but not byte-pinned end to end. |
| LOW | Plausible, explicitly flagged. Do not build on it without re-verifying. |
Evidence kind — where the claim comes from:
| Kind | Meaning |
|---|---|
| OBSERVED | Read directly from a shipped artifact: a byte, a section, a struct compiled with offsetof/sizeof, a config token. |
| INFERRED | Reasoned over OBSERVED facts. |
| CARRIED | Re-used from an upstream finding at its original confidence — never inflated. |
The cardinal discipline: a synthesis never raises a claim's confidence. A
MED/INFERRED fact stays MED/INFERRED when a later page cites it. The full
rationale, the worked examples, and the "what counts as OBSERVED" rules are in
The Confidence & Walls Model.
4.2 Callout markers
Inline markers flag the four things you most need to not miss:
- QUIRK — a genuine hardware/firmware oddity you must reproduce, even if it
looks like a bug (e.g. the Q7_POOL reset vector shifting from
j 0x200toj 0x1e4on Maverick). - GOTCHA — a recovery or reimplementation trap that has bitten before (e.g. objdump's hex column is byte-reversed versus the memory stream — feed a FLIX decoder the real little-endian bytes).
- NOTE — context that aids understanding but isn't a hazard.
- CORRECTION — a place where an earlier or naive reading was wrong, stating the right value. The wiki always states the corrected value and never the superseded one.
4.3 How pseudocode names real symbols
C/pseudocode blocks describe real recovered routines and structures, not illustrative inventions. The naming rules:
- A function the firmware actually contains is named by its recovered identity —
a demangled
.xt.propsection name (the device images ship per-function property tables whose names are the C++ mangled symbols, e.g..xt.prop._Z22cross_lane_reduce_implb→cross_lane_reduce_impl), akernel_info_tableself-name, or a DEBUG-build"S:<Name>"/"P%i:"trace string. Where only an address is known, it appears as a virtual address (sub_<vaddr>orfunc@0x…) and is so marked. - Structs and enums use their header names verbatim —
NEURON_ISA_TPB_DTYPE,S2D2_TS_AS_STRUCT,kernel_info_table— because these ship as cleartext source and were compile-verified, not guessed. - Pseudocode for a desynced span (one objdump rendered as
.byte) is labelled as reconstructed and tagged at the confidence the FLIX-decode achieved — never presented as a clean disassembly it isn't.
4.4 How claims anchor
Every factual claim is anchored to something a reader can independently check. Anchors come in these flavours, and you will see them inline throughout:
| Anchor | Example |
|---|---|
| Address / offset | .text @ 0x01000000 (IRAM); .rodata + kernel_info_table @ 0x02000000 (DRAM) |
| Symbol / section | .xt.prop._Z24get_sequence_bounds_implj20NEURON_ISA_TPB_DTYPE |
| Enum value | NEURON_ISA_TPB_DTYPE — 16 codes, byte-identical across Sunda and Cayman |
| Opcode byte | 0x72 = COPY_PREDICATED, DVE-native (the base of the predicated-op family) |
| String / config token | XCHAL_VISION_TYPE = 7, vq7_isa = 1 in core-isa.h |
4.5 A worked example — what one anchored, tagged claim looks like
Here is a real claim from the kernel catalog, written the way pages write them:
The
kernel_info_tableis aPROGBITSsection present in every device image: an array of 8-byte entries, each[ BE opcode-key : u32 ][ LE funcVA : u32 ]. ThefuncVAis the only relocated field — oneR_XTENSA_RELATIVEper entry, stride 8 — which independently proves the 8-byte stride, the funcVA position, and the entry count (17 for Cayman). Every funcVA validates onto a real function prologue; several land directly on a named.xt.propfunction.[HIGH/OBSERVED]
Read it as: the structural facts (stride, field layout, count) are OBSERVED
from the relocation table and HIGH because the relocation count is the entry
count — a self-checking anchor. Contrast a claim like "Maverick targets HW rev
NX1.1.4", which is tagged [INFERRED] because Maverick's core config does not
ship — its e_flags 0x300 merely suggests core-family continuity. The tag is
not decoration; it tells you exactly how far you can lean on the sentence.
5. The companion Part-0 pages
This page is the index to Part 0. The other front-matter pages each expand one dimension of "how to trust and re-verify the body," and every subsystem page links back here for its provenance framing:
- The Confidence & Walls Model — the
HIGH/MED/LOW × OBSERVED/INFERRED/CARRIEDdiscipline in full, plus the walls: the true static-analysis boundaries (the FLIX-desync ceiling, the FLAT-NX interior, the out-of-corpus and host-supplied content) that no page may silently cross. - Methodology — How This Was Reverse-Engineered — the eight
recovery techniques: getter-driven ELF carving, the native
ncore2gpdisassembler, arch-ISA header compile-verify (offsetof/sizeofproofs),kernel_info_table/ self-name anchoring, cross-generation byte-diffing, four-source ISA triangulation, the confidence-and-correction culture, and the codified corpus gotchas. - FLIX Bundle-Decoding Methodology — the Vision-Q7 multi-slot decoder (a faithful port of the binutils libisa pipeline; 14 formats / 46 slots / 1534 opcodes; validated at 549,375 per-slot mnemonics, 100% agreement against the device-native oracle) — and the scalar-LX vs FLIX rule that prevents the phantom-FLIX mis-decode of the NCFW management core.
- The Corpus, Tiers & Binary Inventory — the substrate: 13 host libraries, 29 embedded device ELFs, the shipped headers/config/TIE, with sha256 anchors and the per-image carve manifest.
- Toolchain Inventory & Versions — the three independent version axes that are routinely conflated: toolchain (14.09 / RI-2022.9 / clang-10 for the four shipped gens, 15.05 / clang-15 for Maverick), package (0.21.2.0 / ucode 1.21.1.0), and HW revision (NX1.1.4 = 281040 = RI-2020.4 = LX7.1.4, MIN == MAX).
- Codename ↔ Generation Cross-Walk — the definitive codename ↔ NC-version ↔ coretype ↔ arch_id ↔ product table, each cell pinned to a binary surface, with the v5-label-overload hazard called out.
- Master Glossary — every term a reimplementer meets, each
grounded in its anchor (Cairo,
ncore2gp, NX1.1.4, Xtensa24, FLIX, FLIX-desync, scalar-LX, NCFW, TPB, the engineengine_idxnumbering, RTTI,.xt.prop,kernel_info_table, the HW-Decode / Sunda-mode dispatch modes, TIE, ISS).
Where to start
If you read three pages before the body, read these:
- Part 1 · What GPSIMD Is — the one-screen map
- Part 1 · Keystone Facts Reimplementers Get Wrong
- The Confidence & Walls Model — so every tag in every subsequent page reads correctly.
6. The compiler seam — cross-references to the neuronx-cc wiki
GPSIMD does not exist in isolation: a custom op is compiled by neuronx-cc into
a NEFF, then dispatched by the host runtime onto the eight Xtensa cores. This
wiki documents the device + runtime half; the compiler half — how the
opcodes documented here are emitted — lives in the sibling
neuronx-cc wiki, whose Part 11 —
Custom Ops & GPSIMD is the matching seam. The cross-references, both directions:
| For… | This wiki | The neuronx-cc wiki |
|---|---|---|
| The 8-core Xtensa ELF layout | Part 7 · Custom-Op ABI, Part 8 · SPMD model | custom-ops/gpsimd-xtensa-layout.md |
| The custom-op CPU ABI | Part 7 · Complete Custom-Op ABI | custom-ops/customop-cpu-abi.md |
| How opcodes are emitted | Part 12 · Compiler Map (emit_*→opcode) | isa/collective-customop-encoding.md, nki/neuroncodegen-builtin-customop.md |
| The SORT / TOPK builtin | Part 5 · Sort / DECODE_SORT | custom-ops/bitonic-sort-topk.md |
The dedicated bridge page is Part 12 · Compiler Cross-Reference to the
neuronx-cc Wiki. One discipline to carry
across the seam: the NKI/compiler engine enum (tensor=1, scalar=2, gpsimd=3, dma=4, vector=5, sync=6) is a different integer space from the firmware
engine_idx (PE 0, ACT 1, POOL 2, DVE 3, SP 4, TOP_SP 5) used throughout this
wiki — do not equate the two. [HIGH/OBSERVED]
Provenance. Every fact in this guide derives solely from static analysis of shipped, redistributable binaries, headers, config files, and TIE artifacts — lawful interoperability reverse engineering under DMCA 17 U.S.C. § 1201(f). No vendor source tree was referenced, consulted, or quoted; everything reads as derived from binary, header, config, and TIE analysis alone.