The ext-ISA Provider (nrtucode_* C-API)

All host-binary addresses on this page apply to libnrtucode_extisa.so from aws-neuronx-runtime-lib 2.31.24.0-0b044f4ce (build-id 7bb03bc42ce1530924a1797ec9d5e518a7ae5e44, ELF64 x86-64 DYN, stripped, 9,656,488 B; .rodata VMA == file offset, so every 0x… host offset is also an analysis VMA). Other versions will differ. Evidence grade: Confirmed (byte-anchored) — the 52 exports (nm -D, twice), the API-level-3 gate, the four handle structs, the device mailbox map, the claim/boot magics, and the 0x40-byte load record are each re-derived from the objdump/xxd of the cited functions, cross-checked against the IDA Hex-Rays decompile. · Part XI — GPSIMD / Q7 Microcode & ISA · back to index

Abstract

libnrtucode_extisa.so is the host-side microcode provider for the Neuron GPSIMD "Q7" pool engine — a self-contained library that ships device code and hands it to the runtime through a stable C-API. It is not a driver and never touches the device: it embeds the device programs (13 Tensilica Xtensa ELF32 microcode blobs plus one SUNDA opcode-manifest JSON — owned by Q7 Blobs), an index/prelink/serialize layer over them (owned by Dispatch Tables and Microcode Loader), and exposes that whole container behind exactly 52 exported nrtucode_* functions (nm -D, all type T, zero other exports; imports are libc only — no dlopen, no Neuron deps). Think of it the way a JIT ships a code cache as a versioned .so: the engine is fixed silicon, the programs are the cataloged pool kernels, and this library is the container + a stable ABI that selects the right blob for the running silicon and the requested opcode set.

This page owns the API surface and the data model that surface threads. The provider is a small object graph: a context (the host↔device accessor + scratch), a per-engine core (the bring-up handle that owns the device CSR mailbox), an ll low-level loader (one device push record per load/unload), and an opset (the compiler-side opcode-set builder). The single most important contract is the API-level-3 handshake: nrtucode_get_api_level() returns the constant 3 (0xaec0: mov $0x3,%eax; ret), and nrtucode_context_create rejects any caller whose declared level ≠ 3 with status 4 VERSION. The consumer — libnrt.so's tdrv/ucode.c (ucode-facade) — dlopens this library, dlsyms a fixed 30-entry subset of the 52, re-checks the level, then drives init → bring-up → load → query → teardown. The other 22 exports (the opset_* family, the _private_*/getter extras) serve the compiler/assembler/disassembler.

The page is four structural units: the export catalog (§1) grouped by the five families (get/context/core/ll/opset = 7/5/22/9/9 = 52), each pinned to symbol+addr; the handle objects + device mailbox map (§2) — the four struct layouts and the core->a4 CSR region the bring-up path drives; a worked export (§3) — the claim/boot handshake nrtucode_core_on_ucode_booted as annotated pseudocode, plus the 0x40-byte load record nrtucode_ll_get_load_sequence emits; and the boundary (§4) — what this page does not re-derive (the RELA relocator + Xtensa encoder are microcode-loader; the blob inventory is q7-blobs; the arch-id routing is dispatch-tables).

For reimplementation, the contract is:

• The 52-export surface, grouped and addressed — five families, exactly 52 nrtucode_* symbols and nothing else; the 30 the runtime dlsyms vs the 22 compiler-side extras.
• The API-level-3 gate — get_api_level → 3; context_create rejects api_level ≠ 3 with 4 VERSION; the consumer re-checks at dlopen.
• The four handle objects — context (0x28), core (0x70), ll (0x48), opset (0x830) — their sizes, key field offsets, and constructors.
• The device CSR mailbox map at core->a4, driven through a caller-supplied rw_impl vtable: claim/boot magic, log ring, DGE priority map, pc-bounds — and the claim handshake magics.
• The 0x40-byte load/unload push record (tag 0x1095) emitted into a caller buffer, the one observable runtime output of the loader.

1. The Export Catalog — 52 nrtucode_* by Role

Purpose

The provider's entire ABI is exactly 52 nrtucode_* functions (nm -D --defined-only, type T, confirmed twice) and nothing else — no C++ symbols, no internal helpers exported, no Neuron-runtime entry points. They partition cleanly into five families by role. A reimplementer reproduces this surface family-by-family; the table below is the symbol+addr anchor for each, grouped by role with the runtime/compiler split called out. The per-export semantics that need more than a line — the claim handshake and the load record — are §3; the loader internals each ll_* drives are microcode-loader.

Family map

get_ family (7) — provider query and image fetch

context_ family (5) — context lifecycle

core_ family (22) — bring-up, logs, DGE, pc-bounds

ll_ family (9) — low-level loader

opset_ family (9) — opcode-set build + query (compiler/assembler)

NOTE — the opset_* family is the producer-side API: the compiler/assembler builds an opcode set (opset_create → opset_add_instruction per kernel instruction) and maps it to a microcode library (opset_get_library_index). The runtime never calls it — which is why the consumer's 30-entry dlsym table is a strict subset of the 52 exports. The 22 unused-by-runtime extras are the 9 opset_*, plus get_hwdecode_table, the core_{get,set}_userdata/get_coretype/set_max_loglevel/enable_logs_with_size_hint/dge_get_priority_class_map/private_* getters, and the ll_get_library_size/get_{load,unload}_sequence_num_instrs getters. CONFIDENCE: HIGH (each enumerated in the decompile). The runtime↔provider dlsym binding (which 30, in what order) is the ucode.c facade.

Status and result codes

Every nrtucode_* that returns a result yields nrtucode_result_t:

2. Handle Objects and the Device Mailbox

Purpose

The 52 exports thread four opaque handles. A reimplementer must reproduce their sizes (the malloc constants are exact), the field offsets the API reads/writes, and — the part that actually reaches silicon — the device CSR mailbox the core_* bring-up path drives through a caller-supplied accessor. The provider itself performs no device I/O; every host↔device transaction goes through the rw_impl vtable installed at context_create.

The four structs

struct nrtucode_context_t {           // malloc 0x28 @ context_create 0xab10
    void*  rw_impl;          // +0x00  host↔device accessor vtable (see below)
    void*  memhandle_impl;   // +0x08  set via context_set_memhandle_impl
    void*  userdata;         // +0x10  get/set_userdata
    u64    scratch_capacity; // +0x18  = 0x200 (movq $0x200,0x18(%r15) @0xab81)
    void*  scratch;          // +0x20  = malloc(0x200)
};

struct nrtucode_core_t {              // malloc 0x70 @ core_create 0x99e0
    nrtucode_context_t* context;  // +0x00
    void*  userdata;              // +0x08
    u32    kind;                  // +0x10  NRTUCODE_CORE_*_NX_POOL, scheme B {2,9,17,25}
    u64    a3;                    // +0x18  pass-through (LOW: likely TPB-idx)
    u64    a4;                    // +0x20  device mailbox/CSR base ("mailbox" below)
    u64    a5;                    // +0x28  pass-through device handle (LOW)
    u32    boot_state;           // +0x30  0=unbooted, 1=BOOTED_LEGACY (after on_ucode_booted)
    u64    logbuf_dev_ptr;       // +0x38  0 until enable_logs
    u32    logbuf_size;          // +0x40
    u32    log_tail_read_idx;    // +0x44
    char   friendly_name[0x21];  // +0x48  "nrtucode_core_t@%p"
};

struct nrtucode_ll_t {                // malloc 0x48 @ ll_create 0x8ef0
    nrtucode_context_t* context;  // +0x00
    u64    dram_alloc;           // +0x08  device buffer (rw-freed at destroy if +0x18!=0)
    u32    iram_target;          // +0x10  device/IRAM target addr → record +0x18
    u64    library_size;         // +0x18  get_library_size → record +0x1c (low dword)
    char   friendly_name[0x21];  // +0x20  "nrtucode_ll_t@%p"
};

struct nrtucode_opset_t {             // malloc 0x830 @ opset_create 0x8840
    nrtucode_context_t* context;  // +0x000
    void*  opcode_slots[256];    // +0x008  slot[op]=ptr to 256-byte specialization bitmap | NULL
    char   friendly_name[0x21];  // +0x808  "nrtucode_opset_t@%p"
};

QUIRK — the engine is named by two distinct coretype numbering schemes that index different machinery: scheme A "ext-isa coretype" {6,13,21,29} (bitmask 0x20202040, used by get_ext_isa/get_num_ext_isa_libs/ll_create/opset_get_library_index) is a function argument; scheme B "core->kind" NRTUCODE_CORE_*_NX_POOL = {2,9,17,25} (bitmask 0x02020204, validated by the DGE/pc-bounds gates) is stored at core+0x10. They are not the same number for the same silicon — get_num_ext_isa_libs does add $0xfffffffa (i.e. coretype-6) into a jump table, while the DGE gate masks {2,9,17,25}. Confusing them is the single easiest mistake here. The full arch_id → coretype → lib-count routing is dispatch-tables. CONFIDENCE: HIGH (both bitmasks read from disasm).

The rw_impl accessor vtable

context_create's first non-null argument is the host↔device memory accessor. The provider invokes it by call-offset; the slot semantics are inferred from the call sites (the concrete implementation lives in libnrt's tdrv, not here):

The names are not in this binary; they are reconstructed from the read (on_ucode_booted @0x9e7d), write (on_ucode_booted @0x9eeb, core_destroy @0x9b67), alloc (log-ring), and map (load record) call patterns. CONFIDENCE: MED (call-offset pattern; concrete impl in libnrt — see ucode-facade).

The device CSR mailbox map

All offsets are from core->a4 = *(u64*)(core+0x20), accessed only through rw_impl:

CONFIDENCE: HIGH — every offset cross-checked across the log/DGE/pc-bounds functions; the claim magic and its +0 location are §3.

3. Worked Export — the Claim Handshake and the Load Record

Purpose

The two exports a reimplementer most needs to get byte-exact are nrtucode_core_on_ucode_booted (the device-ownership handshake that gates every later core_* call) and nrtucode_ll_get_load_sequence (the one observable runtime output of the whole stack — a single device push record). Both are short, both are fully decoded, and both carry a correction against the earlier scaffold.

nrtucode_core_on_ucode_booted — the claim handshake

Entry Point

nrtucode_core_on_ucode_booted (0x9e50)
  ├─ rw_impl->read (call *(%rax))        @0x9e7d   ── read claim magic from mailbox+0
  ├─ cmp $0x6099cb34   (UNCLAIMED)       @0x9e8e   ── fresh boot → proceed
  ├─ cmp $0x502b2da1   (CLAIMED)         @0x9e97   ── already claimed → log + status 8
  └─ rw_impl->write (call *0x8(%rax))    @0x9eeb   ── write CLAIMED; set boot_state=1

Algorithm

// nrtucode_core_on_ucode_booted(core)            // sub @0x9e50
function on_ucode_booted(core):
    if core == NULL:                               // @0x9e55
        fprintf(stderr, "nrtucode: invalid API usage in `%s`: `%s` is null"); abort()

    magic = 0
    rw = core->context->rw_impl                    // (*core)
    rw->read(rw, /*len=*/4, &magic, core->a4)      // call *(%rax)  read mailbox+0  @0x9e7d
    if rc != 0: return rc                           // IO

    if magic == 0x6099CB34:                         // UNCLAIMED  (cmp @0x9e8e)
        magic = 0x502B2DA1                          // CLAIM      (movl @0x9ecf)
        rw->write(rw, 4, &magic, core->a4)          // call *0x8(%rax)  write mailbox+0  @0x9eeb
        if rc != 0: return rc                       // IO
        core->boot_state = 1                        // BOOTED_LEGACY (movl $1,0x30 @0x9ef2)
        return 0                                     // SUCCESS

    if magic == 0x502B2DA1:                         // already CLAIMED by another core (@0x9e97)
        log("Core is claimed by another nrtucode_core_t instance")
        return 8                                     // INVALID
    // any other value = a foreign / incompatible booted image
    log("Magic value `%x` mismatch (booted image is incompatible)")   // @0x9f01
    return 8                                         // INVALID

core_destroy (0x9b20) is the symmetric release: if boot_state != 0 it writes the UNCLAIMED magic back to the same mailbox slot (movl $0x6099cb34,0xc(%rsp) @0x9b4b, then rw->write(…, core->a4) @0x9b67) before free(core).

CORRECTION (UC-API) — the earlier scaffold (P1-UC-API §2/§3, echoed by overview §2) stated the UNCLAIMED/unclaim magic is 0x60969274. The binary truth is 0x6099CB34: cmp $0x6099cb34,%r9d @0x9e8e in on_ucode_booted and movl $0x6099cb34,0xc(%rsp) @0x9b4b in core_destroy, both byte-exact. 0x502B2DA1 (CLAIMED) is correct as documented. A reimplementer that boots the firmware with 0x60969274 as the "unclaimed" sentinel will fail the cmp at 0x9e8e and every claim will fall through to the "booted image is incompatible" path. CONFIDENCE: HIGH (immediate read directly from disasm, two independent call sites).

CORRECTION (UC-API) — the same overview §2 GOTCHA claims core_destroy writes the unclaim magic to mailbox+4 (the log-bufsize slot), not mailbox+0. The disasm shows the rw-write address argument is mov 0x20(%rbx),%rsi = core->a4 (mailbox+0) with no +4 displacement (0x9b56); the claim read in on_ucode_booted uses the identical mov 0x20(%rbx),%rsi (0x9e6c). Both claim and unclaim target mailbox+0. CONFIDENCE: HIGH (address operand has no displacement in either call site).

nrtucode_ll_get_load_sequence — the 0x40-byte push record

Entry Point

nrtucode_ll_get_load_sequence  (0x93f0)   ── mov $1,%edx (seq_type=load) ; jmp 0x9400
nrtucode_ll_get_unload_sequence(0x9880)   ── mov $2,%edx (seq_type=unload); jmp 0x9400
  └─ sub_9400                              ── shared emitter; ONE 0x40-byte record

Algorithm

Both sequence getters tail-jump into the shared emitter sub_9400 with seq_type pre-loaded (mov $0x1,%edx @0x93f6 for load, mov $0x2,%edx @0x9886 for unload). The buffer must be 8-byte aligned (else 8 INVALID). Exactly one record is emitted; the 8-slot unrolled body further into sub_9400 is dead code, reached only on the ll->context != core->context error path.

// sub_9400(ll, core, seq_type, instr_buf, a3, &num_pushed)
struct ucode_push_record {            // 0x40 bytes — byte-exact from sub_9400 disasm
    u16 tag;          // +0x00  = 0x1095   "write block" command (movw $0x1095 @0x9537)
    u8  pad[10];      // +0x02  = 0
    u8  seq_type;     // +0x0c  = 1 (load) | 2 (unload)
    u8  flag;         // +0x0d  = 0xFF     (movb $0xff,0xd(%rsi) @0x9566)
    u16 rsvd;         // +0x0e  = 0
    u64 payload_ptr;  // +0x10  = rw_impl->map(ctx, ll->dram_alloc) for load; 0 for unload
                      //           (mov %rax,0x10(%rsi) @0x9554)
    u32 iram_target;  // +0x18  = ll->iram_target  (mov 0x10(%r15),%edi → 0x18(%rsi) @0x9533/0x956a)
    u32 payload_len;  // +0x1c  = (u32)ll->library_size (mov 0x18(%r15),%ecx → 0x1c(%rsi) @0x9516/0x9558)
    u8  zero[32];     // +0x20  = 0
};
// *num_pushed = 1;  return 0;

This record, written through rw_impl->write, is what installs the relocated microcode body into device IRAM. The relocation that produces the body the record points at — and the "UCPL " device header that frames it — are microcode-loader. CONFIDENCE: HIGH (every field offset and source register read from the sub_9400 disasm).

Function Map

Considerations

The claim handshake is read-modify-write, not a blind store: a second core on the same engine reads CLAIMED and bails with status 8, and a foreign booted image (any magic that is neither sentinel) is rejected rather than silently overwritten. A reimplementer must keep the three-way branch — collapsing it to "write CLAIMED unconditionally" defeats the multi-instance and image-compatibility checks the firmware relies on. The load record's payload_len is read as a 32-bit field from ll->library_size (mov 0x18(%r15),%ecx, a dword move), even though library_size is stored as a u64; a library over 0x10000 is already rejected at ll_create (microcode-loader), so the high dword is always zero on the emit path.

4. Boundary — What This Page Does Not Own

Purpose

The provider is a container with an index, a loader, and a blob store; this page owns only the API surface and the data model. Three adjacent concerns are owned by sibling pages, and a reimplementer should read this page for the what and those for the how.

The boundary

NOTE — the provider's three internal subsystems are each a sibling page; this page links them and does not re-derive their byte-level internals:

• The RELA relocation + Xtensa immediate encoding + "UCPL " device header — everything ll_create invokes between blob-select and the staged image (sub_8380/sub_74E0/sub_79E0, the reloc-type taxonomy, the eight FLIX slot signatures, the geometry descriptor) — is owned by The Microcode Loader. This page stops at the ll handle's exported surface and the 0x40-byte record it emits.
• The blob inventory — the 13 embedded Xtensa ELF32 images (9 distinct), the SUNDA opcode manifest, and the on-device kernel_info_table opcode→entry format the firmware uses after the record installs the body — is owned by The 13 Q7 Blobs.
• The arch_id → coretype → lib-count routing behind get_ext_isa/get_num_ext_isa_libs/ll_get_libraries_from_opcodes (the arch_id-6 jump tables, the per-arch handler tables T0..T3, both coretype schemes in full) is owned by Dispatch Tables.

The runtime consumer that dlopens this library and binds the 30-of-52 subset is The ucode.c Facade. CONFIDENCE: CERTAIN (each subsystem's owning cell is named in the Part-XI map).

Related Components

Cross-References

• Overview: the Q7 GPSIMD Engine — the provider→loader→device map and the 52-export role groups
• The Microcode Loader (RELA + FLIX, UCPL) — the relocation/encoding internals behind ll_create, not re-derived here
• ext-ISA Dispatch Tables and arch-id Scheme — arch_id → coretype → lib-count; the two coretype schemes in full
• The ucode.c dlopen Facade — the runtime's 30-of-52 dlsym binding and the api-level handshake
• The 13 Q7 Microcode Blobs — the embedded images this provider's get_*/ll_* exports serve
• back to index