Globals and Singletons Atlas

Binary: extracted/aws-neuronx-runtime-lib_2.31.24.0-0b044f4ce_amd64/opt/aws/neuron/lib/libnrt.so → libnrt.so.1 → libnrt.so.2.31.24.0 · Version 2.31.24.0-0b044f4ce (.nrt_brazil_version @0xad41f0 = "2.31.24.0") · BuildID[sha1] 8bb57aba0fb2e0035f1d88e9fc4fb3e7387c102e · ELF64 LSB DYN x86-64, NOT stripped, DWARF v4.

Part II — Binary Anatomy & Forensics / REFERENCE atlas · Evidence grade: every address/size below is read from this exact binary — nm -nS --defined-only for the symbol address + size + section nibble; readelf -SW for the section bounds; readelf -rW for the .data.rel.ro R_X86_64_RELATIVE relocations that wire each function-pointer / string-pointer slot; and the IDA decompiled sidecars under ida/**/decompiled/ for the init-by / read-by call graph. · back to forensics index

Abstract

This is the map of libnrt.so's writable global state — every standing singleton, per-arch vtable, and dlopen handle table that lives across the three writable data sections, with provenance for who initializes it and who reads it. The runtime is a single 117 MB C++/Rust shared object, but its process-global control plane is small and named: a nrt_config config singleton, a heap nrt_global_config_t reached through the ngc pointer, two coexisting per-arch HAL vtables (kaena_khal, tdrv_arch_ops), two dlsym-bound external-library function tables (the libnrtucode microcode loader and the libnccom collective-network plugin), a clutch of error/status globals, and a set of compiler-emitted dispatch/jump tables. Everything else in the writable sections is statically-vendored third-party state (protobuf descriptor pools, abseil, Rust std, simdjson, libarchive), flagged here as boundary context rather than re-derived.

The atlas separates the four writable sections by what they hold. .data.rel.ro (0xbf2d80–0xc05540, 0x127c0 B) is RELRO: read-only after the loader applies relocations, so it holds the function-pointer and string-pointer tables that must be immutable at runtime — the ucode_func_symbols dlsym directory, the per-arch *_hw_port_names arrays, the *_instr_block_caps const structs, and the C++ vtables. .data (0xc07e00–0xc37c20, 0x2fe20 B) holds const-valued dispatch tables that are not pointer arrays — nrt_init_state_strings, the queue-bundle allocation tables, sequencer_error_subtypes_str, the all_reduce_128b blob. .bss (0xc37c40–0xcb4300, 0x7c6c0 B) is NOBITS — zero on disk — and holds every mutable singleton and per-engine array: nrt_config, ngc, kaena_khal, tdrv_arch_ops, the vcore tables, the async-send/recv contexts, the proxy-queue map, the resolved dlsym function-pointer stores, and the error-log ring. The single most consequential structural fact for a reimplementer is that the per-arch vtables are .bss symbols — they contain no bytes in the file; their slot contents exist only as stores inside the lazily-run register_funcs / register_* installers, and so cannot be hexdumped.

This page groups the atlas by kind — config singletons, per-arch ops vtables, dlopen handle tables, error/status globals, dispatch/jump tables — and anchors each entry to its nm symbol and the function that fills the zero bytes. The per-slot geometry of the big subsystem structs is owned by the subsystem pages cross-linked at the foot; the atlas records the writable-global view (address, size, type, init-by, read-by, section) rather than re-deriving their internals.

For reimplementation, the contract is:

• The section discipline — RELRO for immutable function/string-pointer tables, .data for const-valued tables, .bss for mutable singletons and the lazily-filled per-arch vtables. A port that places a per-arch vtable in .data.rel.ro and tries to relocate it statically will not match: these tables are written after RELRO is applied, by code, at device bring-up.
• The init-by / read-by provenance per global — which is built by a static constructor (Static-Init), which is installed lazily at nrt_init / device bring-up, and which is filled on demand. The atlas tags every entry with its installer function.
• The two-config split — the static nrt_config singleton (0xc5c480, 656 B) versus the heap nrt_global_config_t reached via ngc (0xc5c460, 296 B). They are distinct objects with distinct lifetimes; both are populated from env/NEFF at init, neither is built complete by a constructor.
• The dlsym table shape — an immutable .data.rel.ro name/slot directory (ucode_func_symbols) paired with a .bss block of resolved function pointers a loader stub fills via dlsym. The libnccom path uses the same shape with individually-named slots.

How to read this atlas

Each group below is a table with a fixed column grammar: symbol (the nm name; _ZL-prefixed names are file-local statics, demangled in the cell), addr, size (bytes; hex where the nm size is hex), type (the recovered C type or struct role), init-by (the function that first writes a valid value — not .bss zeroing), read-by (the principal consumer), section, and Confidence. The init-by column is the page's central claim and carries the provenance; where a global is built by a static constructor the slot number references Static-Init, and where it is installed lazily the installer is named.

Two recurring init-by phrases mean specific things. "static-init ctor #N" means a .init_array constructor writes it at load time, before the first nrt_* call — this is the protobuf/abseil/iostream surface and the empty-container init of nrt_config. "lazy @ bring-up" (named by installer) means the global is .bss-zero until a device is opened and the named installer runs during nrt_init; these globals are deliberately not in .init_array so they are not subject to load-order constraints against the device being present (see the static-init GOTCHA).

Config singletons

The runtime's tunable state is split across two objects with different storage classes. nrt_config is a static .bss singleton; its containers are emptied by a constructor at load (ctor #11) and its scalar fields are filled at nrt_init by the env/NEFF parser. ngc is a single .bss pointer to a heap-allocated nrt_global_config_t — the parser allocates the 296-byte struct and stores it; nrt_gconf() returns it and nrt_config_free() frees and NULLs it.

The nrt_config constructor is examined in full on the Static-Init page; it only wires the two std::vector<int> and one std::map headers and arms ~nrt_config via __cxa_atexit. The full 656-byte field layout and the 296-byte ngc layout are owned by config-structs.

NOTE — two configs, two lifetimes. nrt_config (static, 656 B) and the ngc-pointed nrt_global_config_t (heap, 296 B) are distinct objects, not two views of one struct. nrt_config lives for the process; the heap struct is allocated by the parser and torn down by nrt_config_free() @0x82680, which also NULLs ngc. A reimplementation that folds both into one allocation mishandles the free path: nrt_config_free frees only the heap half.

GOTCHA — nrt_init_state_strings lives in .data, but the strings it points to live in .rodata. The four-pointer array at 0xc08900 is R_X86_64_RELATIVE-relocated; each slot resolves to a .rodata byte string (0xc08900→0x83ed2f = "NRT_STATE_START", then _INIT/_CHILD/_CLOSE). nrt_state_get_string indexes the array by the nrt_init_state enum value; an out-of-range enum reads a wild pointer because there is no bounds guard in the stringifier — the writer (nrt_state_set) is the only place the enum is validated.

Per-arch ops vtables

libnrt.so dispatches device-specific work through two coexisting per-arch function-pointer tables, each a .bss singleton bound once at bring-up by the device's architecture id (2 = sunda, 3 = cayman, 4 = mariana). kaena_khal is the wide HAL "vtable-of-vtables" (~245 qword slots: register/STPB/notific/intc/udma/TopSP accessors); tdrv_arch_ops is a narrower, named 488-byte struct (61 slots: topology counts, DMA-engine descriptors, CSR/event ops). Their division of labour and per-slot geometry are owned by tdrv-arch-ops and hal-adapter; the atlas records the writable-global view — address, size, and the installer that fills the zero bytes.

GOTCHA — the per-arch vtables are empty in the file; do not hexdump them. kaena_khal (0xcaeb80) and tdrv_arch_ops (0xc97180) are .bss (NOBITS) symbols: objdump -s shows zero bytes. Their slots exist only as stores inside the register_funcs_v{2,3,4} / tdrv_arch_register_* installers, which run lazily at device bring-up — not in .init_array. A reimplementer reconstructs them from the installers' decompiled stores (seeded from the .data.rel.ro *_hw_port_names / *_instr_block_caps source arrays), not from a static dump. This is also why these tables carry no _ZTV RTTI entry — they are C function-pointer structs, not C++ vtables (RTTI Class Hierarchy).

QUIRK — the install is single-shot and unguarded against re-arch. kaena_khal has no separate initialized guard (its slot 0 holds the arch int and doubles as the marker); tdrv_arch_ops has tdrv_arch_ops_initialized @0xc97368 but it is a plain 0/1 flag, not a lock. The install runs once, early in nrt_init, on the single thread that brings the device up. There is no re-entrancy protection: a process that opened two devices of different arch would clobber the singleton, because the vtable is process-global, not per-device. The runtime's model is one arch per process; the guard exists to assert use-after-init, not to serialize a second install.

QUIRK — arch enum and version suffix do not align across the two tables. kaena_khal_init maps arch 2→v2 (sunda), 3→v3 (cayman), 4→v4 (mariana); but tdrv_arch_ops_init switches in source order {3 cayman, 4 mariana, 2 sunda} and the seed arrays are named by codename (sunda/cayman/mariana), not by v2/v3/v4. Drive the selector off the arch enum from al_hal_tpb_get_arch_type, never off the numeric suffix.

dlopen handle tables

Two external shared objects are loaded at runtime through dlopen/dlsym and bound into function-pointer tables. The microcode loader (libnrtucode_extisa.so) uses a clean two-part shape: an immutable .data.rel.ro directory of {void** slot, const char* name} pairs (ucode_func_symbols), and a .bss block of resolved function pointers the loader fills by iterating the directory and calling dlsym(name). The collective-network plugin (libnccom) uses the same idea but its slots are individually-named .bss statics rather than a directory.

NOTE — ucode_func_symbols is a directory, the store is the payload. The 60 R_X86_64_RELATIVE relocs over 0xbf2ea0..0xbf3080 interleave two pointer kinds: even offsets point into .bss (0xbf2ea0→0xc96b30, the func slot to fill), odd offsets point into .rodata (0xbf2ea8→0x844350 = "nrtucode_get_api_level", the symbol name). ucode_init_module walks the 30 entries, calls dlsym(handle, entry.symbol), and stores the result through entry.func. So ucode_func_symbols[0] resolves "nrtucode_get_api_level" into the named .bss slot ucode_lib_get_api_level @0xc96b30. The directory is RELRO (immutable after load); the store is plain .bss (written at dlsym time). A reimplementation keeps the name table const and the resolved-pointer block writable.

GOTCHA — both tables are zero until the plugin loads; calling through an unresolved slot is a NULL jump. The .bss fn-ptr stores (0xc96a48.. and 0xc967b8..) hold NULL until ucode_init_module / the net-plugin loader runs. If the external .so is absent or dlopen fails, the slots stay NULL and any call site that does not first check the handle (ucode_lib_handle / libnccl_net_handle) jumps through NULL. The handle is the liveness gate; the resolved-pointer block is not self-describing.

Error & status globals

A small set of process-global error/status objects back logging and the collective-network error path. The error-log is a fixed-capacity ring guarded by a mutex; the nccl error globals capture the last network-plugin failure for diagnostics.

QUIRK — error_log allocates its backing store lazily, not in any constructor. The ring header at 0xc96990 is .bss-zero at load; the char* log buffer is allocated on the first print_to_error_log call and sized by capacity. The wrap logic increments num_wraps and resets tail when tail + len >= capacity. A reimplementation that pre-allocates the ring in a constructor changes the first-write timing but not behavior; the binary defers it.

CORRECTION (GLOBALS, nccl_init_error type) — an earlier inventory described nccl_init_error as an ostringstream (which would imply a std::__cxx11::basic_ostringstream of ~0x1c8 B). nm -nS reports _ZL15nccl_init_error @0xc96640 with size 0x178 (376 B); the smaller footprint and the absence of a stream _ZTV reference at that address indicate a plain capture object/string, not a full ostringstream. The init-by claim (ctor #22 _GLOBAL__sub_I_neuron_nccl.cc + __cxa_atexit) is unchanged; only the type is corrected.

Dispatch / jump tables

The standing dispatch tables that live in the writable sections (as opposed to the .rodata jump-offset spill arrays owned by Dispatch-Table Taxonomy). These are the per-engine state arrays, the proxy-queue map, the vcore tables, and the const-valued allocation/error tables. The big per-engine arrays are zeroed by constructors (ctors #10/#18/#19/#21/#25, see Static-Init) and filled per-vNC at execution time; the vcore tables and async contexts are installed lazily at bring-up.

QUIRK — v2/v3_queue_bundle_alloc_table are value tables, not pointer tables. readelf -rW shows zero relocations over 0xc08d40/0xc08bc0, and the companion *_table_len symbols are .rodata (v2_queue_bundle_alloc_table_len @0x9ba590). These tables hold packed alloc records (engine/queue indices), not function pointers — they are indexed by the model DMA allocator, not jumped through. Contrast with sequencer_error_subtypes_str @0xc091a0, which is a pointer array (12 RELATIVE relocs into .rodata). When porting, place value tables in .rodata; only the string-pointer table needs relocation.

NOTE — the big per-engine arrays are zeroed early but filled late. _tpb_xus (0x24000 B), tensoropxu (0x11000), workers (0x5800), and the barrier-state array are brought to a zero/empty state by their constructors (ctors #10/#18/#19, Static-Init), but their contents are written per-vNC during execution. The constructor cost is the zeroing; the meaningful population is in the execute path. A reimplementer who only ports the constructors gets correctly-sized but empty arrays — the per-vNC indexing logic is owned by the kmgr/xu execution pages.

Vendored statics — boundary context

The bulk of .data.rel.ro and the proto descriptor_table_* objects in .data/.bss are statically-linked third-party state, not first-party runtime globals. They are catalogued by their owning pages and listed here only so the atlas is complete about what else occupies the writable sections.

Cross-References

• Static-Init Pipeline — the 77 .init_array constructors that bring the config singleton, the proxy-queue map, the nccl error object, the ndebug stream locks, and the big per-engine arrays to their initial state; and the rationale for why kaena_khal/tdrv_arch_ops/ngc/vcores are not built there
• Dispatch-Table Taxonomy — the 222 standing data tables + 460 compiler switches; the per-arch vtable dispatch shape and the .rodata jump-offset spill arrays this page only references
• C++ Class Hierarchy and RTTI — why the per-arch ops vtables carry no _ZTV entry (C function-pointer structs, not C++ vtables) and the _default_instance_ DefaultTypeInternal objects
• CRT / PLT / Loader Surface — the RELRO mechanism that makes .data.rel.ro read-only after relocation, and the dlopen/dlsym PLT path the ucode/libnccom tables resolve through
• Configuration: nrt_config and nrt_global_config — the full 656-byte nrt_config and 296-byte nrt_global_config_t field layouts this page summarizes
• TDRV Arch-Ops — the 61-slot tdrv_arch_ops per-slot geometry and the tdrv_arch_register_* installers that fill it
• HAL Adapter (kaena_khal) — the ~245-slot kaena_khal HAL vtable and the register_funcs_v{2,3,4} arch fan-out
• Collective Proxy Driver — the consumer of g_shared_proxy_queues and the libnccom net-plugin function table