The libnrt ↔ libnccom ABI

Two binaries, pinned together. libnrt.so.2.31.24.0 (aws-neuronx-runtime-lib 2.31.24.0-0b044f4ce, build-id 8bb57aba0fb2e0035f1d88e9fc4fb3e7387c102e, SONAME libnrt.so.1) and libnccom.so.2.31.24 (aws-neuronx-collectives 2.31.24.0-1a31ba186, build-id 9c00176c081788c9435d27d11bb40e92495463f0, SONAME libnccom.so.2). Both ELF64 x86-64 DYN, NOT stripped. A third file, libnccom-net.so (build-id 3415f096d479e7d7bef506bb68bc0fa7551a654a), exports the ncclNetPlugin_v4/v5/v6 structs and is catalogued here only at its boundary edge — its internals are owned by net-plugin. Every address on a FORWARD row is a libnccom.so.2.31.24 text VMA; every address on a REVERSE row is a libnrt.so.2.31.24.0 text VMA; .text/.rodata VMA == file offset in both. Definer-binary names appear demangled (nm -D --defined-only), user-binary references with their @version tag (nm -D --undefined-only --with-symbol-versions).

Evidence grade: Confirmed (byte-anchored, both sides re-derived) — the 37 / 16 / 4 counts were re-run with nm -D on each binary independently; every FORWARD row is anchored to its libnccom definer address AND its libnrt .bss slot + call *slot(%rip) site; every REVERSE row to its libnrt @@NRT_2.0.0 definer address AND its libnccom @NRT_2.0.0 UND import; readelf -d/-V confirm the asymmetric link (no DT_NEEDED libnccom in libnrt; DT_NEEDED libnrt.so.1 + VERNEED NRT_2.0.0 in libnccom). · Part XII — Multi-Node Collectives · back to index

Abstract

The runtime and the collectives library link to each other in two opposite directions by two different mechanisms, and this page is the symbol-by-symbol closure of that boundary. In the FORWARD direction (libnrt → libnccom) there is no ELF link at all: libnrt.so carries no DT_NEEDED on libnccom (its NEEDED list is nine toolchain libraries — libgcc_s, libutil, librt, libpthread, libdl, libstdc++, libm, libc, ld-linux), and it holds zero neuron* symbols in its dynamic table. Instead the embedded ncclInit (libnrt 0x1bff30) dlopens "libnccom.so" with RTLD_NOW at runtime and dlsyms 37 neuron* entry points into a contiguous .bss function-pointer table (libnrt 0xc967b8..0xc968d8, 8-byte stride). Thirty-eight thin nccl* trampolines call through those slots (call *0xc968d0(%rip) etc.). The 37 targets are plain unversioned T exports in libnccom; the only handshake is a numeric one — neuronGetVersionInfo must report compatibility id 89, checked at libnrt 0x1bff9b (cmpq $0x59,…) against libnccom's movq $0x59,… (0x464a8).

In the REVERSE direction (libnccom → libnrt) the link is hard and versioned: libnccom.so carries DT_NEEDED libnrt.so.1 and a .gnu.version_r entry requiring (libnrt.so.1, NRT_2.0.0) — and only NRT_2.0.0, never NRT_3.0.0. It imports 20 symbols, all tagged @NRT_2.0.0: 16 nec_* device/topology/proxy reverse-callbacks and 4 nrt_* runtime queries, every one resolving name-for-name against a libnrt @@NRT_2.0.0 definition. The page is organized as one catalogue with three groups — FORWARD (37 neuron* dlsym targets), REVERSE (16 nec_* + 4 nrt_* versioned imports), and the NET edge (ncclNetPlugin_v4/v5/v6, which belong to libnccom-net's ncclNet_vN ABI and are not part of the nec_/neuron boundary) — followed by the resolution-mechanism notes and the asymmetry quirk that a reimplementer must reproduce exactly.

For reimplementation, the contract of the boundary is:

• The FORWARD table is a dlsym target list, not an export list. A reimplementation of libnccom must define all 37 neuron* symbols as plain global text exports (no version node) so libnrt's dlsym finds them; a reimplementation of libnrt must dlopen+dlsym them in order, NULL-check every result, and zero all 37 slots on the first failure (collectives degrade to all-or-nothing). There is no DT_NEEDED on this leg in either direction.
• The REVERSE table is a versioned linker import. A reimplementation of libnccom must DT_NEEDED libnrt.so.1 and bind all 20 nec_*/nrt_* imports at @NRT_2.0.0; a reimplementation of libnrt must export those same 20 names at @@NRT_2.0.0. Dropping any one of the 20 from the NRT_2.0.0 node breaks libnccom at load time (before any dlopen).
• The two skew axes are independent. The reverse leg is gated by the ELF symbol version (NRT_2.0.0); the forward leg is gated by the numeric compat id 89 returned by neuronGetVersionInfo. They are checked by different mechanisms at different times (link-time vs. first-call). Reproduce both, or one half of the boundary silently mis-negotiates.
• The NET structs are a different ABI. ncclNetPlugin_v4/v5/v6 are exported by libnccom-net.so, follow the upstream ncclNet_vN C-struct-of-function-pointers shape, and reach libnrt only through the single nrt_get_libnccl_net@NRT_2.0.0 handoff. Do not conflate them with the neuron* dlsym set.

At a glance

Resolution Mechanisms

The two halves of the boundary do not just point in opposite directions — they are resolved by two unrelated dynamic-linker facilities, and conflating them is the classic reimplementation error.

FORWARD — runtime dlopen + dlsym, unversioned

libnrt never names libnccom in its ELF. readelf -d libnrt.so lists no NEEDED libnccom, and nm -D libnrt.so | grep neuron is empty — there is no neuron* symbol, defined or undefined, in the runtime's dynamic table. The link is forged at runtime: ncclInit (0x1bff30) calls dlopen("libnccom.so", RTLD_NOW), then dlsyms each neuron* name out of the returned handle and stores the resolved pointer into a fixed .bss slot. Because dlsym matches on name only, the 37 targets are exported from libnccom as plain T globals with no @version tag — and indeed nm -D --defined-only libnccom.so | grep neuron shows all 37 as bare T, none versioned. The nccl* trampolines that the rest of libnrt calls are one-instruction indirect jumps through the slot: e.g. ncclGetUniqueId @0x1c0a80 : call *0xad5e4a(%rip) # 0xc968d0 <neuronGetUniqueId> and ncclInitComm @0x1c0e57 : call *0xad5a63(%rip) # 0xc968c0 <neuronInitComm> (both re-read from objdump -d).

REVERSE — hard DT_NEEDED + versioned import

libnccom does name libnrt: readelf -d libnccom.so shows NEEDED libnrt.so.1, and readelf -V shows a .gnu.version_r entry File: libnrt.so.1 Cnt: 1 Name: NRT_2.0.0 (version index 3). Every nec_*/nrt_* import therefore arrives at the dynamic linker tagged @NRT_2.0.0 and is resolved by the standard symbol-versioning rule against libnrt's @@NRT_2.0.0 definitions — at load time, before ncclInit ever runs. There is exactly one required version node from libnrt (NRT_2.0.0); libnccom requires nothing from NRT_3.0.0 (the nrta_* async family), so a newer libnrt that keeps NRT_2.0.0 stays load-compatible.

QUIRK — the boundary is asymmetric by mechanism, not just by direction. FORWARD is a runtime dlsym keyed on bare name with no ELF version and a separate numeric compat gate (id 89); REVERSE is a load-time linker bind keyed on NRT_2.0.0 with no numeric gate. Exporting a nec_* callback unversioned breaks libnccom's load outright — its import is @NRT_2.0.0 and an unversioned definition does not satisfy a versioned need. Versioning a neuron* export, by contrast, changes nothing the forward leg observes: dlsym ignores the version and binds the default. The same mistake fails loudly on one leg and silently passes on the other.

The ABI Catalogue

One row per symbol. Direction is FORWARD (libnrt→libnccom, dlsym), REVERSE (libnccom→libnrt, versioned import), or NET (libnccom-net→host, ncclNet_vN). Definer is the binary that holds the T/D definition and its address; User is the binary that resolves/calls it. Signatures are from the definer-side DWARF; NRT_STATUS is the runtime's 28-value status enum. Every row is HIGH (byte-anchored on both sides) unless tagged.

FORWARD — 37 neuron* dlsym targets (definer libnccom, user libnrt)

The version probe (neuronGetVersionInfo) is dlsym'd first and gates the rest. The .bss slots descend from 0xc968d8 (probe) to 0xc967b8 (last) in 8-byte steps; each row gives the libnccom definer addr, the libnrt .bss slot, and the libnrt nccl* trampoline that calls through it.

FORWARD count = 37, re-derived: nm -D --defined-only libnccom.so.2.31.24 | rg -c ' neuron' → 37; every addr above matches the nm dump, and libnrt's dynamic table holds no neuron* symbol (forward leg is pure dlopen+dlsym, confirmed).

NOTE — libnccom exports these 37 as bare T with no @version — there is no .gnu.version_d node for them at all (it consumes libnrt's NRT_2.0.0 node and defines none of its own). The version tag a reimplementer might expect on a "collectives ABI" simply does not exist on the forward leg; the only forward compatibility token is the numeric 89.

REVERSE — 4 nrt_* runtime queries (definer libnrt, user libnccom)

All four are libnrt @@NRT_2.0.0 exports; libnccom imports them @NRT_2.0.0. Addresses are libnrt text VMAs.

REVERSE — 16 nec_* device/topology/proxy callbacks (definer libnrt, user libnccom)

All libnrt @@NRT_2.0.0 exports; thin thunks into libnrt's NDL/encd layer (target named where traced). libnccom imports them @NRT_2.0.0.

REVERSE count = 16 nec_ + 4 nrt_ = 20, re-derived: nm -D --undefined-only libnccom.so.2.31.24 | rg -c ' nec_' → 16; … | rg -c ' nrt_' → 4; … --with-symbol-versions | rg -c '@NRT_2.0.0' → 20 (and … | rg 'NRT_3.0.0' → empty). Each name resolves against a libnrt @@NRT_2.0.0 definition at the address above (nm -D --defined-only --with-symbol-versions libnrt.so.2.31.24.0).

NOTE — the libnrt-local nec_get_version_info (0x1bfde0, jmp ncclGetVersionInfo 0x1c0a10) and the nec_vil_* helpers are not in the dynamic symbol table and are correctly absent from libnccom's import list — they are intra-libnrt calls, not boundary symbols. The catalogue is the dynsym boundary only.

NET (edge) — ncclNetPlugin_v4/v5/v6 (definer libnccom-net)

Catalogued here only to fix the distinction: these are not part of the nec_/neuron boundary. They are the upstream ncclNet_vN plugin ABI — flat C structs of function pointers in libnccom-net.so's .data, dlopened by libnrt (whose handle reaches libnccom via nrt_get_libnccl_net, the REVERSE row above). libnccom binds ncclNetPlugin_v6 only.

NET count = 3, re-derived: nm -D --defined-only libnccom-net.so | rg -c ncclNetPlugin → 3. These three are data objects (D), not text — distinguishing them at a glance from the 37 neuron* T functions and the 20 nec_/nrt_ T callbacks.

Counts — re-derived from nm on both binaries

All three headline counts hold: 37 forward / 16 nec_ + 4 nrt_ reverse / 3 net. No CORRECTION to the source figures.

Cross-References

• The libnrt ↔ libnccom Boundary (nec_ / nccl) — the conceptual boundary and the nec_* thunk semantics; this page is its symbol-level catalogue
• The Net Plugin (ncclNet_v6) — owns the ncclNetPlugin_v4/v5/v6 structs and the nrt_get_libnccl_net handoff the NET edge points at
• Communicator Init and Bootstrap — the forward call ordering (neuronInitComm/neuronGetUniqueId/bootstrap) and the compat-89 handshake in flight
• Proxy & Progress Engine (and the libnrt Bridge) — the ncclStart/Progress/Stop seam and the nec_inc_semaphore proxy-completion callback
• Symbol-Version Manifest — libnrt's NRT_2.0.0 / NRT_3.0.0 version nodes and the full exported-symbol inventory