Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

NEFF Container

All addresses on this page apply to libwalrus.so from neuronx_cc 2.24.5133.0+58f8de22 (cp310 wheel; .text @ 0x62d660, .rodata @ 0x1c72000, VMA == file-offset for both). The NEFF write path is byte-identical across the cp310/cp311/cp312 wheels and the wheels share the same section bases; function bodies, however, shift by a per-wheel delta, so every address on this page is version-pinned to cp310 to match the cited disassembly. Other versions will differ.

Abstract

A *.neff file is the single deliverable a neuronx_cc compilation produces: the packaged executable image a Neuron device loads. The natural assumption is that a NEFF is an ELF-style object container, with a magic number, an e_type, a section header table, and per-section sh_offset/sh_size/sh_flags. That assumption is false in every particular. There is no ELF header, no bespoke "NEFF" magic, and no section table anywhere in the file.

A NEFF is a libarchive PAX-format tar stream, gzip-wrapped by default. Its "sections" are ordinary tar members — named files (info.json, def.json, PE.bin, Pool.bin, weight .bin blobs, protobuf .dbg files) copied verbatim from a build directory. Its "magic" is whatever the container layer puts first: the gzip member header 1f 8b in the default compressed mode, or the tar ustar magic at byte 0x101 of the first member header in --uncompressed mode. The structured metadata that a reader might expect in a leading binary record — version, UUID, engine-present bitmap, feature mask — lives inside info.json (itself a tar member), assembled in-process and never written as a header at offset 0.

This is provable three ways from the binary alone, and each is independent. (1) The container writer, NeffFileWriter::writeArchiveFile, calls the complete libarchive tar API (archive_write_set_format_pax, archive_write_add_filter_gzip, archive_entry_set_*) and no ELF/object-file emitter exists in the symbol table. (2) The CLI help text calls the output a "file.neff archive". (3) The shipped consumer, analyze_neff_artifacts.py, parses no container header at all — it open()s members by name and json.load()/os.stat()s them, the behavioural signature of a tar/directory, not an object file. The page reproduces the pack and unpack paths as annotated pseudocode and grounds every step in a symbol, address, or string.

For reimplementation, the contract is:

  • The container identity: a PAX tar + gzip stream, not an object file. The exact libarchive call sequence, the format selector, and the gzip filter options.
  • Deterministic packing: owner nobody, mtime/ctime/atime zeroed, gzip compression-level=1 + timestamp=false — so two compiles of identical IR produce byte-identical NEFFs.
  • Member layout and order: members are streamed in the key order of an in-memory std::map<path, member-name> BOM; data is padded to libarchive's default 512-byte tar record.
  • The header POD: where neff_header (version 2, region 0x400, RFC-4122-v4 UUID, feature mask) is built and where it actually lands (inside info.json).
  • The UUID CSPRNG: a ChaCha20 keystream seeded from the constant expand 32-byte k, masked to RFC-4122 version 4.
Container formatlibarchive PAX tar (archive_write_set_format_pax), gzip-filtered by default
Offset 0gzip 1f 8b (default) or tar ustar@0x101 (--uncompressed) — never ELF 7f 45 4c 46, never a NEFF tag
Container writerNeffFileWriter::writeArchiveFile @ 0x153df90 (0x153e030 is the per-member loop body)
Packager driverNeffPackager::writePackageFile @ 0x15200e0; NeffPackager::run @ 0x15307e0 (pass neff_packager)
Header builderNeffFileWriter::initializeNeffHeader @ 0x1540a00neff_header POD (≥556 B), lands in info.json
BOMstd::map<boost::filesystem::path, std::string> @ writer+144; addToBom @ 0x153fb80
Member alignmentlibarchive default 512-byte tar record (no set_bytes_per_block override present)
Identity hashMD5 "IR signature" over the artifact-subset members (boost/llvm MD5, 16-byte digest)
ELF-writer symbolszero — `nm -DC
Shipped consumeranalyze_neff_artifacts.py (md5 125d8537cdc2d034ba1d171cd82d0138, identical cp310/311/312)

1. The ELF Refutation

This is the headline, so it goes first and is made airtight before anything else is described.

A NEFF is a libarchive PAX tar, gzip-filtered by default. It has no e_ident, no e_type/e_machine, no section header table, no sh_type/sh_flags/sh_offset, and no magic check at offset 0.

GOTCHA — the one string that looks like a container magic check is a JSON lexer message. "invalid BOM; must be 0xEF 0xBB 0xBF if given" comes from the nlohmann::json UTF-8 byte-order-mark lexer rejecting a malformed BOM at the head of a sidecar JSON member. It is neither the NEFF Bill-Of-Materials nor a container magic. The two unrelated things called "BOM" are disentangled in §5.

What an ELF would require, and why none of it is present

If a NEFF were an object file, a reader would have to: read a 4-byte magic at offset 0, dispatch on e_type/e_machine, seek to e_shoff, walk a section header table, and resolve each section by sh_offset+sh_size. The producer would have to emit that header table. Neither side does any of this.

PROOF 1 — the WRITER calls libarchive, and there is no ELF emitter.
  nm -D libwalrus.so | rg archive_         → 20 symbols (full PAX tar API, see §3)
  nm -DC libwalrus.so | rg -i \
    'Elf_Ehdr|Elf_Shdr|MCObjectWriter|MachO|COFF|e_shoff|writeELF|emitELF'
                                            → 0 hits

PROOF 2 — the CLI vocabulary is a tar's, not an ELF's.
  strings -a libwalrus.so | rg 'file.neff archive'
   → "Write the compilation output (file.neff archive) in uncompressed format
      which results in faster loading of the archive during model execution."
  The only knob is gzip on/off ("archive", "uncompressed"); never a link/section flag.

PROOF 3 — the CONSUMER parses no header (see §7).
  analyze_neff_artifacts.py imports: re json operator functools argparse os sys math
   — no `struct`, no `tarfile`, no `.seek()`, no magic. It open()s members by NAME.

GOTCHA — a substring grep for ELF section fields gives false positives, so verify the hits, never the count. nm -DC libwalrus.so | rg -i 'sh_offset|elf' returns 15 symbols — all 15 are DMA dynamic-offset routines (handle64BitStaticOffset, setStaticOffsetPortion, calcOffsets…) where the regex matched …StaticOffset…/…SetOffset…. None is an ELF emitter. The precise patterns (Elf_Ehdr, Elf_Shdr, MCObjectWriter, e_shoff, writeELF) return zero. The no-ELF-writer claim survives the adversarial check.

The byte at offset 0

default (gzip):   1f 8b 08 …            gzip member header (RFC 1952)   [filter present in binary]
--uncompressed:   <tar member header>   ustar\0 ASCII at header byte 0x101
NEVER:            7f 45 4c 46           ELF magic — absent
NEVER:            4e 45 46 46 ("NEFF")  no bespoke container tag at offset 0

The gzip filter is binary-confirmed (archive_write_add_filter_gzip, §3); the 1f 8b bytes themselves are the gzip member header defined by RFC 1952 — a spec fact about gzip's output, not a literal stored in libwalrus.so. The ustar magic at offset 0x101 of a tar header is likewise the POSIX tar layout that libarchive's PAX writer emits.

The answers, in the form the question actually has

ELF concept asked aboutReality in a NEFFAnchor
e_ident / magicgzip 1f 8b (default) or tar ustar@0x101; never 7f454c46§3
e_typeN/A — container "type" is POSIX PAX tararchive_write_set_format_pax
e_machineN/A — Neuron ISA lives inside per-engine .bin members (0x40-byte bundles)§4
section header tabletar member directory: the in-memory BOM std::map<path, member> in key orderaddToBom @ 0x153fb80
sh_offset / sh_sizea member's pathname + its file bytes + 512-byte tar block alignment§3
SHF_ALLOCnone — nothing in a NEFF is a loadable ELF segment; libnrt tar-extracts then maps

2. Producer Topology

Purpose

Two C++ classes cooperate to build a NEFF. NeffPackager walks the per-core bir::Module vector, emits the JSON metadata members (neff.json, def.json, ucode_lib.json, …), and registers every artifact file into the BOM. NeffFileWriter owns the header POD and the actual libarchive tar/gzip write. The split matters to a reimplementer: the manifest (which files, in which order, under which member names) is NeffPackager's job; the byte emission (tar framing, gzip, determinism) is NeffFileWriter::writeArchiveFile's.

Entry Point

NeffPackager::run(vector<unique_ptr<Module>>&)          @ 0x15307e0   (pass "neff_packager", last)
  │  asserts modules.size() == options.vnc_nc_count     (neff_packager.cpp:113)
  └─ NeffPackager::writePackageFile(NeffFileWriter&, modules)  @ 0x15200e0
       ├─ writeNeffJson   @ 0x152c740   → neff.json   (subgraph table, schema v0.5)
       ├─ writeDefJson    @ 0x152a0e0   → def.json    (per-core definition, schema v0.6)
       │    ├─ writeVarDefinitions       @ 0x1526530  (var/tensor table + const dedup)
       │    ├─ writeDMAQueueDefinitions  @ 0x1524d10
       │    ├─ writeCCInfo               @ 0x1523af0
       │    └─ writeFP8Config            @ 0x1521e70
       ├─ writeNEFFFeatures @ 0x15294b0  → 64-bit feature mask → neff_header+192
       └─ NeffFileWriter::writeFile(outPath, uncompressed)  @ 0x1541040
            ├─ initializeNeffHeader(neff_header&)  @ 0x1540a00   (POD build, §6)
            ├─ writeArchiveFile(hdr, md5, compress) @ 0x153df90  (libarchive PAX writer, §3)
            └─ MD5::final → "IR signature: <hex> for neff artifacts" → dump BOM

Function Map

FunctionAddressRoleConfidence
NeffPackager::run0x15307e0Driver; one bir::Module per virtual NeuronCoreCERTAIN
NeffPackager::writePackageFile0x15200e0BOM assembler; iterates module vectorCERTAIN
NeffFileWriter::writeFile0x1541040Top-level write driverCERTAIN
NeffFileWriter::initializeNeffHeader0x1540a00Builds neff_header POD on the stackCERTAIN
NeffFileWriter::writeArchiveFile0x153df90The libarchive PAX+gzip writerCERTAIN
NeffFileWriter::addToBom0x153fb80Upserts one path → member-name BOM entryCERTAIN
NeffFileWriter::findInfoJson0x153ca20Resolves <baseDir>/info.jsonCERTAIN
NeffFileWriter::computeOutputPath0x153c7a0Output path compositionCERTAIN

NOTE — the 0x153df90 entry above is the true function start (where archive_write_newarchive_write_set_format_paxarchive_write_open are called). The 0x153e030 frame is the per-member loop body, ~0xa0 bytes into the function. Both are correct frames of the same routine; this page anchors the setup to the entry and the loop body to 0x153e030+. (Two-VA-frame: the archive_*@plt thunk addresses are a distinct frame again — see §3.)

NOTE (M2 — verifier frame) — the NeffPackager/NeffFileWriter method addresses in the tables above and in §2 (writePackageFile 0x15200e0, writeDefJson 0x152a0e0, writeNeffJson 0x152c740, writeNEFFFeatures 0x15294b0, initializeNeffHeader 0x1540a00) are the .dynsym symbol-start / function-entry addresses on the cp310 binary — each is the prologue (push %r15), confirmed against the IDA functions.json addr field and the demangled .dynsym value (both equal). A verifier running objdump -d --start-address=0x15200e0 lands on the prologue, not mid-body. (Do not confuse these with the internal per-symbol frame IDA also reports — e.g. 0x60ed60 for writePackageFile — which is the addToBom-style second VA frame, ~0xa0 lower; the entry addresses cited here are the ones to verify against.)


3. The Write Path

Purpose

writeArchiveFile is where bytes hit disk. It configures one libarchive write-archive object as a PAX tar, optionally adds a gzip filter, then loops the BOM and streams each member. The determinism is engineered here, not incidental.

Algorithm

// NeffFileWriter::writeArchiveFile(void *hdr, boost::md5 &sig, bool compress)  @ 0x153df90
// compress = (uncompressed_flag ^ 1)  — gzip is ON by default.
function writeArchiveFile(hdr, sig, compress):
    a = archive_write_new()                       // @0x153dfb0  → archive_write_new@plt 0x61daf0
    archive_write_set_format_pax(a)               // @0x153dfbd  → set_format_pax@plt   0x5f68c0
                                                  //   the ONLY format selector; no ustar/gnutar/v7/zip
    if compress:
        archive_write_add_filter_gzip(a)          // @0x153f134  → add_filter_gzip@plt  0x622270
        archive_write_set_filter_option(a, "gzip", "compression-level", "1")  // @0x153f14d
        archive_write_set_filter_option(a, "gzip", "timestamp",         "false") // @0x153f166
    archive_write_open(a, sink, open_cb, write_cb, close_cb)  // @0x153dfe7 → 0x5f1750
                                                  //   custom std::function sink (in-memory/buffered),
                                                  //   NOT archive_write_open_filename — bytes are
                                                  //   handed to computeOutputPath afterwards.

    for entry in BOM(std::map @ writer+144, KEY ORDER):   // sorted by absolute on-disk path
        if stat(entry.path, &st) != 0:            // @0x153e05a → stat@plt 0x5eb270
            NeuronAssert("statResult == 0")       // neff_file_writer.cpp:310
        ae = archive_entry_new()                  // @0x153e067 → 0x61e3d0
        archive_entry_copy_stat(ae, &st)          // @0x153e07a → 0x62ca10
        archive_entry_set_pathname(ae, entry.member_name)  // @0x153e08b → 0x625ec0
        // ----- DETERMINISTIC PACKING (the byte-reproducibility guarantee) -----
        archive_entry_set_uname(ae, "nobody")     // @0x153e09a → 0x5f4960   ("nobody" rodata literal)
        archive_entry_set_gname(ae, "nobody")     // @0x153e0a9 → 0x5f0d90
        archive_entry_set_mtime(ae, 0, 0)         // @0x153e0b5 → 0x6052b0
        archive_entry_set_ctime(ae, 0, 0)         // @0x153e0c1 → 0x606710
        archive_entry_set_atime(ae, 0, 0)         // @0x153e0cd → 0x5efc00
        // ----------------------------------------------------------------------
        fp = fopen(entry.path, "rb")              // else NeuronAssert("fp")  (cpp:322)
                                                  //   + "writeKelp missing file <path>"
        archive_write_header(a, ae)               // @0x153e0fa → 0x615b30
        while (n = fread(buf, 1, 0x2000, fp)):    // 8 KiB streaming chunks
            if entry.member_name in IR-sig set @ writer+216:   // §5 G1
                sig.update(buf, n)                // "Adding <n> bytes of <file> to the IR Signature."
            archive_write_data(a, buf, n)         // @0x153e2b9 → 0x6135c0
        fclose(fp); archive_entry_free(ae)
    archive_write_close(a); archive_write_free(a)

Determinism — why a NEFF is byte-reproducible

Every nondeterministic field a tar normally carries is pinned to a constant:

FieldValueMechanismConfidence
owner / group name"nobody"archive_entry_set_uname/set_gname @ 0x153e09a/0x153e0a9CERTAIN
mtime / ctime / atime0archive_entry_set_mtime/ctime/atime @ 0x153e0b50x153e0cdCERTAIN
gzip mtime stampsuppressedset_filter_option(gzip, "timestamp", "false") @ 0x153f166CERTAIN
gzip level1set_filter_option(gzip, "compression-level", "1") @ 0x153f14dCERTAIN
member orderpath-sortedBOM is a std::map, iterated in key order — not insertion orderCERTAIN

QUIRK — member order is std::map key order over absolute on-disk source paths, not addToBom call order. A reimplementer who tars members in insertion order produces a different byte stream and breaks the IR-signature comparison. The BOM key is the source path; the value is the in-tar member name. (The two are usually different — e.g. key …/nc0/sg00/sgLnk/PE.bin, member name PE.bin.)

Member alignment

libarchive's PAX writer pads each member's data to a 512-byte tar record boundary, and uses a 10240-byte default blocking factor. writeArchiveFile imposes no extra alignment: nm -D libwalrus.so | rg 'set_bytes_per_block' returns zero, so no override is present and the libarchive defaults stand. The 512-byte record is the only on-NEFF alignment of a member; the logical in-DRAM tensor alignment (const_ap_offset) is a separate concern computed by getAlignmentSize @ 0x1091890, unrelated to the tar padding.


4. The Member Layout

Purpose

"Sections" in a NEFF are tar members. There is no section header table to enumerate them; the member directory is the BOM, and a consumer discovers members by extracting the tar (or by opening a known name in an already-extracted directory). The roster below is the member set, with each entry's format and whether its bytes feed the MD5 IR signature.

Member roster

Sig? = member bytes feed the MD5 IR signature (the artifact subset). Per = T top-level / C per-core (under nc<core>/sg<subgraph>/sgLnk/).

Member nameContentFormatSig?Per
info.jsonNEFF header oracle: version major/minor, engine-present bool array, output filename, archJSONYT
neff.jsonsubgraph table v0.5: node main; per-core sg_coreV1<i> op __kelfkelf-<N>.jsonJSON v0.5nT
def.jsonper-core definition v0.6: engine→{_instr,_dbg,_asm_dbg,dma}; var table; DMA queues; CCInfo; FP8; features[]JSON v0.6nC
tensor_map.jsonper-core unified tensor map (IO binding)JSONnC
PE.bin Pool.bin Activation.bin SP.bin DVE.binper-engine ISA instruction streams (0x40-byte bundles, program order; DMA descriptors folded in)raw bundlesnC
PE.json Pool.json Activation.json SP.json DVE.jsonper-engine DMA-descriptor table ({"dma":[{queue, desc:[{from,to,…}]}]})JSONnC
kelf-<N>.jsonper-core KELF index: engine .bin filenames + offsetsJSONnC
ucode_lib.jsoncustom-op micro-code lib ([{name,opcode,sub_opcode}])JSONnC
<const>.bin / <w>.npyconstant/weight backing data, de-dup'draw bytesn / .npy onlyC
debug_info_backend.<eng>.dbg / debug_info_asm.<eng>.dbgBIR-/ASM-layer ir_debug_info protobufprotobufYC
kernel_debug_info.jsonextended kernel debug infoJSONnT
global_metric_store.json / hlo_metrics.json / metrics.json / icMetadata.json / loop.jsonmetric / autotuner / loop-spec surfacesJSONnT/C
cpu.so / cpu.paramscustom-op host CPU kernels (if present)ELF .so / bytesnT

QUIRK — cpu.so is a real ELF shared object — but it is a custom-op host kernel carried verbatim as a tar member. It is not the NEFF container and has nothing to do with the (non-existent) NEFF ELF header. A reimplementer scanning for an ELF magic inside a NEFF will find 7f454c46 only at the start of this member's bytes, never at the NEFF's offset 0.

Engine-name duality

The single most error-prone point. Two spellings of each engine live at different layers:

  • On-disk .bin/.json basenames use bir::EngineInfo2string = TitleCase: PE, Pool, Activation, SP, DVE. The shipped analyzer opens exactly these (§7).
  • def.json "definition" key tokens use the NEFF-local formatter sub_15248C0 (a standalone engine→string function, called by writeDefJson/writeDMAQueueDefinitions) = lowercase: 1→"pool", 2→"act", 3→"pe", 4→"dma", 5→"dve", 6→"sp", suffixed _instr/_dbg/_asm_dbg.

So def.json["definition"]["pe"]["_instr"] == "PE.bin"; the BOM maps the on-disk …/PE.bin path → the in-tar member name. dma(4) is a def.json token but has no .bin of its own — DMA descriptors fold into the issuing compute engine's stream. The analyzer supplies the TitleCase set; the lowercase map comes from the jump table in sub_15248C0 (see the note below).

GOTCHA — the six engine tokens are not string literals; grepping for them fails. None of pool/act/pe/dma/dve/sp exists as a \0token\0 literal in libwalrus.so. The map comes from a switch jump table at 0x15248f1, where each case loads its token as a substring lea into a longer string: case 1 "pool", case 2 "act" (an offset into a longer name), case 3 "pe" (= "Bad Shape"+7), case 4 "dma" (the tail of enableRemoteSemaphoreDMA), case 5 "dve" (SyncPoolDve+0xa), case 6 "sp" (= "Ssp"+1). Note also that sub_15248C0 (range 0x15248c00x1524d03) is its own function, adjacent to but not part of findRemoteSBVars (0x15244d00x1524884).

Per-core directory tree

Members come from a per-core build directory; the paths become the BOM keys.

<build-dir>/
  nc<core>/                      # one per virtual NeuronCore (nc0/nc00/nc01/…)
    sg<subgraph>/                # sg00 …
      sgLnk/                     # bir_linker symlink dir; analyzer --prefix points HERE
        PE.bin Pool.bin SP.bin DVE.bin Activation.bin
        PE.json Pool.json SP.json DVE.json Activation.json
        def.json  tensor_map.json
        debug_info_backend.<eng>.dbg  debug_info_asm.<eng>.dbg
        kelf-0.json  ucode_lib.json  icMetadata.json  <const>.bin
  neff.json  info.json           # top-level members
  global_metric_store.json  hlo_metrics.json  metrics.json  loop.json
  kernel_debug_info.json  cpu.so  cpu.params

NeffPackager::run asserts modules.size() == options.vnc_nc_count (neff_packager.cpp:113): one bir::Module per virtual NeuronCore → one sgLnk subtree per core, indexed by neff.json's sg_coreV1<i>. The per-member JSON sidecars and the header POD are covered in 12.2 (NEFF header + BOM writer) and 12.3 (NEFF JSON sidecars).


5. The Two BOMs

There are two unrelated things called "BOM" in this codebase, and conflating them is what makes a NEFF look like an object file with a magic number.

  1. The NEFF BOMNeffFileWriter's std::map<boost::filesystem::path, std::string> @ writer+144. Key = absolute on-disk source path; value = in-tar member name. Built one entry at a time by addToBom @ 0x153fb80, iterated in key order by writeArchiveFile to drive the tar. This is the "section index" of a NEFF — an in-memory manifest, not an on-disk record, and not an ELF section header table. Dumped at LogLevel 10 between the literals The NeffPacakger BOM: … per-entry … End of NeffPacakger BOM (the Pacakger typo is in the binary; both literals are present).
  2. The "0xEF 0xBB 0xBF" BOM — the nlohmann::json UTF-8 byte-order-mark. The string "invalid BOM; must be 0xEF 0xBB 0xBF if given" is the JSON lexer rejecting a malformed BOM at the head of a sidecar JSON member. It has nothing to do with the NEFF container or with the manifest above.

The IR-signature subset (writer+216) is a third set: the BOM entries whose bytes feed the MD5.

The membership predicate is explicit in addToBom: a member enters the IR-signature set iff its on-disk basename equals ".npy" (string::compare==0 @ 0x153fc6f) or contains ".dbg" (string::find @ 0x153fc9b), on top of the constructor's "info.json" pre-seed. The signed set is therefore exactly {info.json} ∪ {*.dbg} ∪ {*.npy}.

GOTCHA — the signature does not cover the program code. Neither the per-engine .bin instruction streams nor the <const>.bin blobs are signed. The IR signature certifies debug info, weights, and the header — not the instructions. See 12.4 (Per-Engine .bin) and 12.2 (NEFF header + BOM writer).


6. The Header POD and the UUID CSPRNG

The neff_header POD

initializeNeffHeader @ 0x1540a00 builds a flat POD (≥556 B) on the stack. It is consumed by writeArchiveFile as metadata and serialized into info.jsonit is never emitted as a leading binary record. Selected fields (full layout in 12.2):

OffsetFieldValue / sourceConfidence
+0NEFF format version2 (rodata OWORD; 0200000000000000)CERTAIN
+8region / header size0x400 (1024)CERTAIN
+192feature flags64-bit mask (writeNEFFFeatures @ 0x15294b0)CERTAIN
+204UUID16-byte RFC-4122 v4CERTAIN
+220filename"file.neff" (override path) or info.json valueCERTAIN
+480engine-present bitmapone byte/engine slot, from info.json bool arrayCERTAIN

The override branch (internalOverride, the single-output file.neff fast path) sets filename file.neff, major=1, minor=1, zeroes the engine array, and skips info.json. The "file.neff" literal is emitted as movabs 0x66656e2e656c6966 ("file.nef") plus 'f'.

NOTE — the def.json schema string "0.6" and the neff.json schema string "0.5" are JSON-member schema versions, distinct from the binary neff_header version 2. Three independent version numbers; do not collapse them.

The UUID — ChaCha20 seeded from expand 32-byte k

The UUID at neff_header+204 is RFC-4122 version 4 (random). Its randomness comes from a ChaCha20 keystream whose constant block is the canonical seed string "expand 32-byte k" — present verbatim in .rodata.

// UUID generation (RFC-4122 v4 via ChaCha20 CSPRNG)
// seed constant "expand 32-byte k" @ .rodata fileoff 0x1dd7910 (cp310, the frame this page
//   is pinned to; the cp312 wheel places the same string at 0x1dd5ed0 — a per-wheel rodata delta).
// ".rodata VMA == file-offset", so the VMA equals the file-offset.
function fill_uuid(uuid[16]):
    chacha20_block(state)                 // state[0..3] = "expand 32-byte k" (the 16-byte sigma)
    copy 16 keystream bytes into uuid
    uuid[6] = (uuid[6] & 0x0F) | 0x40     // version nibble  → 4
    uuid[8] = (uuid[8] & 0x3F) | 0x80     // variant bits    → 10xx (RFC-4122)

The "expand 32-byte k" constant is the ChaCha/Salsa20 sigma — its presence is a binary-confirmed fingerprint of a ChaCha20 keystream generator feeding the UUID. The masking (&0x0F|0x40, &0x3F|0x80) is the standard RFC-4122-v4 version/variant stamp. The sigma string is present in the binary; the ChaCha20→UUID wiring itself is [INFERRED] from that constant being co-located with the header builder.


7. The Consumer Proof

Purpose

The decisive, source-independent confirmation that there is no ELF/NEFF-magic container: the shipped consumer parses no container header at all. analyze_neff_artifacts.py (md5 125d8537cdc2d034ba1d171cd82d0138, byte-identical across cp310/311/312) is run on a directory of already-extracted members and opens them by name.

What it imports — and what it does not

imports:  re  json  operator  functools.reduce  argparse  os  sys  math
absent :  struct      (0)   →  no binary unpacking
          tarfile     (0)   →  it does not even read the tar; it reads extracted files
          .seek(      (0)   →  no offset arithmetic
          magic       (0)   →  no magic check
          e_ident/e_shoff (0) → no ELF parsing

How it reads members

# analyzeDMA(prefix)  — lines 22-41: open the five per-engine DMA sidecars BY NAME
poolFileName = "{}/Pool.json".format(prefix)            # line 24
# … SP.json, DVE.json, PE.json, Activation.json …
for fname in [poolFileName, spFileName, dveFileName, peFileName, actFileName]:
    f = open(fname)                                     # line 32 — open by NAME
    data = json.load(f)                                 # line 37 — parse JSON, no struct

# analyzeBins(prefix)  — line 74-75: stat the five ISA streams BY NAME
for engine in ['Pool', 'SP', 'DVE', 'PE', 'Activation']:
    file_size = os.stat(f'{prefix}/{engine}.bin').st_size   # line 75 — os.stat, no read/seek

analyzeTmpBuf likewise json.loads def.json and walks data['var'] / data['dma']. There is no magic check, no version check, no struct/seek/unpack anywhere in the 502-line file. A consumer that needed an ELF would read e_shoff/sh_offset; this one open()s files. That is the behavioural signature of a tar or directory, and it is the cleanest proof on the page that a NEFF is not an object file.


8. Reproducing Pack and Unpack

Pack (reimplementation skeleton)

// Equivalent of NeffFileWriter::writeArchiveFile, standalone.
struct archive *a = archive_write_new();
archive_write_set_format_pax(a);                       // PAX, the only format
if (compress) {
    archive_write_add_filter_gzip(a);
    archive_write_set_filter_option(a, "gzip", "compression-level", "1");
    archive_write_set_filter_option(a, "gzip", "timestamp", "false");   // determinism
}
archive_write_open_filename(a, "model.neff");          // (walrus uses a custom sink)

// members in std::map key order over absolute source paths:
for (const auto &[src_path, member_name] : bom) {      // KEY ORDER, not insertion
    struct stat st; stat(src_path.c_str(), &st);
    struct archive_entry *ae = archive_entry_new();
    archive_entry_copy_stat(ae, &st);
    archive_entry_set_pathname(ae, member_name.c_str());
    archive_entry_set_uname(ae, "nobody");  archive_entry_set_gname(ae, "nobody");
    archive_entry_set_mtime(ae, 0, 0);  archive_entry_set_ctime(ae, 0, 0);
    archive_entry_set_atime(ae, 0, 0);                 // all zeroed
    archive_write_header(a, ae);
    FILE *fp = fopen(src_path.c_str(), "rb");
    char buf[0x2000]; size_t n;
    while ((n = fread(buf, 1, sizeof buf, fp)))         // 8 KiB chunks
        archive_write_data(a, buf, n);
    fclose(fp); archive_entry_free(ae);
}
archive_write_close(a); archive_write_free(a);

Unpack (verification recipe — no sample NEFF ships in the wheel)

file model.neff                         # → "gzip compressed data" (default) or "POSIX tar archive"
gunzip -c model.neff | tar tvf -        # (gzip default); else: tar tf model.neff
#   expect: info.json neff.json + nc*/sg*/{PE,Pool,Activation,SP,DVE}.{bin,json}
#           def.json tensor_map.json kelf-0.json debug_info_*.dbg ucode_lib.json …
tar xf model.neff -C /tmp/neff
python3 analyze_neff_artifacts.py -p /tmp/neff/nc0/sg00    # the shipped analyzer
jq '.version, .name' /tmp/neff/info.json                   # header major/minor + filename
jq 'keys' /tmp/neff/nc0/sg00/def.json                      # "definition","var","0.6",features…
xxd model.neff | head                   # leading = gzip 1f 8b, OR (uncompressed) ustar @0x101;
                                        # NEVER ELF 7f454c46, NEVER a "NEFF" tag at offset 0.

NOTE — no *.neff sample ships in any wheel (fd -e neff = 0 across all extracted trees) — a NEFF is compiler output, not a wheel asset. The recipe above is for a NEFF produced by a real compilation; the producer/consumer behaviour it relies on is fully grounded in the binary.


9. Re-Verification Ceiling

What this page can and cannot stand behind, honestly:

  • Read directly off the cp310 binary (the frame this page is pinned to): the full libarchive PAX+gzip API (20 archive_* symbols); zero ELF-writer symbols (the 15 substring hits are all StaticOffset DMA false positives); the archive_write_new/set_format_pax/open setup sequence and the per-member set_uname/gname/mtime/ctime/atime + add_filter_gzip + two set_filter_option call sites at the cited addresses; the "nobody", "expand 32-byte k", "file.neff archive", "The NeffPacakger BOM:", and "invalid BOM; must be 0xEF 0xBB 0xBF if given" strings; the absence of any set_bytes_per_block override; the analyzer's import set and by-name member opens (md5-pinned).
  • Pinned on the addToBom disassembly in the sibling Part-12 pages: the IR-signature member-subset predicate (writer+216), resolved as {info.json} ∪ {*.dbg} ∪ {*.npy} (.npy-equals @ 0x153fc6f, .dbg-contains @ 0x153fc9b), with the .bin streams excluded.
  • Argued rather than byte-walked: the ChaCha20-keystream→UUID wiring. The sigma constant is present and co-located with the header builder, but the keystream-to-uuid[16] copy was not single-stepped.
  • [INFERRED] or deferred: the exact info.json key names feeding header fields +168/+476/+480, where nlohmann's operator[] is inlined — deferred to 12.2/12.3. The gzip member byte 1f 8b and the tar ustar at 0x101 are RFC/POSIX spec facts rather than literals stored in the binary, though the gzip filter itself is present.
  • Three VA frames coexist here, and mixing them is the standard trap. Every address on this page is the cp310 frame. The cp310 and cp312 wheels share .text/.rodata bases, but their function bodies differ by a per-wheel delta of roughly 0xa0, so a cp312 address will not objdump-verify against a cp310 body. The expand 32-byte k seed, for instance, is at 0x1dd7910 on cp310 and 0x1dd5ed0 on cp312. Separately, the @plt thunk addresses form a third frame, distinct from both the .dynsym U entries and the function-body addresses.

NameRelationship
12.2 — NEFF header + BOM writerThe neff_header POD layout and addToBom/BOM mechanics in full
12.3 — NEFF JSON sidecarsinfo.json/neff.json/def.json/tensor_map.json schemas
NeffPackager driverEmits the JSON members and assembles the BOM (§2)

Cross-References

  • Perf-Sim WiringPerfSimPackagePass::createTarball @ 0x1670ed0 builds file.perf-sim with the same libarchive PAX+gzip API (archive_write_set_format_pax + archive_write_add_filter_gzip); that tarball is later appended into the NEFF as the debug_info_pttf member.