CSR — NOTIFIC Queue (notific_n_queue)

The NOTIFIC block is the hardware instruction-notification queue: every TPB engine (PE / ACT / POOL / DVE / SP), the DMA/DGE path, the error logic, the HAM throttle monitor, and the AXI event/semaphore path emit fixed 16-byte notification records through this block, which coalesces them and writes them over AXI into one of up to ten software-owned ring queues (SW NQ) in SoC RAM. The host runtime and/or the SP core drains those rings. This page documents the APB control block (notific_10_queue / notific_1_queue), the 16-byte wire record (aws_neuron_isa_notification.h), the notific_type enum, the trigger bits, the wr_buffer mask enum, and the nine interrupt outputs.

Provenance. Byte-exact from the shipped Cayman arch-regs CSR schema csrs/notific/notific_10_queue.json (template notific_n_queue.json.pp, UnitName = notific_${makodata}_queue), the shipped ISA header common/aws_neuron_isa_notification.h, and the interrupt-trigger manifest intc/tpb_triggers.yaml. These are firmware/RTL-generated artifacts shipped in the customop library; semantics below come from the per-field Description strings and the header struct definitions — there is no separate HAL doc (HalName / HalFilenameUnitName / RegFile.Description are empty strings).

Per-gen applicability. OBSERVED on Cayman / NC-v3 (arch-regs cayman-arch-regs). The identical block ships under mariana, mariana_plus, maverick, and sunda arch dirs in the customop-lib — the register layout and the record header are arch-common. v5 (Maverick-next) is INFERRED CARRIED: no v5-specific schema is present; treat any v5 statement here as carried-forward.

1. Regfile metadata (HIGH · OBSERVED)

Two register-bundle arrays:

Register definitions: 41 + 6 = 47. Physical readable registers after array expansion: 41 + 6·10 = 101. No 0xb1 placeholder reset-value leak is present (the CSR-02/03 management-reg artifact) — every ResetValue here is one of {0, 0x1, 0xf, 0x0400, 0x1000, 0xffffffff}. (HIGH · OBSERVED)

2. Bundle 0 notific — global control block (base 0x000)

Offsets are absolute (bundle base 0x0). acc = field access; rst = reset value. Every row is literal from notific_10_queue.json. (HIGH · OBSERVED unless tagged)

2.1 Timestamp (64-bit free-running ps counter)

This 64-bit value is snapshotted into bits [127:64] of every notification record. Per notification.h the unit is 1 ps (not clocks; differs from the older Tonga part). The increment is set by timestamp_inc (§2.10).

2.2 Backpressure control

⇒ up to 64 hardware instruction inputs feed one NOTIFIC. (MED · INFERRED — from the two 32-bit hw_backpressure halves.)

2.3 Queue enable / overflow policy

2.4 AXI / transport config

The wr_buf_enable bit positions are named by the enum NEURON_ISA_TPB_NOTIFIC_WR_BUFFER (§6): they gate which record-class write-buffers are live, so firmware masks whole error/evsem classes away per NOTIFIC by clearing the corresponding bit. Per the header, "currently only evsem and error wr_buffer controls are exposed."

2.5 AXI write-error + pipeline-stall diagnostics

2.6 NIQ — Notification INPUT Queue debug readback

⇒ one NIQ entry = 144 bits (18 B): the 128-bit (16-B) record plus 16 b of internal input metadata (queue-select / input-id). Read protocol: write idx, then read niq_buffer_0..4. (MED · INFERRED — the 128 b/16 b split is from the field widths, not labeled in-schema.)

2.7 NAQ — Notification AXI Queue (coalesced egress) debug readback

⇒ one NAQ entry = 224 bits (28 B): the post-coalescer staged AXI write payload — a 16-B record plus dest-address / burst bookkeeping. The Write-Order-Buffer (wob_ctrl) and coalescer (coal_*) drain the NAQ into AXI bursts toward queue RAM. (MED · INFERRED.)

2.8 Statistics

2.9 Coalescer

The coalescer merges multiple notifications destined for the same SW NQ into one wider AXI burst; only the first 4 SW queues are coalescible. A coalescer hit across multiple streams raises intr_8 (non-deterministic AXI).

2.10 Write-Order Buffer, threshold/full status, timestamp increment, spares

CORRECTION (vs SX-CSR-06). SX-CSR-06 lists spare_0s reset as 0x00000000; the schema literal is ResetValue: "0x0000" (same numeric value). No semantic difference; recorded for byte-exactness.

3. Bundle 1 notific_nq — per-queue ring descriptor (base 0x100, stride 0x28)

Six registers replicated NUM_SW_Q (10) times. Offsets below are within the per-queue descriptor; the descriptor for queue i starts at 0x100 + i·0x28. (HIGH · OBSERVED)

SpecialAccess=PulseOnW on base_addr_lo/hi and head_ptr means a write also emits a one-cycle pulse (latch / re-arm) — an RTL-gen attribute; software just writes the value. (MED · INFERRED — RTL semantics.)

3.1 Absolute offset map (HIGH · OBSERVED)

Queue i descriptor base = 0x100 + i·0x28:

The array spans 0x100..0x290 (10·0x28 = 0x190), well inside the 4 KiB window.

4. The 16-byte notification record (wire format)

Source: aws_neuron_isa_notification.h. NEURON_ISA_NOTIFICATION_NBYTES = 0x10 ⇒ every queue entry is 16 B = 128 b, little-endian, four uint32 words. (HIGH · OBSERVED)

4.1 Common header byte (word0 [31:24])

The header is a single packed byte (NEURON_ISA_NOTIFICATION_HEADER); in a little-endian packed bitfield notific_type occupies the byte's LSBs:

typedef struct {
    uint8_t notific_type:5;             // header byte [4:0]  = word0[28:24]
    uint8_t software_queue_overflow:1;  // header byte [5]    = word0[29]
    uint8_t hardware_queue_overflow:1;  // header byte [6]    = word0[30]
    uint8_t phase:1;                    // header byte [7]    = word0[31]
} __attribute__((packed)) NEURON_ISA_NOTIFICATION_HEADER;

Bit defines (header lines 38-43):

The header notes the byte layout is "similar to Tonga except the type field is widened to 5 bits and the block_id field is removed."

4.2 Generic record (the common 16-B frame)

typedef struct {
    uint16_t                          metadata_0;  // bytes[1:0]  = word0[15:0]
    uint8_t                           metadata_1;  // byte[2]     = word0[23:16]
    NEURON_ISA_NOTIFICATION_HEADER    header;      // byte[3]     = word0[31:24]
    uint32_t                          metadata_2;  // bytes[7:4]  = record bits[63:32]
    struct { uint32_t low; uint32_t high; } timestamp; // bytes[15:8] = bits[127:64], 1 ps
} __attribute__((packed)) NEURON_ISA_GENERIC_NOTIFICATION;

Field offset map (header lines 95-109):

4.3 Phase-bit fast path (HIGH · OBSERVED)

Software polls the phase bit of the next-expected entry at base_addr + head_ptr. When it matches the current epoch the entry is valid — no tail_ptr readback is needed on the fast path (and recall tail_ptr does not guarantee AXI drain completion; the phase bit in RAM is the authoritative arrival signal). Each ring wrap inverts the expected phase.

4.4 Per-type record bodies (word0[23:0] + word1)

All bodies share the header byte and the trailing 64-b timestamp; they differ in how the 24-b [23:0] + 32-b [63:32] payload is interpreted. (HIGH · OBSERVED)

The DMA sub-type lives in dma_notific_type:4 (NEURON_ISA_DMA_NOTIFICATION_TYPE): M2S_SOP=0x1 M2S_EOP=0x2 M2S_COMPLETION=0x3 S2M_SOP=0x4 S2M_EOP=0x5 S2M_COMPLETION=0x6 AXI_REQ_0/1=0x8/0x9 AXI_RSP_0/1=0xa/0xb APB_REQ_0/1=0xc/0xd.

4.5 DGE completion record (HIGH · OBSERVED)

A DGE op's completion is reported as a TPB EXPLICIT notification whose debug_hint == NEURON_ISA_TPB_INSTRUCTION_METADATA_TYPE_DGE (0x1) (NEURON_ISA_TPB_DGE_METADATA_NOTIFICATION):

typedef struct {
    uint32_t program_counter:20;        // NX IRAM offset, 64-B resolution
    uint32_t debug_hint:4;              // == DGE (0x1)
    NEURON_ISA_NOTIFICATION_HEADER header;
    uint16_t dma_map;                   // bitmap of 16 DMA engine IDs used
    uint16_t num_packets:12;            // DMA packets per engine; 0 == 4096 (0x1000)
    uint16_t reserved:2;
    uint16_t infer_program_counter:1;   // 1 => take PC from most recent start_notification
    uint16_t dma_group_id:1;            // 0 => engine IDs 0-15, 1 => 16-31
    struct { uint32_t low, high; } timestamp;
} __attribute__((packed)) NEURON_ISA_TPB_DGE_METADATA_NOTIFICATION;

This is the firmware-visible payload the host reads from the SW NQ to learn which DMA engines/packets a dynamic DGE instruction used. See DGE error notifications.

5. notific_type enum (the notification kinds)

NEURON_ISA_NOTIFICATION_TYPE, header lines 66-92 — 24 values in the 5-bit field. (HIGH · OBSERVED)

Codes 0x00, 0x01, 0x18–0x1a, 0x1c–0x1d are unassigned (reserved). The engine quad pattern {INST_START, INST_END, EXPLICIT, EVT_SEM} repeats with a 4-code stride per engine: PE 0x04, ACT 0x08, POOL 0x0c, DVE 0x10, SP 0x14.

The TPB_ERROR (0x1f) body carries error_id from NEURON_ISA_TPB_ERROR_TYPE: FP_UNDERFLOW(0x00) FP_NAN(0x01) FP_INF(0x02) FP_OVERFLOW(0x03) MEMORY_ERROR(0x04) FAKE_ERROR(0x05) SEMAPHORE_ERROR(0x06) EVENT_ERROR(0x07) PSUM_COLLISION(0x08) SEQUENCER_NONFATAL(0x09) SEQUENCER_FATAL(0x0a).

6. wr_buffer mask enum + trigger-bit map

6.1 NEURON_ISA_TPB_NOTIFIC_WR_BUFFER (HIGH · OBSERVED)

One-hot 64-bit class mask used by wr_buf_enable_lo/hi (§2.4) to mask records by header.notific_type. Header lines 533-578:

The reduced NEURON_ISA_TOPSP_NOTIFIC_WR_BUFFER exposes only EVSEM (1<<0) and ERROR_SEQUENCER (1<<2, both fatal+nonfatal) — matching the header note "only evsem and error wr_buffer controls are exposed."

6.2 Trigger-bit map (HIGH · OBSERVED)

#defines at the tail of notification.h (lines 778-784) — these set the NOTIFIC interrupt trigger bits driving the INTC:

Trigger bits 0–6 map 1:1 to the named #defines; intr_7 (SW-NQ overlap) and intr_8 (coalescer multi-stream hit) are generated inside the block and have no #define trigger bit — they are pure INTC sources (next section).

7. The nine NOTIFIC interrupt sources

From intc/tpb_triggers.yaml: tpb_notific_intr_0 .. tpb_notific_intr_8 — 9 interrupts total. All share needs_cdc: true, source_clock: clk_core_gated, source_reset_n: core_reset_extended_n, edge_triggered: true, msix_mask: 0. (HIGH · OBSERVED)

(The YAML additionally enumerates wr_buffer0..29 drop/full diagnostic triggers — one per HW input buffer — which feed intr_2/intr_3; they are not part of the nine top-level NOTIFIC interrupts.) For INTC routing/binding of these nine, see the device→host interrupt/notification path.

8. NUM_SW_Q = 10 vs NUM_SW_Q = 1

notific_1_queue.json is the same block with NUM_SW_Q=1: identical 41 + 6 register layout, identical SizeInBytes=0x1000, and the bundle-1 ArraySize follows the parameter so the per-queue descriptor array collapses to a single entry at 0x100..0x128. The [NUM_SW_Q-1:0] fields (sw_backpressure, nq_enable, nq_sw_overflow, nq_threshold_en, nq_full, nq_threshold_passed) collapse to 1 bit, and SW_Q_RESET_TO_ALL_1 is 1 (vs 1023). (HIGH · OBSERVED)

Why two instantiations. The 10-queue variant is the production SW-visible queue set — it gives the routing layer (§9) a 4-bit index space so the cluster can dedicate separate rings to different notification classes (per-engine instruction completions, the error/errors_NT_ route, the DGE-completion route, HAM, Q7). The 1-queue variant is the minimal instantiation for a context that needs a single notification sink (a single output ring with no per-class routing) — same MMIO map, no per-queue index logic. (MED · INFERRED — from the parameter difference and the routing structure.)

9. Queue protocol + producer/consumer signaling

9.1 Configuration (host/SP, via APB) — HIGH · OBSERVED

1. For each SW NQ i: write notific_nq[i].base_addr_lo/hi (RAM ring address), .size (bytes), optional .threshold; clear/program head_ptr.
2. Set bit i in nq_enable (0x14). Choose the overflow policy: either sw_backpressure[i]=1 (lossless — stall producers when full) or nq_sw_overflow.ignore_full_en[i]=1 (lossy ring-overwrite). Optionally set nq_threshold_en[i] and coal_ctrl.stream_en (NQ 0–3 only).

9.2 Enqueue (producer side) — pseudocode

// Engine/firmware producer path through one NOTIFIC instance.
// Register/field symbols are the literal CSR/header names.
void notific_enqueue(notific_csr_t *nf,
                     const NEURON_ISA_NOTIFICATION *rec_seed,
                     uint8_t sw_nq_idx /* from tpb routing, sec 9.4 */)
{
    // 1. Producer (PE/ACT/POOL/DVE/SP seq, AXI, HAM, error) drops a record
    //    into a HW input buffer (NIQ). hw_backpressure_lo/hi gates full-buffer
    //    behaviour: stall -> tpb_notific_intr_2, or drop -> tpb_notific_intr_3.

    // 2. NOTIFIC builds the 16-B record: copy seed body, snapshot the 64-bit
    //    timestamp (timestamp_lo|hi, unit 1 ps) into rec[127:64], set the
    //    header byte (notific_type, phase = current ring epoch, and the
    //    hardware/software_queue_overflow bits if a prior drop occurred).
    NEURON_ISA_NOTIFICATION rec = *rec_seed;
    rec.generic.timestamp.low  = nf->timestamp_lo;
    rec.generic.timestamp.high = nf->timestamp_hi;
    rec.generic.header.phase   = ring_epoch(nf, sw_nq_idx);

    // 3. Routing/enable checks.
    if (!(nf->nq_enable & (1u << sw_nq_idx))) {        // disabled NQ
        raise(NOTIFIC_QUEUE_DISABLED_ERR_BIT);          // -> intr_1, drop
        return;
    }
    if (nq_is_full(nf, sw_nq_idx)) {                    // nq_full bit set
        if (nf->nq_sw_overflow & (1u << sw_nq_idx)) {
            rec.generic.header.software_queue_overflow = 1; // lossy overwrite
        } else if (nf->sw_backpressure & (1u << sw_nq_idx)) {
            raise(NOTIFIC_SW_NQ_FULL_ERR_BIT);          // -> intr_0, stall
            return;
        }
    }

    // 4. Coalescer (coal_*) + WOB (wob_ctrl) stage the record in the NAQ and
    //    issue an AXI write (AXI ID = nq_axi_id) to base_addr + tail_ptr;
    //    tail_ptr advances by 16 B (mod size). nq_uart_mode pins all writes to
    //    base_addr. On AXI write-response error: nq_error_addr_lo/hi <- addr,
    //    raise NOTIFIC_AXI_WRITE_ERR_BIT (-> intr_4).
    naq_stage_and_write(nf, sw_nq_idx, &rec);           // tail_ptr += 0x10
    nf->nq_stats++;                                     // read-clearing total
}

9.3 Drain (host or SP consumer) — pseudocode

// Host/SP consumer of one SW NQ ring in RAM.
size_t notific_drain(notific_nq_desc_t *q /* notific_nq[i] */,
                     volatile uint8_t  *ring /* mapped base_addr */,
                     int expected_phase)
{
    size_t  off = q->head_ptr;                 // delta-from-base, consumer ptr
    size_t  consumed = 0;
    for (;;) {
        NEURON_ISA_NOTIFICATION *e = (void *)(ring + off);
        // Fast path: poll the phase bit; valid when it matches the epoch.
        if (e->generic.header.phase != expected_phase)
            break;                              // no new entry
        dispatch_by_type(e);                    // e->generic.header.notific_type
        off = (off + NEURON_ISA_NOTIFICATION_NBYTES) % q->size; // 0x10, ring
        if (off == 0) expected_phase ^= 1;      // wrap inverts phase
        consumed++;
    }
    // Drain ACK: write head_ptr. Frees ring space AND re-arms intr_6 for this NQ.
    q->head_ptr = off;                          // PulseOnW latches + re-arms
    return consumed;
}

Notes: tail_ptr (RO) reflects the last write but does not guarantee AXI drain completion — the in-RAM phase bit is authoritative. intr_6 (threshold) and intr_0 (full) let SW drain lazily instead of per-notification polling. Diagnostics: nq_stats (read-clearing total), nq_full / nq_threshold_passed, nq_error_addr_lo/hi, and the NIQ/NAQ buffer readback (*_buffer_index + *_buffer_N) for HW debug.

9.4 Routing (which SW NQ a notification lands in) — MED · CARRIED

The SW-queue index is supplied by the cluster-level routing block (the tpb top-level notific bundle; routing is described in tpb.md). Each engine's three notification classes (NX-implicit / explicit / engine-implicit) plus the error / event-semaphore / HAM / Q7 classes are mapped to a 4-bit SW-queue number; a per-engine queue_idx_ctrl bit lets the SW-queue index come from the instruction instead of the static field. The 4-bit index selects one of this block's NUM_SW_Q descriptors. (CARRIED from SX-CSR-06 §5 / SX-CSR-04; tpb.md is a stub pending authoring — the routing CSR offsets are unverified here.)

The committed anchor (#901/#902/#913/#916): notific_ctrl @ 0x808 lives in tpb.json events_semaphores, inside the EVT_SEM container at 0x2802700000; the NOTIFIC queue documented here is the routing target of that cluster event/notification path.

9.5 Producers / consumers

• Producers: TPB sequencers PE / POOL / ACT / DVE / SP (0x04..0x17); the DMA/DGE path (0x02 + the DGE-metadata explicit record §4.5); error logic (0x03 / 0x1f); HAM (0x1e); AXI evt/sem (0x1b); the NX/Q7 cores via their per-core notific intr_info bundle.
• Consumers: host runtime and/or the SP core read the RAM rings and advance head_ptr. The DGE/error path surfaces to runtime via the errors_NT_ queue. The host→firmware DGE mailbox is the complementary downstream channel: the NOTIFIC queue is the firmware→host completion/telemetry direction; together they form the DGE control loop. The drained record stream is what the NTFF trace parser reconstructs.

10. Cross-references

• Device→Host Interrupt / Notification Path — the producer/INTC side of the nine NOTIFIC interrupts.
• DGE error notifications — the DGE-completion metadata record (§4.5).
• NTFF trace parse state — host-side decode of the drained record stream.
• CSR — tpb — the cluster routing source (sw_queue_num / queue_idx_ctrl) that selects which notific_nq[idx] a record targets.
• The ENS notification-queue mirror struct (host execution-state struct census) — NOTE: the dedicated struct-device-firmware-globals.md appendix page is not yet authored; the closest existing appendix target is linked here.

Confidence summary

• §1–3, §4.1–4.5, §5, §6, §7, §8: HIGH · OBSERVED — literal from notific_10_queue.json, notific_1_queue.json, aws_neuron_isa_notification.h, and tpb_triggers.yaml.
• NIQ=144 b / NAQ=224 b internal splits (§2.6–2.7), 64 HW inputs (§2.2), PulseOnW semantics (§3): MED · INFERRED from field widths / RTL attributes.
• Routing CSR offsets (§9.4) and the 1-queue rationale (§8): MED · CARRIED from SX-CSR-06 / SX-CSR-04 pending tpb.md authoring.
• v5 applicability: INFERRED CARRIED (no v5-specific schema present).