User-Facing Hint Strings

All offsets on this page are file offsets (hex) into libtpu.so from the libtpu-0.0.40-cp314 wheel (libtpu_lts_20260413_b_RC00, build-id md5 89edbbe81c5b328a958fe628a9f2207d, 781,691,048 bytes, ELF x86-64 DYN, not stripped). In this build .rodata VMA == file offset (1:1, base 0x84a0000), so a file offset is also the load address. Other wheels will differ.

Abstract

This page catalogs libtpu's hint strings: the advisory prose the runtime and the XLA:TPU compiler emit alongside an error to tell the operator or JAX user what to do about it. They are the actionable companions to the raw error/status templates catalogued on error-templates.md. Where an error template states the fact ("X must have rank >= N: got %s"), a hint appends the remedy ("Please use --xla_tpu_rwb_fusion=false", "Reduce TPU memory usage", "Please remove the hosts from the fleet"). The two live adjacent in .rodata and are joined at a .text absl::StrCat/StrFormat callsite; this page owns the hint half and its trigger/knob mapping, not the templates.

The hint surface is signposted by five marker tokens, and the choice of token is itself a taxonomy of who fixes the problem. --flag=value, "Reduce … memory", and "Please remove the hosts …" are operator-actionable — the user has a self-service remedy. b/<id>, go/<link>, and "please file a bug" point the user back at the XLA/TPU team — they signal an internal limitation or a known issue, not a knob the user can turn. This split (operator-actionable vs file-it-upstream) is the page's primary axis, refined into seven actionability classes: flag-suggestion, doc-link, bug-report, capacity/OOM, deprecation, perf-tuning, and operator-action.

This is a reference catalog, not an algorithm. Each section groups hints by trigger and, where the binary shows it, pairs the hint with the condition that fires it and the flag/knob it names. Flag names are byte-confirmed real absl::Flag globals (each has an AbslFlagHelpGenFor<name> symbol); their help-text bodies live in the config flag families and are out of scope here. The page does not claim to know which error template each hint is concatenated onto — that link is a .text callsite not yet traced (see Not Traced).

The catalog contract:

• The five marker tokens and how they partition operator-actionable from file-it-upstream.
• The seven actionability classes, each with its hints grouped by subsystem.
• For each hint: the verbatim string, its file offset, and the flag/knob or tracker coordinate it references.
• The direction-of-default caveat: whether the advised flag value is the workaround or the on-switch — directional, not byte-confirmed.

At a glance, by trigger:

The Five Marker Tokens

The hint surface was located by rg over the string table for five token families. They are not just search anchors — the token a hint carries is a reliable predictor of its actionability class, because libtpu's authors use each token consistently.

--<flag>=<value>     OPERATOR-ACTIONABLE  → set this knob and retry
"Reduce … memory"    OPERATOR-ACTIONABLE  → free HBM / raise a limit
"Please remove …"    OPERATOR-ACTIONABLE  → fleet/topology remedy
go/<link>            FILE-IT-UPSTREAM     → read the team's doc/playbook
b/<id>               FILE-IT-UPSTREAM     → known issue, comment/track
"please file a bug"  FILE-IT-UPSTREAM     → internal invariant broke

NOTE — the split is a heuristic, not a hard partition: a single hint can carry both ("…suboptimal MXU throughput … Please file a bug with XLA:TPU compiler team, and use --xla_tpu_accumulate_into_mrb=false in the meantime." at 0xa083064 is both a bug-report and a flag-suggestion). When a hint mixes tokens it is catalogued under its primary remedy and cross-noted.

GOTCHA — the go/ and b/ tokens are dominated by non-actionable uses. 514 of ~640 go/ occurrences are per-gen ISA-spec links embedded in hardware-perf-counter description strings ([counter](http://go/glc-sc-isa#anchor)), not error remedies. A reimplementer scanning for "actionable hints" by grepping go/ alone will drown in counter documentation; filter to the non-counter set (Doc-Link Hints).

Flag-Suggestion Hints

The self-service remedy class: an error or log line names a specific --xla_* / --megascale_* / --tpu_* / --deepsea_* flag and the value to set. Every flag named below was confirmed to be a registered absl::Flag global by the presence of its AbslFlagHelpGenFor<name> symbol (10/10 spot-checked resolved 1:1; flag help-text bodies belong to the config section).

Fusion / MXU

SparseCore / embedding

Scheduler / collective / infeed

Memory / OOM / runtime / debug

Direction-of-default caveat

GOTCHA — the value a hint tells you to set is not necessarily the opposite of the default; read the verb. The "use --flag=false … in the meantime" / "Please use --flag=false" hints (rwb_fusion, dot_dot_fusion, accumulate_into_mrb) name the workaround (a non-default toggle to escape a broken path). The "use --flag=true to enable" hints (tile_log_recorder, sc_log_recorder, sparse_core_reduce_scatter_v2, nested_dot_fusion) name a feature that is off by default, where =true is the on-switch. Treat these as direction-of-default; only the ~13 rodata-evidenced defaults are byte-confirmed (config section). Do not assume the advised value equals "the default".

Doc-Link Hints (go/<link>)

The go/ shortlink token splits cleanly into a small actionable playbook set and a large ISA-counter-documentation family. Only the first is a remedy; the second is reference material the profiler attaches to perf counters.

Actionable / playbook links (TPU/XLA-relevant)

NOTE — these are the only go/ links a user can act on to resolve a TPU/XLA failure. A second cluster (go/protection-key-fault, go/general-protection-fault, go/stacktraces, go/cppstackoverflow, go/lsan, go/prod-naming-1-5, go/grpc-without-init-google-exemption, go/streamz-force-collection, go/no_file_or_rpc_during_init, go/redact-debug-string) comes from statically-linked Google library prose (signal handlers, sanitizers, gRPC, telemetry), not from TPU-specific code paths. They are present in the binary but are not libtpu remedies; treat them as library noise (MEDIUM relevance).

Per-gen ISA-counter doc links (the bulk: 514 occurrences)

The dominant go/ use is not a remedy. Each hardware performance counter the profiler exposes carries a Markdown link to its ISA-spec section inside the counter's description string, e.g. 0xa0dcfac "This counter counts hold-scalar-issue cycles for [yN_reserved](http://go/vfc-sc-isaa#yn-reserved)." The link prefix encodes the TPU generation:

go/jfc-isa, go/jfc-errata                       JXC  (Jellyfish / Dragonfish)
go/pfc-isa, go/pfc-sc-isa, go/pfc-bc-isa,
  go/pfc-memory-system                          PXC  (Pufferfish)
go/vfc-sc-isa (94+ occ), go/vfc-mem             VXC vector-fetch-core (Viperfish)
go/vxc-isa, go/vxc-sc-isa                        VXC ISA (Viperfish / Ghostlite vector)
go/glc-isa, go/glc-sc-isa (94 occ)              GXC general-load-core (Ghostlite)
go/gfc-isa, go/gfc-sc-isa (97 occ), go/gfc-power GXC general-fetch-core
go/core (70 occ)                                generic core-doc anchor

QUIRK — one of these links is misspelled in the shipped binary: go/vfc-sc-isaa (confirmed at 0xa0dcfac, double a). A reimplementer reproducing the counter-description corpus should preserve the typo to byte-match this build, not "fix" it. The full counter→anchor table is a separate large extraction and is not enumerated here.

Bug-Report Hints

The file-it-upstream class. These prose strings point the user at a tracker, a team, or a b/<id> known issue. They signal that the user has hit an internal limitation, not a knob to turn.

"file a bug" prose with tracker coordinates

The TPU-lowering invariant block (0xa0c8b1e..0xa0c9758)

A contiguous run of "should not happen" templates, one per HLO op the TPU backend expects to have been legalised away before lowering. These are pure internal-bug markers — there is no user remedy. The block is contiguous but not strictly homogeneous: the first string in the run, at 0xa0c8b1e, is a sibling variant phrased about layouts rather than an un-eliminated op — "Encountered mismatched layouts for select-and-scatter. This should not happen - please file a bug against XLA." The "Encountered <Op> op during TPU lowering…" template proper begins at 0xa0c8b8d (Dot).

0xa0c8b1e  Encountered mismatched layouts for select-and-scatter.
           This should not happen - please file a bug against XLA.
0xa0c8b8d  Encountered <Op> op during TPU lowering that should have been eliminated
           during an earlier phase of compilation.
           This should not happen - please file a bug against XLA.
   <Op> ∈ { Dot, Call, BatchNormTraining, BatchNormInference, BatchNormGrad,
            Pad, Reverse, select-and-scatter, custom-fusion, OutputFusion, … }   (~14 ops)

b/<id> known-issue / TODO references (31 distinct)

These pin a limitation to a tracked bug. They are not actionable beyond "comment/track" — the most a user does is leave a note on the bug. Representative rows from the 31 distinct ids follow.

NOTE — b/<id> ids that point a user at a workaround (b/36072659 infeed, b/494604538 cancellables) are the closest this class gets to actionable; the rest are progress-tracking only. The 6-digit-minimum rg threshold means shorter b/<id> or cl/<id> changelist references may exist below it (LOW confidence on completeness).

Capacity / OOM Hints

The operator-actionable memory-pressure class: free memory, raise a limit, or wait for compaction. Grouped by where the pressure is detected.

Compile-time and runtime OOM

Embedding capacity

Fusion / VMEM / Smem capacity guards

NOTE — the VMEM/Smem/routing-table guards are capacity rejections, not remedies — they tell the user the limit was hit but name no flag. The actionable OOM hints (xla_tpu_impure_oom_fast_exit_threshold, jellyfish_executor_max_wait_time_for_releasing_memory_on_oom) are also flag-suggestions and cross-appear above.

Deprecation Notices

Sparse, and mostly flag-migration. Library "constexprs are no longer supported"-style noise is excluded; only TPU/XLA notices are listed.

NOTE — the two [DEPRECATED_XLA_TPU_FLAG_USE] strings are runtime markers the runtime prints when a deprecated TpuCompilationEnvironment flag is set; they prefix the offending flag list rather than naming a single flag. The deprecated-flag names themselves belong to the config section.

Perf-Tuning Suggestions

"Suboptimal but correct" notices: the code path works, but the user can do better.

NOTE — the MSA knob xla_tpu_msa_inefficient_use_to_copy_ratio (a proto field at 0xbfc053e) is the heuristic behind the "inefficient use-to-copy" perf path; it is named in the config layer, not in a hint string (LOW confidence that any user-facing hint prints it directly).

Operator-Action Prose

Self-service remedies that carry no bug-id: fleet/topology actions, op-sequencing protocols, init/lifecycle guidance, and team pointers.

Megascale hang digest (8 cause-branch remedies)

Each "Megascale detects a hang that is likely caused by <cause>. <remedy>." string names a different operator action keyed to the detected cause. The cause→remedy dispatch (which Cause enum selects which string) is a .text switch and is not traced.

Associated aborts: 0xa045c93 "Aborting the coordinator as an unrecoverable error is reported …"; 0xa045d37 "Aborting the coordinator after collecting errors … as megascale_error_reporter_abort_on_hang is set to true …".

QUIRK — the bad-SC-chips (0xa058553) and bad-TC-chips (0xa05862f) remedies are near-identical English with a one-word grammatical drift ("If problem persists" vs "If the problem persists"). They are two distinct rodata strings, one per cause-branch, not a single shared template — a reimplementer building the digest must emit the matching variant per cause, not deduplicate them.

TPU embedding op-sequencing protocol (0xa009da1..0xa04698b)

~10 "Ensure that … before …" hints enforce the embedding-parameter op ordering (load_ → retrieve_ → enqueue_/infeed). They describe a strict op-protocol the JAX/TF-TPU user must follow; the underlying state machine is not traced here.

"Cannot retrieve embedding parameters … until the previous retrieve/load operation
 has completed. Ensure that the previous … ops have completed before …"

"TPU embedding infeeds cannot be run in the middle of retrieving/loading the embedding
 parameters. Please quiesce all infeeds before retrieving/loading parameters and restart
 the infeeds after … is complete. Ensure that the retrieve_/load_tpu_embedding_*_parameters
 ops have completed before the enqueue_tpu_embedding_*_batch ops are run."   (0xa046624 / 0xa046853)

"Embedding parameters can be loaded … only after the TPU system has been first initialized.
 Ensure that the load_… ops are run only after tpu.initialize_system() is complete."

Init / driver-lifecycle hints

Contact-team pointers

tfrt-devs@         alignment / allocation / topology-bounds bugs
barna-core-devs@   embedding hot-id-replication threadpool sizing
Shardy team / XLA  manual-axes / unreduced-axes sharding
Megascale XLA team persistent hang after host removal
Cloud TPU Support  TPU-system init failure (the one external channel)

The CPU-Feature Startup Guard (go/sigill-fail-fast)

A fatal pre-main guard that aborts if the host CPU lacks an ISA feature the build requires. Twelve variants at 0xbe7d460+, one per feature:

FATAL ERROR: This binary was compiled with <isa> enabled, but this feature is not
available on this processor (go/sigill-fail-fast).
   <isa> ∈ { aes, avx, mmx, pclmul, popcnt, sse, sse2, sse3,
             sse4.1, sse4.2, ssse3, cmpxchg16b }                 (12 variants)

NOTE — this is the absl CPU-feature startup guard from statically-linked Google code, not a TPU-specific path — it tells the operator their host CPU is too old for the build. It is included here because it is user-facing fatal prose with a go/ remedy link, but it is not a libtpu remedy. The earlier error-template catalog named 11 of the 12 variants; the twelfth, cmpxchg16b, is confirmed present at 0xbe7d460+ (CERTAIN).

Not Traced

This page is the hint half of the diagnostic surface; several links to the rest of the surface are deliberately left open because they require disassembly this catalog did not do.

• Hint ↔ error-template link. Hints sit adjacent to their parent error template in .rodata, but the concatenation is a .text absl::StrCat/StrFormat callsite. Which of the error-template surface each hint is appended to — and the resulting absl::StatusCode — is not traced.
• Direction-of-default. Whether a flag-suggestion's advised value is the default or the workaround is a DefaultDebugOptions() / FLAGS_* ctor question, not byte-confirmed for every flag (only ~13 rodata-evidenced defaults are). The verb-based heuristic above is directional, not authoritative.
• Log severity. The severity each hint is emitted at (WARNING / ERROR / FATAL) is set at the callsite, except where the prose itself says "FATAL" / "Aborting" / "WARNING".
• Megascale cause→remedy dispatch. The 8 hang-digest strings are recovered, but the Cause-enum → string switch is not traced.
• go/<link> content resolves to internal Google URLs not present in the binary; only the link tokens and referencing prose are recovered.
• go/<gen>-isa#anchor → counter map. The 514-occurrence ISA-doc-link family is summarised by gen-codename mapping only; the per-counter anchor table is a separate large extraction.

Cross-References

• Error Templates — the raw error/status templates these hints are appended to; the fact, where the hint is the remedy.
• Runtime Overview — where in the runtime the diagnostic surface sits.
• Internal Pass Names — the pass/lowering names that appear in many "should not happen during TPU lowering" bug-report hints.
• Flag Families (config) — the registered absl::Flag globals these hints name, with their help-text bodies and defaults.
• XLA Flag Atlas (config) — the full --xla_tpu_* / --megascale_* knob catalog the flag-suggestion hints point to.