Record: SP8192 + LQER + Sparse Attn Gate + BOS-Fixed SmearGate + 9-Hparam Greedy Stack — val_bpb 1.06108 (3-seed mean)

[codemath3000]

Summary

11L 512d 8H/4KV transformer with U-Net skips, parallel residuals, partial RoPE, Polar-Express Newton-Schulz Muon, LQER asymmetric int4 rank-4 quant correction, sparse attention head-output gate, SmearGate with cross-document leak fix on BOS positions (audit response), fused LeakyReLU-square MLP, fused softcapped CE Triton kernel, GPTQ int6 + int7 embed + per-row int8 attn-gate, per-group lrzip + brotli compression pipeline (added in this submission — PR #1797's base only ships lzma / brotli), phased TTT eval (3 cumulative phases at doc-boundaries 833 / 1666 / 2500, max prefix=2500 docs), with 9 hyperparameter overrides validated by greedy forward-selection on 8×H100 real fixed-step.

3-seed mean: 1.06108 BPB (std 0.00090) on 8×H100 SXM, all artifacts under the 16 MB cap.

seed	post-TTT val_bpb	artifact bytes	eval_time
42	1.05989	15,897,259	508.8 s
0	1.06125	15,900,947	455.1 s
1234	1.06209	15,907,550	470.0 s
mean	1.06108	15,901,919	478.0 s

vs current leaderboard (1.0810 BPB): −0.01992 BPB / −0.04359 nats.

SmearGate cross-document leak fix

SmearGate's per-token forward-1 mixing (x[:, 1:] + g * x[:, :-1]) leaks the last token of doc N into the BOS embedding of doc N+1 in a packed validation stream. Fix masks the prev-token term wherever the current token is BOS:

not_bos = (input_ids[:, 1:] != BOS_ID).to(x.dtype).unsqueeze(-1)
x = torch.cat([x[:, :1], x[:, 1:] + g * x[:, :-1] * not_bos], dim=1)

Applied symmetrically in _forward_hidden and forward_ttt so training and TTT eval are leak-free.

Per-group compression pipeline

PR #1797's base only exposes lzma / brotli compressors. This submission adds a per-group serializer (COMPRESSOR=pergroup):

1. Buckets the int6 GPTQ tensors by role (qo_bank, kv_bank, mlp_up_bank, mlp_down_bank, etc.) so similarly-distributed weights compress together.
2. For "hot" 2D groups (_tok_emb, attn.c_q, mlp.fc), runs an L1 nearest-neighbour similarity sort on rows before transposing — adjacent rows in the serialized stream are now numerically close, giving the entropy coder longer runs of small deltas. Permutation indices are stored as uint16 and brotli-compressed.
3. Compresses each group blob with lrzip -z -L 9 (ZPAQ context-mixing back-end). lrzip's long-range deduplication catches cross-tensor repetition that brotli's 24-bit window misses.
4. Falls back to brotli for the remainder (state-dict scaffolding, scales, LQER factors, gate tensors) and the code wrapper.

Net effect on this stack: ~280 KB smaller artifact than COMPRESSOR=brotli, at the cost of ~75 s of additional serialize time. The lrzip binary must be present on the system before the training script runs (e.g. install with apt-get install lrzip during instance setup). The script itself does not run apt-get; the Python subprocess.run wrapper just shells out to the already-installed lrzip binary.

Hyperparameter stack

9 greedy-validated overrides:

hparam	value	default
MLP_CLIP_SIGMAS	11.5	10.0
EMBED_CLIP_SIGMAS	14.0	20.0
WARMDOWN_FRAC	0.85	0.75
BETA2	0.99	0.95
TTT_BETA2	0.99	0.999
TTT_WEIGHT_DECAY	0.5	1.0
TTT_LORA_RANK	80	96
SPARSE_ATTN_GATE_SCALE	0.5	1.0
PHASED_TTT_PREFIX_DOCS	2500	2000

Each individually accepted on a strict greedy-keep rule (mean improvement vs current best stack) at fixed-step.

See records/track_10min_16mb/2026-04-27_SP8192_LQER_SparseGate_BOSSmearFix_9HpStack_1.0611/README.md for full architecture lineage and credits.

Test plan

• Trains within 600s wallclock on 8×H100 80GB SXM (4917-4945 steps achieved, ~121.7 ms/step mean)
• All 3 artifacts under 16 MB cap (max 15,907,550 B; min 15,897,259 B)
• TTT eval completes within 600s eval cap (max 508.8 s)
• 3-seed mean reproduced; per-seed numbers verified in attached logs
• SmearGate fix verified by code-diff audit; applied to both _forward_hidden and forward_ttt

🤖 Generated with Claude Code

[codemath3000]

Quick note: the initial commit, 612a1a9, was the only one with any code changes; the other two were README-only changes to clarify things and fix some AI hallucinations. In terms of evaluating when this PR was submitted relative to other PRs, the timing of the initial commit, not that of the most recent commit, should be used, since the initial commit contains all the code, logs, etc., and the other commits are just README changes. Thank you so much!

[aquariouseworkman]

apt-get install lrzip is required at runtime, thus this breaks rule 3

[okezue]

Independent reproduction (3 seeds)

Re-ran the stack on a fresh 8×H100 SXM pod (cu129/torch 2.9.1/FA3 cu129_torch291) with the env vars from this PR's hparam table plus the gates (`SMEAR_GATE_ENABLED=1 SPARSE_ATTN_GATE_ENABLED=1 EMBED_BITS=7 MIN_LR=0.1 GPTQ_RESERVE_SECONDS=0.5 PHASED_TTT_NUM_PHASES=3` etc.). 600 s wallclock, otherwise identical to this PR.

seed	pre-quant	quantized	post-TTT
42	1.06383	1.07237	1.05965
314	1.06450	1.07316	1.06041
999	1.06545	1.07398	1.06124
mean			1.06043

3-seed mean 1.06043 vs this PR's reported 1.06108 — within 1σ. Independent reproduction confirms the stack.

These three runs used `COMPRESSOR=brotli` and produced 16,112,007-byte artifacts (over the 16 MB cap; the BPB numbers are unaffected by compressor choice but the artifacts are technically non-compliant). One additional pergroup re-run with seed 42 produced a compliant 15,902,285-byte artifact at val_bpb 1.06052, matching seed 42's brotli value within run-to-run noise. (Couldn't get 3 pergroup seeds due to a string of RunPod capacity / image-pull failures over a 4-hour window — the locked volume on the original machine still has all the brotli logs and the s42 pergroup artifact saved.)

Net: the stack reproduces. The −0.019 BPB jump over PR #1493's 1.0810 is real on independent hardware.

[codemath3000]

apt-get install lrzip is required at runtime, thus this breaks rule 3

@aquariouseworkman Thanks for flagging — you're right that the README/PR wording was imprecise about when lrzip is needed, and I've fixed that. To clarify the substantive question:

The training/eval script never runs apt-get itself. The lrzip binary is installed once during instance setup and the script just shells out to the already-installed binary via subprocess.run. No apt-get, no network calls, and no external downloads occur during the 600 s training window or the 600s eval window.

The official FAQ explicitly authorizes external dependencies handled this way: "Yes, you're free to import any package or library you want… Just include a requirements.txt in your records folder and mention setup instructions in your README.md." Both are present (requirements.txt documents lrzip; the README has the install command).

For precedent: the current leaderboard SOTA (PR for the 2026-04-09 SP8192 record at 1.0810 BPB) installs FlashAttention 3 from a custom third-party wheel host (pip install flash_attn_3 --no-deps --find-links https://windreamer.github.io/...) — also an external download performed before training begins. lrzip via apt-get is actually a more conservative dependency than that: it pulls from the official Debian/Ubuntu package repos rather than from a single contributor's GitHub Pages site. If the standard FA3 setup is acceptable, this should be acceptable a fortiori.

Also worth noting that "rule 3" as numbered in the field guide is the author's paraphrase, not the official rule text. The literal official rule (FAQ) is the one I quoted above.

[aquariouseworkman]

apt-get install lrzip is required at runtime, thus this breaks rule 3

@aquariouseworkman Thanks for flagging — you're right that the README/PR wording was imprecise about when lrzip is needed, and I've fixed that. To clarify the substantive question:

The training/eval script never runs apt-get itself. The lrzip binary is installed once during instance setup and the script just shells out to the already-installed binary via subprocess.run. No apt-get, no network calls, and no external downloads occur during the 600 s training window or the 600s eval window.

The official FAQ explicitly authorizes external dependencies handled this way: "Yes, you're free to import any package or library you want… Just include a requirements.txt in your records folder and mention setup instructions in your README.md." Both are present (requirements.txt documents lrzip; the README has the install command).

For precedent: the current leaderboard SOTA (PR for the 2026-04-09 SP8192 record at 1.0810 BPB) installs FlashAttention 3 from a custom third-party wheel host (pip install flash_attn_3 --no-deps --find-links https://windreamer.github.io/...) — also an external download performed before training begins. lrzip via apt-get is actually a more conservative dependency than that: it pulls from the official Debian/Ubuntu package repos rather than from a single contributor's GitHub Pages site. If the standard FA3 setup is acceptable, this should be acceptable a fortiori.

Also worth noting that "rule 3" as numbered in the field guide is the author's paraphrase, not the official rule text. The literal official rule (FAQ) is the one I quoted above.

Strange, from what I can see in your current code, you would have to do one of the following to be valid (code review based, not based on your AI README post/information) :

a) re-run all three seeds with COMPRESSOR=brotli and reports those numbers
b) replaces the lrzip subprocess call with a pure-Python ZPAQ implementation that lives inside the artifact
c) maintainers would have to to add lrzip to the base eval image.

Evidence:

1. hard calls to lrzip train_gpt.py:2381-2390
2. hard routrer to lrzip # train_gpt.py:2700
3. requirements.txt:13:# System dep (apt): lrzip (used by per-group compressor) - pip thus cant install this as it isnt a py package (ref apt) thats why adding it to requirements.txt is not a fix, which is why the AI in your README has to call out apt-get install lrzip as a separate setup step, which is exactly what makes it a side dependency

[codemath3000]

Hi @aquariouseworkman, thanks for the detailed read. Walking through the three options:

(a) Re-run with COMPRESSOR=brotli — this would actually disqualify the submission on a different rule. @okezue's independent reproduction earlier in this thread used COMPRESSOR=brotli and got 16,112,007-byte artifacts — about 112 KB over the 16 MB cap. Pergroup is what brings this submission's artifacts under the cap (15.9 MB max). Re-running with brotli isn't a fix; it produces a non-compliant submission on artifact size.

(b) Pure-Python ZPAQ embedded in the artifact — no rule requires this. The official rule (FAQ on artifact size) prohibits "external downloads, training dataset access, or network calls during evaluation" — not subprocess shell-outs to OS utilities. lrzip is invoked only via subprocess.run against an already-installed binary; no network call and no download at runtime.

(c) Maintainers add lrzip to the base eval image — this is the standard workflow for the challenge, explicitly authorized by the FAQ:

"Yes, you're free to import any package or library you want… Just include a requirements.txt in your records folder and mention setup instructions in your README.md."

The FAQ uses "package or library" (not "pip package") and treats requirements.txt and "setup instructions in your README.md" as separate, both-acceptable declaration mechanisms. Pip-installability isn't a rule criterion. We declare lrzip in both places: requirements.txt documents it as a comment ("System dep (apt): lrzip"), and the README spells out the install command.

A point worth being explicit about: the rule's verb is "include a requirements.txt." That means have one, present it as a reference — not "every dependency must resolve automatically via pip install -r requirements.txt." The official README itself reinforces this on line 176, describing requirements.txt as "provided as a reference if you want to self-setup" — a reference document, not a self-executing manifest. Submission contents rule (line 227) similarly says "any other dependencies," with no restriction to pip-installable packages. So a requirements.txt that mentions lrzip in a comment alongside the README's install instructions satisfies the rule as written.

For precedent: the current leaderboard SOTA (2026-04-09, val_bpb 1.0810 by @bigbag) installs FlashAttention 3 from windreamer.github.io, a single contributor's GitHub Pages site, via a custom --find-links URL that does not resolve from PyPI. By the "if it isn't a normal pip-resolvable package, it's a side dependency" reading, that submission would also fail. It doesn't, because the FAQ allows external deps via README setup instructions — the exact same mechanism we're using for lrzip. lrzip from official Debian/Ubuntu apt repos is structurally a more conservative external-dep pattern than installing custom-built CUDA wheels from a GitHub Pages site.

The literal self-containment rule (FAQ on artifact size) is about runtime behavior during evaluation: "No external downloads, training dataset access, or network calls are allowed during evaluation." lrzip is invoked locally against an already-installed binary — no network, no download, no training-data access at runtime.

So to be explicit on the conclusion: the submission is valid as-is. The artifact is under 16 MB, the script makes no network calls or external downloads during evaluation, and the lrzip dependency is declared exactly the way the official FAQ asks external dependencies to be declared (requirements.txt + README setup instructions) — the same way every recent record has handled FA3, including the current leaderboard SOTA. None of options (a)–(c) is required; (a) would in fact create a new compliance violation.

[aquariouseworkman]

Hi @aquariouseworkman, thanks for the detailed read. Walking through the three options:

(a) Re-run with COMPRESSOR=brotli — this would actually disqualify the submission on a different rule. @okezue's independent reproduction earlier in this thread used COMPRESSOR=brotli and got 16,112,007-byte artifacts — about 112 KB over the 16 MB cap. Pergroup is what brings this submission's artifacts under the cap (15.9 MB max). Re-running with brotli isn't a fix; it produces a non-compliant submission on artifact size.

(b) Pure-Python ZPAQ embedded in the artifact — no rule requires this. The official rule (FAQ on artifact size) prohibits "external downloads, training dataset access, or network calls during evaluation" — not subprocess shell-outs to OS utilities. lrzip is invoked only via subprocess.run against an already-installed binary; no network call and no download at runtime.

(c) Maintainers add lrzip to the base eval image — this is the standard workflow for the challenge, explicitly authorized by the FAQ:

"Yes, you're free to import any package or library you want… Just include a requirements.txt in your records folder and mention setup instructions in your README.md."

The FAQ uses "package or library" (not "pip package") and treats requirements.txt and "setup instructions in your README.md" as separate, both-acceptable declaration mechanisms. Pip-installability isn't a rule criterion. We declare lrzip in both places: requirements.txt documents it as a comment ("System dep (apt): lrzip"), and the README spells out the install command.

A point worth being explicit about: the rule's verb is "include a requirements.txt." That means have one, present it as a reference — not "every dependency must resolve automatically via pip install -r requirements.txt." The official README itself reinforces this on line 176, describing requirements.txt as "provided as a reference if you want to self-setup" — a reference document, not a self-executing manifest. Submission contents rule (line 227) similarly says "any other dependencies," with no restriction to pip-installable packages. So a requirements.txt that mentions lrzip in a comment alongside the README's install instructions satisfies the rule as written.

For precedent: the current leaderboard SOTA (2026-04-09, val_bpb 1.0810 by @bigbag) installs FlashAttention 3 from windreamer.github.io, a single contributor's GitHub Pages site, via a custom --find-links URL that does not resolve from PyPI. By the "if it isn't a normal pip-resolvable package, it's a side dependency" reading, that submission would also fail. It doesn't, because the FAQ allows external deps via README setup instructions — the exact same mechanism we're using for lrzip. lrzip from official Debian/Ubuntu apt repos is structurally a more conservative external-dep pattern than installing custom-built CUDA wheels from a GitHub Pages site.

The literal self-containment rule (FAQ on artifact size) is about runtime behavior during evaluation: "No external downloads, training dataset access, or network calls are allowed during evaluation." lrzip is invoked locally against an already-installed binary — no network, no download, no training-data access at runtime.

So to be explicit on the conclusion: the submission is valid as-is. The artifact is under 16 MB, the script makes no network calls or external downloads during evaluation, and the lrzip dependency is declared exactly the way the official FAQ asks external dependencies to be declared (requirements.txt + README setup instructions) — the same way every recent record has handled FA3, including the current leaderboard SOTA. None of options (a)–(c) is required; (a) would in fact create a new compliance violation.

Yes .. my bad, this does appear valid. :) your now the #1 spot

[codemath3000]

Following up on the val_docs=10_000 default question with a more detailed answer.

This submission's training and evaluation data come from the published romeerp/parameter-golf-caseops-v1 HuggingFace dataset, downloaded directly via:

MATCHED_FINEWEB_REPO_ID=romeerp/parameter-golf-caseops-v1 \
python3 cached_challenge_fineweb.py \
  --variant sp8192_lossless_caps_caseops_v1_reserved --train-shards 80

The script used is the cached_challenge_fineweb.py bundled with PR #1729 (record path), which extends dataset_dir_for_variant with a 2-line fallback so non-numeric variant names like sp8192_lossless_caps_caseops_v1_reserved resolve. The canonical data/cached_challenge_fineweb.py on main only handles byte260 and sp<VOCAB_SIZE> numeric variants. The patch is purely a variant-name accommodation and doesn't change the download logic — the data still pulls from romeerp/parameter-golf-caseops-v1 directly.

The shipped prepare_caseops_data.py in our record folder is a reference data-prep script and was not run for this submission.

The romeerp dataset's manifest.json authoritatively documents the partition:

"stats": {
    "docs_val": 50000,
    "docs_train": 8181945,
    "files_val": 1,
    "files_train": 80,
    "tokens_val": 47853344,
    "tokens_train": 8000000058
}

This is the canonical 50K val docs / 8B train tokens setup. Our train_seed42.log reports train_shards: 80 and val_tokens: 47,851,520 — matching the manifest's files_train: 80 and tokens_val: 47,853,344 (the latter rounds to 47,851,520 via the standard (numel-1)//eval_seq_len*eval_seq_len + 1 truncation with eval_seq_len=2048).

The val partition this submission evaluated on is the canonical 50K val docs from romeerp's dataset.