# MIPIIL audit paper — arXiv submission package Author: yxpil Artefact under study: ## Title **A Transformer That Learned Only the Marginal: A Reproducibility Audit of a Published 19.2M-Parameter Edge Music Model** ## Suggested arXiv classification - Primary: `cs.SD` (Sound Audio) — the object is a published music-generation model and every measurement is made on MIDI token streams. - Cross-lists: `cs.CL` (Computation and Language, because the finding is a next-token-distribution result and the method is a causal Transformer decoder), `cs.LG` (Machine Learning, for the collapse-along-depth measurement and the trained-vs-untrained control), `cs.AI` (general readership for the generation-loop findings). `cs.SD` is the right primary: the paper's subject is an audio-domain artefact and its evidence is a MIDI file. If a moderator moves it, the natural fallback primary is `cs.LG` with `cs.SD` cross-listed — the audit method (total variation between next-token laws, entropy accounting, forward hooks) is domain-neutral and reads as ML. ## Abstract (verbatim) > MIPIIL is a published model presented as an edge-oriented music generator: a > 19,197,204-parameter Transformer over a byte-level BPE alphabet, with a model > card that promises unlimited-length real-time generation from a 375,000-file > MIDI corpus. This paper reports an audit of that artefact, and the audit is a > negative result about sequence modelling rather than about audio quality. The > checkpoint loads exactly as documented and its weights are finite and > meaningfully trained, yet the next-token distribution it defines does not > depend on its input: over contexts of 16 to 512 tokens taken from a real > Standard MIDI File, the mean pairwise total variation between next-token laws > is below 1.4×10⁻⁷, the number of distinct argmax positions is exactly one, and > the mutual information between context and prediction is 0.000 nats. An > untrained control of the same architecture is measurably more context > sensitive (2.12×10⁻² versus 9.08×10⁻⁸ total variation at the logits). Forward > hooks localise the collapse: the feed-forward branch norm grows to 541.6 while > the LayerNormed residual stream stays near √d, and cross-context spread > contracts by a factor of 209,963 with depth in the trained model against 2.0 > in the control. The consequence is that the artefact is, to within 5×10⁻⁶ bits > per token, a 276-entry histogram that costs 5.9613 bits, and it codes real MIDI > at 5.4461–5.4541 bits per token while a held-out unigram of the same file needs > 4.454. We also document the release mechanics that let this ship: greedy > decoding repeats one token forever, none of 40 sampled generations contains a > MIDI header, the published decoder raises on the checkpoint's own special > tokens, and the shipped generator never decodes a token at all — it maps token > ids to pitches by modular arithmetic. Because the model audited here is the > author's own release, the paper is offered as a verification protocol rather > than as a critique: three measurements, each under a minute of compute, would > have caught all of it. ## Abstract for the arXiv form (1,920-character limit) The abstract printed in the paper is longer than the form accepts; paste this version into the arXiv metadata field instead. It keeps every measured number and drops only prose. > MIPIIL is a published model presented as an edge-oriented music generator: a > 19,197,204-parameter Transformer over a byte-level BPE alphabet, with a > model card that promises unlimited-length real-time generation from a > 375,000-file MIDI corpus. This paper audits that artefact; the result is > negative about sequence modelling, not about audio quality. The checkpoint > loads as documented and its weights are finite and meaningfully trained, yet > the next-token distribution it defines does not depend on its input: over > contexts of 16 to 512 tokens from a real Standard MIDI File, the mean > pairwise total variation between next-token laws is below 1.4×10⁻⁷, and the > mutual information between context and prediction is 0.000 nats. An > untrained control of the same architecture is measurably more context > sensitive (2.12×10⁻² versus 9.08×10⁻⁸ total variation at the logits). > Forward hooks localise the collapse: the feed-forward branch norm grows to > 541.6 while the LayerNormed residual stream stays near √d, and cross-context > spread contracts by a factor of 209,963 with depth against 2.0 in the > control. The consequence is that the artefact is, to within 5×10⁻⁶ bits per > token, a 276-entry histogram that costs 5.9613 bits; it codes real MIDI at > 5.4461–5.4541 bits per token while a held-out unigram of the same file needs > 4.454. We also document how this shipped: greedy decoding repeats one token > forever, none of 40 sampled generations contains a MIDI header, the > published decoder raises on the checkpoint's own special tokens, and the > shipped generator never decodes a token at all — it maps token ids to > pitches by modular arithmetic. Because the model audited here is the > author's own release, the paper is offered as a verification protocol rather > than a critique: three measurements, each under a minute of compute, would > have caught all of it. ## BibTeX entry (after announcement) ```bibtex @misc{yxpil2026mipiil, author = {Yang, Daozhendong}, title = {A Transformer That Learned Only the Marginal: A Reproducibility Audit of a Published 19.2M-Parameter Edge Music Model}, year = {2026}, note = {arXiv:XXXX.XXXXX [cs.SD]} } ``` ## Files | path | role | | --- | --- | | `main.tex` | the paper, one column-balanced build, 8 pages | | `common.tex` | shared preamble, byte-identical to `../_shared/common.tex` | | `refs.bib` | 10 entries, all real and all cited | | `figures/fig_sweep.pdf` | total variation and bits/token across context lengths 16–512 | | `figures/fig_mutual.pdf` | per-context entropy vs entropy of the average, trained vs untrained | | `figures/fig_collapse.pdf` | cross-context spread per stage; FFN branch norm vs residual | | `figures/fig_bits.pdf` | five coders of the same MIDI file as horizontal bars | | `figures/fig_emission.pdf` | model emission law vs the file's own symbol histogram | | `figures/fig_notes.pdf` | the pitch distribution the shipped generator can reach | | `scripts/inspect_ckpt.py` | tensors, dtypes, parameter count, training/scheduler state, PE check | | `scripts/sampling_study.py` | after-SOS law, context dependence, greedy loop, throughput, control | | `scripts/context_sweep.py` | next-token TV and bits as a function of context length | | `scripts/conditional_value.py` | bigram/trigram baselines on the same file vs the network | | `scripts/midi_byte_tests.py` | round-trip, vocabulary capacity proof, cross-entropy, header rate | | `scripts/marginal_fit.py` | emission law vs corpus unigram, KL, held-out unigram, generator law | | `scripts/signal_path.py` | forward hooks per stage, branch/residual norms, contraction accounting | | `scripts/make_figures.py` | draws all six figures and writes `figure_support.json` | | `scripts/results/*.json` | raw output; every number in the paper traces here | | `README.md` | this file | Every quantitative macro in `main.tex` is a `\newcommand` with a trailing comment naming the JSON key it was copied from (`ckpt_audit.json` = CA, `sampling_study.json` = SS, `context_sweep.json` = CS, `conditional_value.json` = CV, `midi_byte_tests.json` = MB, `marginal_fit.json` = MF, `signal_path.json` = SP, `figure_support.json` = FS). Nothing in the text is a number that is not in a shipped evidence file. ## Build ```bash tectonic -X compile main.tex --keep-logs ``` or classically: ```bash pdflatex main && bibtex main && pdflatex main && pdflatex main ``` The build is clean: no undefined references, no error lines, 8 pages, one overfull box of 2.04 pt on the final page (vertical, cosmetic). ## Packaging for arXiv ```bash bash ../make_package.sh mipiil ``` The tarball stages `main.tex`, `common.tex`, `refs.bib`, `figures/`, `scripts/` and `scripts/results/`, and strips build artifacts (`*.aux *.log *.out *.bbl *.blg`, `__pycache__`), so a reader can re-run every number. The published checkpoint file (`MIPIIL.pt.real`, 232,559,932 B — model, optimizer and scheduler states in one pickle) is **not** shipped in the tarball — it is downloaded from the ModelScope repository named in the paper, and the paper states its byte length so a reader can check they got the same file. ## Reproduction Requirements: Python 3.11+, `torch` (2.14.1+cpu measured, 14 threads, CPU only), `numpy`, `tokenizers`, `mido`, `matplotlib`. No GPU is used or needed. Two inputs are needed, and the scripts look for them at fixed paths: 1. The released files — `MIPIIL.pt`, `MIPIIL_tokenizer.json`, `train_streaming.py`, `official_bpe_tokenizer.py`, `generate_midi_interactive.py` — in `%LOCALAPPDATA%/Temp/ms_mi` (Linux/macOS: edit the `TMP` constant at the top of each script). The repository's `MIPIIL.pt` entry is a 330-byte Git-LFS pointer; the weights are what it redirects to, and the scripts load that resolved file as `MIPIIL.pt.real`. 2. One real Standard MIDI File. The measurement scripts take the first `*.mid` found in `~/Downloads` sorted by name (`WCIBFCVBSCB.mid`, 3,497 tokens after tokenisation, is what every number in the paper was measured on). Put a different file there and the numbers change; the paper's Limitations section says so explicitly. ```bash python -X utf8 scripts/inspect_ckpt.py # reads the pickle only python -X utf8 scripts/sampling_study.py # includes the 256-step greedy loop python -X utf8 scripts/context_sweep.py python -X utf8 scripts/conditional_value.py python -X utf8 scripts/midi_byte_tests.py # 40 generations at two temperatures python -X utf8 scripts/marginal_fit.py python -X utf8 scripts/signal_path.py # trained model + untrained control, hooked python -X utf8 scripts/make_figures.py # only reads the JSONs above ``` Everything runs on CPU with the 19.2M-parameter model in fp32; the only timing measured and reported in the paper is the generation loop itself (256 greedy steps in 27.94 s, 0.1091 s/token, `sampling_study.json`). `signal_path.py` is the only script that instantiates an untrained control as well as the released checkpoint, so it is the slowest. ## Submission steps 1. New submission at . 2. Primary `cs.SD`, cross-lists `cs.CL`, `cs.LG`, `cs.AI`. 3. Paste the title and the abstract above; author `Daozhendong Yang`, the same identity used in the paper and here. 4. Upload the tarball; arXiv detects `main.tex` and runs `pdflatex` + `bibtex`. 5. Inspect the auto-built PDF against the local one (8 pages, 6 figures, 7 tables). 6. Choose a license and submit. ## Provenance and limits of the claims The artefact audited here is the paper author's own release. That is stated in the abstract, in the Introduction, and again in Limitations, and it changes what the paper is allowed to claim: - **It is a verification protocol, not an accusation.** The findings are the same class of finding a reviewer would raise on any release, and the closing section reduces them to three checks that cost under a minute each. - **Single-file measurement.** The context-dependence, coding-cost and baseline numbers were measured on one MIDI file that was available locally, not on the 375,000-file training corpus. This bounds the *coding* claims only; it does not weaken the marginality claim, because that claim is about the model's output law over arbitrary contexts and is checked at six lengths with 512 in-window positions. - **Architecture is reconstructed, not asserted.** `main.tex` describes the network as the released `train_streaming.py` defines it, and the checkpoint loads into it with `strict=True`. The paper says plainly that a different definition of the same name would change the hook analysis. - **Mechanism is partial.** The forward-hook accounting explains four of the five observed decades of contraction with the context-free FFN branch alone and leaves an unexplained factor of 18.1 in the trained model (0.14 in the control). The paper reports the residual gap instead of closing it rhetorically. - **No retraining was performed.** Nothing here shows that the architecture *cannot* learn; it shows that this checkpoint, at this training state (epoch 1, optimizer step 10,801, reported loss 4.411172 nats), does not. - **Release mechanics are facts, not interpretation.** The LFS-pointer indirection, the `decode()` exception on the checkpoint's own special tokens, the 0/40 MIDI-header generation rate, the `base_model: Pillow_tiny_chat` / `repackage` relation, and the generator's unpublished `streaming_checkpoint.pt` are all read off the shipped files, and the paper quotes the file and line for each. ## What the paper does not claim - It does not claim the model sounds bad. No listening evaluation was performed; the paper says so and confines itself to sequence-level measurement. - It does not claim the corpus claim is false. The 375,000-file figure is reported as a card claim that was not testable from the release. - It does not claim the architecture is unsuitable for MIDI. The control experiment shows an *untrained* model of the same architecture is more context-sensitive than the released one, which is a statement about this optimisation trajectory. - It does not claim greedy decoding was the intended sampling configuration. Temperature sampling is measured too (22, 63 and 80 distinct pitches per 128 tokens at 0.6, 1.0 and 1.5), and the finding is that the distribution behind all three is context-independent.