# Pillow_tiny_chat audit paper — arXiv submission package

Author: yxpil
Artefact under study: <https://modelscope.cn/models/Yxpilow/Pillow_tiny_chat>

## Title

**A Smoothed Bigram Beats a 263-Million-Parameter Chat Model: Held-Out
Measurements of a From-Scratch Chinese Dialogue Release**

## Suggested arXiv classification

- Primary: `cs.CL` (Computation and Language) — the object is a published
  Chinese dialogue language model and every measurement is a next-token
  cross-entropy on a standard chat corpus.
- Cross-lists: `cs.LG` (Machine Learning, because the method is a controlled
  comparison against Witten--Bell and Dirichlet estimators plus an attention-band
  ablation), `cs.AI` (general readership for the model-card verification angle).

`cs.CL` is the right primary: the claim being tested is a property of a
next-token distribution over text, and the corpus (LCCC-base) is the standard
one. If a moderator moves it, `cs.LG` with `cs.CL` cross-listed is the natural
fallback — the audit protocol (fixed windows, fixed template, teacher-forced
nats, per-class decomposition) transfers to any released checkpoint.

## Abstract (verbatim)

> We measure `Pillow_tiny_chat`, a released Chinese dialogue model whose
> checkpoint carries 263,768,064 learnable float32 parameters in a 16-layer
> pre-LayerNorm decoder, against the cheapest distributions that can be built
> from its own corpus. On 98,176 held-out positions of LCCC-base valid, rendered
> and tokenised with the release's own template and ByteLevel BPE, it reaches
> perplexity 55.91 (4.0238 nats per token): 5.9% better than a Witten--Bell
> unigram (59.40) fitted from a 4,000,034-token sample of LCCC train, and 3.8x
> worse than a Witten--Bell bigram (14.72) fitted from the same sample into
> 70,219 context rows. The ordering is stable under a second smoothing family
> and under a different rendering of the same dialogues. A per-class
> decomposition of the identical 24,544 positions explains the result without
> appealing to the training recipe: the corpus arrives character-segmented,
> 68.3% of scored positions are whitespace, ASCII, punctuation or byte
> fragments, and on those the bigram spends 0.7904 nats while the transformer
> spends 3.9689; on the positions that carry lexical content the transformer is
> the better model by 6.8373 - 4.1451 = 2.6922 nats. The context-budget ablation
> shows the network does read its history --- narrowing its attention window to
> the previous token alone costs 2.9139 nats --- but deleting the positional
> embedding costs more (3.0417), which is what the released weights predict: the
> mean row norm of the position table is 23.0 times that of the word table.
> Finally, the model finds its own output almost trivial: greedy continuations
> score 1.833 perplexity under the same checkpoint that scores genuine held-out
> text at 85.56. Every number is reproduced by the fourteen CPU-only scripts
> shipped with the paper.

## BibTeX entry (after announcement)

```bibtex
@misc{yang2026pillowtinychataudit,
  title         = {A Smoothed Bigram Beats a 263-Million-Parameter Chat Model:
                   Held-Out Measurements of a From-Scratch Chinese Dialogue
                   Release},
  author        = {Daozhendong Yang},
  year          = {2026},
  eprint        = {XXXX.XXXXX},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://modelscope.cn/models/Yxpilow/Pillow_tiny_chat}
}
```

## Files

| path | role |
| --- | --- |
| `main.tex` | the paper, eleven pages, two-column, nine tables and seven figures |
| `common.tex` | shared preamble, byte-identical to `../_shared/common.tex` |
| `refs.bib` | 20 entries, all real and all cited |
| `figures/fig_ladder.pdf` | the six-estimator ladder on a log scale |
| `figures/fig_context.pdf` | mean nats per token as the attention band shrinks |
| `figures/fig_position.pdf` | loss against position inside the 768-token window |
| `figures/fig_classes.pdf` | share of positions vs share of cross-entropy, per class |
| `figures/fig_vocab.pdf` | touched vs untouched embedding-row norms; wpe norm by bin |
| `figures/fig_freerun.pdf` | the checkpoint scoring its own greedy and sampled output |
| `figures/fig_replies.pdf` | six sampler continuations of one prompt, in the source script |
| `scripts/inspect_checkpoint.py` | state-dict inventory, parameter accounting, buffer shapes |
| `scripts/tokenizer_stats.py` | vocabulary, merges, tokens-per-character probes |
| `scripts/scan_train.py` | LCCC structural census (dialogues, turns, separators) |
| `scripts/prep_eval.py` | the 128 non-overlapping 768-token evaluation windows |
| `scripts/ppl_eval.py` | held-out teacher-forced perplexity of the checkpoint |
| `scripts/baselines.py` | Dirichlet unigram and bigram on the same train sample |
| `scripts/trigram_ladder.py` | Witten--Bell ladder, normalisation checks, per-class costs |
| `scripts/sens_eval.py` | the same three estimators on unspaced renderings |
| `scripts/weight_forensics.py` | row norms, vocabulary coverage, cosine structure |
| `scripts/context_ablation.py` | band-$k$ attention masks, positional and identity ablations |
| `scripts/free_run.py` | self-likelihood of greedy/sampled output, shipped stop rule |
| `scripts/gen_timing.py` | throughput, diversity probes, sampler behaviour |
| `scripts/template_roundtrip.py` | the extraction assumption made by `chat.py` |
| `scripts/make_figures.py` | draws all seven figures and writes `figure_support.json` |
| `scripts/results/*.json` | raw output; every number in the paper traces here |
| `scripts/results/*.log` | stdout of the same runs, including the wall-clock fields |
| `README.md` | this file |

Every quantitative macro in `main.tex` is a `\newcommand` whose trailing comment
names the JSON key it was copied from (`checkpoint_inventory.json` = CI,
`weight_forensics.json` = WF, `tokenizer_stats.json` = TS,
`perplexity_model.json` = PM, `perplexity_ngram_ladder.json` = NL,
`perplexity_baselines.json` = PB, `perplexity_sensitivity.json` = SE,
`lccc_census.json` = LC, `eval_windows.json` = EW, `generation.json` = GE,
`template_roundtrip.json` = TR, `context_ablation.json` = AB,
`free_run.json` = FR, `figure_support.json` = FS). Nothing in the text is a
number that is not in a shipped evidence file; derived quantities show their
arithmetic inline.

## Build

```bash
tectonic -X compile main.tex --keep-logs
```

or classically:

```bash
pdflatex main && bibtex main && pdflatex main && pdflatex main
```

The build is clean: no undefined references, no `\todo`, no overfull line wider
than 2.2pt (a 1.5pt overfull box on the second-to-last page, inside the bottom
margin).

## Packaging for arXiv

```bash
cd .. && ./make_package.sh pillow-tiny-chat
```

The script stages the whole directory, then strips top-level `.aux`, `.log`,
`.out`, `.bbl`, `.blg` and any `__pycache__`; `refs.bib` stays in the tarball and
arXiv runs BibTeX itself, while the `scripts/results/` evidence tree and the
`scripts/*.log` wall-clock captures survive because the strip is top-level only.
The result is `../dist/<name>.tar.gz`. Submit it with `main.tex` as the root
document; `common.tex` is included so the tarball is self-contained.

## Reproduction

The measurements need the released artefact and LCCC-base, neither of which is
redistributed here:

| input | where it comes from |
| --- | --- |
| `yinghe_tiny.pt` | the ModelScope release, 1,092,894,695 bytes, 213 keys |
| `chat.py`, `book-vocab.json`, `book-merges.txt`, `configuration.json` | the same release |
| `train.txt`, `valid.txt`, `test.txt` | LCCC-base (`thu-coai/lccc`), valid used for scoring, train sampled for the baselines |

Each script takes its paths from constants at the top of the file. Order and
cost, all on one CPU (PyTorch 14 threads, `OMP_NUM_THREADS=6` where the script
loads the model):

```bash
python -X utf8 scripts/inspect_checkpoint.py    # seconds
python -X utf8 scripts/tokenizer_stats.py       # seconds
python -X utf8 scripts/scan_train.py            # ~1 minute, streams train.txt
python -X utf8 scripts/prep_eval.py             # writes eval_windows.json
python -X utf8 scripts/ppl_eval.py              # 417 s shipped in the JSON
python -X utf8 scripts/baselines.py             # ~2 minutes
python -X utf8 scripts/trigram_ladder.py        # ~4 minutes, 4M-token sample
python -X utf8 scripts/sens_eval.py             # ~7 minutes, two passes
python -X utf8 scripts/weight_forensics.py      # ~2 minutes
python -X utf8 scripts/context_ablation.py      # 790 s summed over 11 masks
python -X utf8 scripts/free_run.py              # ~3 minutes, 32-token generations
python -X utf8 scripts/gen_timing.py            # ~4 minutes
python -X utf8 scripts/template_roundtrip.py    # seconds
python -X utf8 scripts/make_figures.py          # seconds, needs matplotlib
```

`trigram_ladder.py` carries the two self-checks the paper cites: each
Witten--Bell conditional is verified to sum to 1.0 over the vocabulary, and its
per-class counts are produced by the same classifier function that
`context_ablation.py` uses, copied verbatim, so the two tables are comparable.
`free_run.py` fixes `torch.manual_seed(0)`, which is what makes the sampled rows
replayable.

## Submission steps

1. `tectonic -X compile main.tex --keep-logs` from this directory; confirm zero errors.
2. `python ../lint.py pillow-tiny-chat`; confirm zero errors.
3. `../make_package.sh pillow-tiny-chat`; upload the tarball at
   <https://arxiv.org/submit/> (interface: <https://arxiv.org/>).
4. Root document `main.tex`, `cs.CL` primary, cross-list `cs.LG` and `cs.AI`.
5. Comments field: state that the artefact audited is the author's own release
   and that the paper ships the measurement scripts.

## Provenance and limits of the claims

- The perplexity figure is teacher-forced on 128 fixed, non-overlapping 768-token
  windows of LCCC-base valid, positions 2 through 768 scored, 98,176 positions.
  The windows are frozen in `eval_windows.json` so the ablation and the ladder
  score the identical stream.
- The baselines are fitted from a 4,000,034-token sample of LCCC train (the
  sample rule is in `trigram_ladder.py`) and are deliberately given the same
  alphabet, the same template and the same evaluation positions as the network.
- The band-$k$ ablation, the two input ablations, the per-class decomposition and
  the position curve use the first 32 of those windows (24,544 positions). The
  subset's full-context mean is 4.0248 nats against 4.0238 on all 128, a
  0.002-nat difference, which the Limitations section states.
- The class labels come from decoding each target id with the shipped tokenizer;
  `cjk_word` is empty in this corpus, and the paper explains why (the corpus
  arrives character-segmented) rather than hiding it.
- Wall-clock numbers are what the shipped JSON recorded on the author's machine
  and are not claimed to be portable.

## What the paper does not claim

It does not claim the release is worthless, that the training recipe caused any
of this, or that a bigram is a better chatbot. It does not retrain anything, does
not evaluate on human judgements, and does not report any number for a
configuration it did not run. Every card claim it addresses is quoted from the
published `README.md` of the artefact and answered by a measurement in the
claims table.