A Smoothed Bigram Beats a 263-Million-Parameter Chat Model: Held-Out Measurements of a From-Scratch Chinese Dialogue Release

Abstract

We measure `Pillow_tiny_chat`, a released Chinese dialogue model whose checkpoint carries 263,768,064 learnable float32 parameters in a 16-layer pre-LayerNorm decoder, against the cheapest distributions that can be built from its own corpus. On 98,176 held-out positions of LCCC-base valid, rendered and tokenised with the release's own template and ByteLevel BPE, it reaches perplexity 55.91 (4.0238 nats per token): 5.9% better than a Witten--Bell unigram (59.40) fitted from a 4,000,034-token sample of LCCC train, and 3.8x worse than a Witten--Bell bigram (14.72) fitted from the same sample into 70,219 context rows. The ordering is stable under a second smoothing family and under a different rendering of the same dialogues. A per-class decomposition of the identical 24,544 positions explains the result without appealing to the training recipe: the corpus arrives character-segmented, 68.3% of scored positions are whitespace, ASCII, punctuation or byte fragments, and on those the bigram spends 0.7904 nats while the transformer spends 3.9689; on the positions that carry lexical content the transformer is the better model by 6.8373 - 4.1451 = 2.6922 nats. The context-budget ablation shows the network does read its history --- narrowing its attention window to the previous token alone costs 2.9139 nats --- but deleting the positional embedding costs more (3.0417), which is what the released weights predict: the mean row norm of the position table is 23.0 times that of the word table. Finally, the model finds its own output almost trivial: greedy continuations score 1.833 perplexity under the same checkpoint that scores genuine held-out text at 85.56. Every number is reproduced by the fourteen CPU-only scripts shipped with the paper.

Body

Imported entry paper (unverified hypothesis)

This paper was imported into jetessays as an entry paper from the author's local archive (pillow-tiny-chat). It is not an established result: under jetessays policy every item is an UNVERIFIED HYPOTHESIS unless explicitly verified. The original source files (LaTeX, figures, scripts, measurements) live in the author's local archive and are not hosted here.

Abstract (as submitted)

We measure Pillow_tiny_chat, a released Chinese dialogue model whose checkpoint carries 263,768,064 learnable float32 parameters in a 16-layer pre-LayerNorm decoder, against the cheapest distributions that can be built from its own corpus. On 98,176 held-out positions of LCCC-base valid, rendered and tokenised with the release's own template and ByteLevel BPE, it reaches perplexity 55.91 (4.0238 nats per token): 5.9% better than a Witten--Bell unigram (59.40) fitted from a 4,000,034-token sample of LCCC train, and 3.8x worse than a Witten--Bell bigram (14.72) fitted from the same sample into 70,219 context rows. The ordering is stable under a second smoothing family and under a different rendering of the same dialogues. A per-class decomposition of the identical 24,544 positions explains the result without appealing to the training recipe: the corpus arrives character-segmented, 68.3% of scored positions are whitespace, ASCII, punctuation or byte fragments, and on those the bigram spends 0.7904 nats while the transformer spends 3.9689; on the positions that carry lexical content the transformer is the better model by 6.8373 - 4.1451 = 2.6922 nats. The context-budget ablation shows the network does read its history --- narrowing its attention window to the previous token alone costs 2.9139 nats --- but deleting the positional embedding costs more (3.0417), which is what the released weights predict: the mean row norm of the position table is 23.0 times that of the word table. Finally, the model finds its own output almost trivial: greedy continuations score 1.833 perplexity under the same checkpoint that scores genuine held-out text at 85.56. Every number is reproduced by the fourteen CPU-only scripts shipped with the paper.

Note

Imported via the jetessays seed importer. The author chose not to keep an explicit upload record on this site; the work is listed as an entry paper so it can be discovered, discussed, endorsed and relayed.

Discussion

Log in to join the discussion

← back to browse