# 187 Languages on Paper: An Audit of a Machine-Translated Multilingual Sensitive-Word Lexicon

Author: yxpil

Suggested arXiv categories: **cs.CL** (primary), cross-lists **cs.CY** and **cs.SE**.

A PDF is not included in this directory; build it with the commands below. The
orchestrator compiles the paper, so nothing here assumes a local TeX
installation.

## Abstract

A keyword list is the cheapest fallback for content moderation when a
classifier is unavailable, and a machine-translated keyword list is the cheapest
way to cover many languages. This paper audits one such artifact: a public
lexicon that advertises 187 languages, five categories, and 2,951,608 entries.
Every advertised language directory exists, every one of the five category files
is present, and the JSON parses without error. That is where the simple story
ends. Of the 2,951,608 entries, 2,685,296 are copies of the same 14,594-term URL
list, passed through unchanged in every language, and 218,960 are semantic
translations of 15,784 Chinese source terms in the 184 non-Chinese targets; the
remainder belongs to the three Chinese variants. Only 70.9% of the semantic
target entries differ from their source. The non-Chinese targets split into two
sharply separated groups: 113 languages with 0.71% identity on Han-containing
source entries, and 71 languages with 72.2%. The partial group is not a set of
71 different failures. Its members leave almost exactly the same source entries
untranslated---the pairwise Jaccard similarity to the modal failure set is
99.98% for political terms and 99.62% for porn terms---and only 18 distinct
failure signatures appear across the 71 languages. The pattern is consistent
with a pipeline that stopped on specific inputs, not with 71 languages that are
individually hard to translate. An expected-script check separates real script
coverage from superficially non-empty output: 13 languages exceed 98.5%, while
Japanese kana reach 64.6% (an all-kanji lower bound) and Serbian reaches 27.9%.
A duplicate analysis finds 5,933 excess duplicate occurrences, and 26,681
translated entries (17.2%) are shared verbatim across at least two languages,
rising to 46.2% for political terms. A 40-pair inspection sample of benign
advertising terms finds 27 accurate, 7 acceptable, 4 wrong-sense, and 2 other
outcomes. The paper does not print the sensitive source strings; it reports the
audit method, the coverage arithmetic, and what a maintainer should fix first.

## Abstract for the arXiv form (1,920-character limit)

The abstract printed in the paper is longer than the form accepts; paste this
version into the arXiv metadata field instead. It keeps every measured number
and drops only prose.

> A keyword list is the cheapest fallback for content moderation when a
> classifier is unavailable, and a machine-translated keyword list is the
> cheapest way to cover many languages. This paper audits one such artifact: a
> public lexicon that advertises 187 languages and 2,951,608 entries. Every
> advertised language directory and category file exists. Of these, 2,685,296
> are copies of the same 14,594-term URL list, passed through unchanged in
> every language, and 218,960 are semantic translations of 15,784 Chinese
> source terms in the 184 non-Chinese targets; the remainder belongs to the
> three Chinese variants. Only 70.9% of the semantic target entries differ
> from their source. The non-Chinese targets split into two sharply separated
> groups: 113 languages with 0.71% identity on Han-containing source entries,
> and 71 languages with 72.2%. The partial group is not 71 different failures:
> its members leave nearly the same source entries untranslated — the pairwise
> Jaccard similarity to the modal failure set is 99.98% for political terms
> and 99.62% for porn terms — and only 18 distinct failure signatures appear
> across the 71 languages. The pattern fits a pipeline that stopped on
> specific inputs, not 71 individually hard languages. An expected-script
> check separates real coverage from superficially non-empty output: 13
> languages exceed 98.5%, while Japanese kana reach 64.6% (an all-kanji lower
> bound) and Serbian reaches 27.9%. A duplicate analysis finds 5,933 excess
> occurrences, and 26,681 translated entries (17.2%) are shared verbatim
> across at least two languages, rising to 46.2% for political terms. A
> 40-pair sample of benign advertising terms finds 27 accurate, 7 acceptable,
> 4 wrong-sense, and 2 other outcomes. The paper prints no sensitive source
> strings; it reports the method, the coverage arithmetic, and what to fix
> first.

## Files

| Path | What it is |
|---|---|
| `main.tex` | The paper: 8 sections, 5 tables, 4 figures, 11 references. |
| `numbers.tex` | Generated macros for every quantity quoted in the prose. |
| `tables/` | Generated `inventory`, `groups`, `coverage`, `script`, and `sample` tables. |
| `refs.bib` | Bibliography (MT evaluation, dataset documentation, content-moderation governance). |
| `common.tex` | Copy of `../_shared/common.tex` (shared preamble). |
| `figures/fig_identity_distribution.pdf` | Identity-rate distribution over the 184 non-Chinese targets. |
| `figures/fig_identity_heatmap.pdf` | Per-language, per-category identity matrix, sorted by group. |
| `figures/fig_coverage_ecdf.pdf` | ECDF of target coverage per Han-containing source term. |
| `figures/fig_script_match.pdf` | Expected-script match rate per language. |
| `scripts/audit.py` | The audit: inventory, identity, expected script, duplicates, collisions, anomalies, sample. |
| `scripts/results/audit.json` | The audit's output. Every quantitative claim traces here. |
| `scripts/make_tables.py` | Renders `numbers.tex` and `tables/*.tex` from the JSON. |
| `scripts/make_figures.py` | Generates the four figures from the JSON (matplotlib, Agg, deterministic). |
| `README.md` | This file. |

## Build

```bash
pdflatex main
bibtex main
pdflatex main
pdflatex main
```

The result is `main.pdf`. The preamble uses only packages in a standard TeX Live
image: `times`, `amsmath`, `amssymb`, `amsthm`, `graphicx`, `booktabs`,
`multirow`, `array`, `algorithm`, `algpseudocode`, `listings`, `xcolor`,
`microtype`, `balance`, `geometry`, `hyperref`, plus `natbib` and `enumitem`,
which are loaded by `main.tex` after `common.tex`.

## Packaging for arXiv

Minimal tarball (what arXiv needs to compile):

```bash
tar czf sensitive-words-audit-arxiv.tar.gz main.tex numbers.tex refs.bib common.tex \
    tables/*.tex figures/*.pdf
```

Full artifact tarball (adds the audit and figure scripts):

```bash
tar czf sensitive-words-audit-artifact.tar.gz main.tex numbers.tex refs.bib common.tex \
    tables/*.tex figures/*.pdf scripts/audit.py scripts/make_tables.py \
    scripts/make_figures.py scripts/results/audit.json README.md
```

`common.tex` must stay byte-identical to the shared copy. Verify before
packaging:

```bash
cmp common.tex ../_shared/common.tex && echo identical
```

## Submission steps

1. Start a new arXiv submission at <https://arxiv.org/submit>.
2. Choose **cs.CL** as the primary category; add **cs.CY** and **cs.SE** as
   cross-lists.
3. Paste the title and the abstract exactly as above; author `yxpil`.
4. Upload `sensitive-words-audit-arxiv.tar.gz`. arXiv detects `main.tex` and
   runs `pdflatex` + `bibtex` itself.
5. Inspect the auto-built PDF: 4 figures, 5 tables, references rendered by
   `abbrvnat`, no overfull boxes that push text into the margin.
6. Select a license and submit. A first submission to cs.CL may require an
   endorsement; arrange it before uploading.
7. Keep `sensitive-words-audit-artifact.tar.gz` for the ancillary-files slot or
   for a companion repository; the paper refers to the audit JSON, not to a
   public URL beyond the repository.

## Reproduction

Requirements: Python 3.10+ and matplotlib for the figures; no network access and
no third-party packages beyond matplotlib. The audit resolves the source
repository by default to `../../_repos/multilingual-sensitive-words` relative to
this directory; pass `--repo` to point elsewhere. The audited commit is
`1a8de85` (2026-10-18).

```bash
# 1. Re-run the audit and rewrite the JSON.
python scripts/audit.py --repo ../../_repos/multilingual-sensitive-words \
    --out scripts/results/audit.json

# 2. Regenerate the macros and tables from that JSON.
python scripts/make_tables.py

# 3. Regenerate the four figures.
python scripts/make_figures.py
```

Expected audit output: `claimed=187 dirs=187 targets=184`,
`partial=71 high_coverage=113 zh_variants=2`, the four Han identity rates, and
`total entries (187)=2,951,608`. The JSON also records the per-language identity
rates, the failure signatures, the duplicate and cross-language-collision counts,
the anomaly inventory, and the 40-pair human sample.

### Provenance of the audit numbers

`numbers.tex`, `tables/*.tex`, and `scripts/results/audit.json` are generated
output, not hand-edited files. Two corrections were made before the final run.
First, an earlier draft quoted the URL and semantic target counts without
accounting for the 47,352 entries of the three Chinese variants (3 x 15,784), so
the parts did not sum to the reported total; the prose and the abstract now state
the partition explicitly, and the total is unchanged. Second, six generated
macro names ended in decimal digits (`\AnomGt100` and similar). TeX control words
contain letters only, so the digits terminated the name and the definitions
failed to parse; the macros were renamed (`\AnomOverHundred`, ...) in the
generator and in the prose, with no change to any value. Both defects were in the
presentation layer; the audit itself was not rerun for them.

## What the paper does not claim

- It does not introduce content moderation, keyword filtering, or machine
  translation; those are used unchanged and the relevant literature is cited.
- It does not evaluate translation quality in general. The human sample covers
  40 benign advertising pairs in seven languages; it is not a benchmark, and its
  categories are coarse.
- It does not treat identity as a perfect untranslated detector. A borrowed word
  or proper noun can legitimately be identical, which is why the paper reports
  the sample's false-positive estimate and pairs identity with a script check.
- It does not claim that every non-identical entry is a correct translation.
  Duplicate and collision counts show the opposite for a subset.
- It does not print, quote, or link to the moderated terms. Only benign
  advertising vocabulary from the sample table appears in the paper.
- It does not measure runtime detection performance, false-positive rates on
  real traffic, or the moderation decisions a deployed system would make.
- It does not claim that the lexicon is safe, complete, or suitable for
  deployment in any jurisdiction. It audits what the files contain.
- It does not claim that the 18 failure signatures prove a specific upstream
  cause. The pipeline is not public; the signatures are consistent with a
  stopping condition, and the paper says only that.
- It does not claim that the audit generalizes to other translated lexicons.
  One artifact at one commit is measured.
- It does not claim that the repository's own documentation is wrong. Where the
  documentation and the files disagree, both are reported.