# Small Interpreter, Wide Boundary: A Measurement of Error Classification, Capability Stubs, and the C-ABI Type Envelope in PILScript

Daozhendong Yang, School of Intelligent Science and Engineering,
Yunnan Technology and Business University, Kunming, China — <hi@yxpil.com>

Paper about **PILScript**, a ~2.5k-line JavaScript-style scripting language written in
Rust: lexer with Go-style automatic semicolon insertion, recursive-descent parser,
tree-walking interpreter, 21 builtins, a four-code error taxonomy with panic
isolation behind a 256 MiB evaluation thread, two capability flags
(`--no-ffi`, `--no-fs`), and a `dlopen`-based FFI that dispatches through a
63-arm match over two machine register slots.

## arXiv classification

- **Primary:** `cs.PL` — Programming Languages
- **Cross-lists:** `cs.SE` (Software Engineering), `cs.CR` (Cryptography and Security)

The same classification is stated in the header comment of `main.tex`.

## Abstract (plain-text rendering)

A scripting language embedded in a host application is usually sold on one of two
properties: it is small, or it is contained. PILScript — written in Rust, 2,465
source lines, one third-party dependency — advertises the first and, in its own
documentation, gestures at the second: scripts are described as untrusted input,
capabilities can be switched off with `--no-ffi` and `--no-fs`, panics are isolated
behind four error codes, and a single expression, `dlopen`, reaches any
`extern "C"` symbol in any dynamic library the host can load. This paper measures
where each of those mechanisms actually stops. The harness drives the released
command line only — no interpreter code is modified, no internal function is
called — over 252 generated programs, a 14-case automatic-semicolon matrix, an
arity ladder, and 11 deliberately misdeclared `extern "C"` signatures, each in its
own process. The error contract holds: no generated program produced an
unclassified error or an abort, and the 1500-frame recursion guard fires exactly
where documented. The capability flags are call-time stubs, and they work: each
flag denies exactly the builtins it names, though the disabled names remain bound
as functions. The boundary is the story. Of 11 misdeclared signatures, the runtime
notices 2; 8 run to completion, including reading a `void` function's return
register as an integer and calling a three-argument function through a
two-argument declaration; and 1 kills the process when a small integer is passed
where the library dereferences a pointer. Separately, there is no step, time or
memory budget: 6 of 7 runaway programs were still running when the harness killed
them. Finally, replaying the project's own CI in its own step order fails 3 tests
that pass once a debug build exists, because the FFI tests resolve their library
under `target/debug` while CI builds only the release profile.

## The two harness artifacts, and why they are disclosed

**`scripts/ffi_probe/`** is a `cdylib` we wrote. The released `demo_lib` exports
six functions with at most three arguments, which cannot answer where the
dispatcher's arity limit is, what reading a `void` return as an integer yields, or
what passing an integer to a dereferencing parameter costs. The probe library
exports functions of arity 0–6, one of every return type, one that dereferences
its argument, one that leaks and one that overflows. It is compiled into
`scripts/build/ffi_probe/` by cargo and is **not part of the release**; every
claim in §6.4 and §6.5 depends on it, and the paper says so. The released
`demo_lib` independently reproduces the `str`, `i64`, `f64` and mixed-slot cases.

**`scripts/build/<label>/`** holds mutated copies of the repository for §6.9: five
one-line changes, each anchored on an exact source line (remove the recursion
limit, make either capability flag a no-op, shrink the evaluation stack to 1 MiB,
turn the arity rejection into an empty message). `build_pilscript.py` copies the
repository and never writes into it.

Everything else — the language probes, the ASI matrix, the fuzz corpus, the
resource sweeps, the capability matrix, the benchmarks — runs the released
`pilscript` binary through its documented command line. No interpreter code is
patched, no internal function is called, and the harness needs no Rust build of
its own to produce those numbers.

## What is measured

| Scenario | Fixture | What it answers |
| --- | --- | --- |
| `ci` | scratch tree, CI's five steps | does the published pipeline pass here |
| `fresh_ci` | clean copy, release profile only | do the FFI tests pass in CI's own order |
| `language` | 16 semantic probes | closures, recursion, indices, coercion, short-circuit |
| `asi` | 14 two-line programs | which line-ending tokens terminate a statement |
| `errors` | 252 generated programs + 9 designed | does the four-code taxonomy hold |
| `resources` | depth sweep + 7 runaway programs | what the interpreter actually bounds |
| `ffi` | probe library: 8 returns, 7 arities, 11 misdeclarations, 2 leak shapes | reach and safety envelope |
| `capabilities` | 5 builtins × 4 flag combinations | what `--no-ffi` / `--no-fs` deny |
| `throughput` | 6 benchmarks vs CPython, best of three | speed, honestly mixed |
| `claims` | README.md and TESTING.md vs the tree | documented vs counted |
| `ablations` | 5 one-line mutations | whether the shipped suite notices |

## File manifest

| Path | Contents |
| --- | --- |
| `main.tex` | Paper source: 9 pages, two columns, six figures, sixteen tables |
| `refs.bib` | Bibliography (specs, published papers, and named artifacts as misc) |
| `common.tex` | Shared preamble, copied from `papers/_shared/common.tex` |
| `numbers.tex` | Generated `\newcommand` definitions for every inline number |
| `tables/*.tex` | Generated table environments (artifact, language, asi, errors, fuzz, recursion, runaway, memory, ffiret, ffiarity, ffisafe, ffileak, caps, bench, ci, ablations) |
| `figures/fig_flow.pdf` | Evaluation path and the three mechanisms |
| `figures/fig_taxonomy.pdf` | Corpus outcomes and designed-input codes |
| `figures/fig_resources.pdf` | Recursion RSS and runaway time-to-stop |
| `figures/fig_ffi.pdf` | Arity ladder and misdeclaration outcomes |
| `figures/fig_bench.pdf` | Ratio against CPython |
| `figures/fig_claims.pdf` | Documented vs counted, and the CI steps |
| `scripts/pil_lab.py` | CLI runner, RSS sampler, probe-library builder, C-compiler probe |
| `scripts/build_pilscript.py` | Scratch-tree copy, both profiles, the five ablations |
| `scripts/measure.py` | Scenario driver, writes the JSON artifact |
| `scripts/audit_repo.py` | Static audit (lines, builtins, dispatcher arms, deps, doc claims) |
| `scripts/make_tables.py` | Generates `numbers.tex` and `tables/*.tex` |
| `scripts/make_figures.py` | Generates `figures/*.pdf` |
| `scripts/ffi_probe/` | The disclosed C-ABI probe library |
| `scripts/results/pil_measure.json` | Every measured number in the paper |
| `scripts/data/audit.json` | Static audit artifact |
| `scripts/build/builds.json` | Build evidence for all six trees |
| `README.md` | This file |

## Build

Requires a TeX distribution with `pdflatex`, `bibtex`, `natbib` and `abbrvnat.bst`
(TeX Live or MiKTeX), or Tectonic. The paper uses `xurl`, `enumitem` and `array`.

```bash
tectonic -r 12 main.tex
# or
pdflatex main && bibtex main && pdflatex main && pdflatex main
```

## arXiv submission

Upload the contents of `dist/pilscript.tar.gz` (built with
`bash ../make_package.sh pilscript`) at <https://arxiv.org/submit>. The archive
contains `main.tex`, `refs.bib`, `common.tex`, `numbers.tex`, `tables/` and
`figures/`; include the generated `main.bbl`. Choose `cs.PL` as the primary
category and `cs.SE` plus `cs.CR` as cross-lists. Suggested comments: *9 pages,
6 figures, 16 tables; measurement paper with a reproduction script set*.

## Reproduce the measurements

Requirements: Windows (the harness samples child RSS through `psutil`), Python
3.13 with matplotlib and psutil, a Rust toolchain, and a clone of the repository.
A C compiler is optional: if `gcc`/`clang`/`cc` is on PATH the harness also
compiles a small C library and re-runs the FFI cases through a genuinely
non-Rust ABI partner.

```bash
# 1. static audit (read-only over the repository)
python scripts/audit_repo.py

# 2. scratch trees: base (both profiles) and the five ablations
python scripts/build_pilscript.py --repo /path/to/PILScript --all

# 3. every measurement; --ci replays the CI steps and the fresh-tree variant,
#    --ablations runs cargo test inside each mutated tree
python scripts/measure.py --ci --ablations

# 4. regenerate every number, table and figure, then rebuild
python scripts/make_tables.py
python scripts/make_figures.py
python ../_shared/check_paper.py .
```

A full run takes about six minutes, most of it in cargo. The runaway programs are
killed by the harness after 10–20 s each; nothing on disk is harmed, and the
capability probes write only into `scripts/results/run/`.

## Key results

- Taxonomy: 252 generated programs — 89 ran, 87 `E2001`, 54 `E3001`, 18 `E1001`,
  4 hit the harness timeout; **0 unclassified, 0 aborts**. One designed input
  (`"\q"`, an undefined escape) is accepted silently.
- Semantics: 16/16 probes behave as documented; 14/14 automatic-semicolon cases
  parse and evaluate to the hand-computed value.
- Bounds: the recursion guard fires between 1,499 and 1,500 frames at ~7 MiB;
  `while (true) {}`, `for (;;) {}`, string doubling, array and object growth and a
  memoisation-free `fib(38)` were all still running when killed (1 of 7 stopped). There is no fuel, time or memory budget.
- FFI reach: 8/8 return types round-trip; arity 0–5 accepted, 6 rejected with a
  message naming the limit; the dispatcher is 63 slot arms.
- FFI safety: of 11 misdeclared signatures, 2 are caught as clean errors, 8 run
  to completion (a `void` function read as `i64` yields a 15-digit number; a
  3-argument function called through a 2-argument declaration returns a
  plausible-looking garbage sum), and 1 aborts the process with an access
  violation.
- FFI allocation: leaking 512 MiB of untouched pages moves reported peak RSS by
  ~0 MiB (3.8 MiB peak); writing one byte per page makes it real (515.9 MiB).
- Capabilities: `--no-ffi` denies `dlopen` only; `--no-fs` denies the four
  filesystem builtins only; the disabled names stay bound as functions.
- Throughput vs CPython: 3.4× slower on fib, 5.5× on an integer loop, 0.57×
  (faster) on string concatenation, 22× slower building a 50k-key object; ~12 ms
  process start-up.
- CI: clippy `-D warnings` and both test runs pass on our tree (29 tests);
  `cargo fmt --check` fails and is not in the CI file at all; in a fresh tree
  following CI's own order, 3 FFI tests fail until a debug build exists.
- Docs: README says "about 1500 lines", the library target is 2,465; README says
  22 integration tests, TESTING.md says 29, the tree has 29;
  `hook_lifecycle_tests.rs` tests FFI registration and the word "hook" appears
  zero times in `src/`.
- Ablations: 4 of 5 mutations are caught by the shipped suite — 2 of those because
  the test *binary* stack-overflows (exit 0xC00000FD) rather than because an
  assertion failed — and 4 by the CLI probe; the union is 5. The `>5 arguments`
  arity cap is the one the suite does not notice at all.

## Honest limitations of this paper

One machine, one profile; the register-class results are ABI-dependent. The probe
library is ours. 252 template-mutated programs is a spot check, not a fuzzing
campaign. CPython is an availability-driven baseline. RSS is a 1 ms lower bound.
We replayed the published workflow file, not the project's CI history, which a
single-commit mirror does not expose. See §8 of the paper.
