# Contract Drift in a Multi-Protocol LLM Gateway: Black-Box Measurement of PLocalSwitch v0.2.27 Daozhendong Yang, School of Intelligent Science and Engineering, Yunnan Technology and Business University, Kunming, China — Paper about **PLocalSwitch**, a Tauri desktop application that embeds an OpenAI-compatible LLM gateway written in Rust: four inbound protocol vocabularies, `ProtocolKind` adapters to five natively-translating upstreams plus eight pass-throughs, an alias/group/weight router, a retry chain with a protocol sniffer, an in-memory cache, a SQLite trace store and client ledger, and an HTTP management plane. The paper measures the shipped binary against the contracts the same binary advertises at start-up, and reports the divergences. ## arXiv classification - **Primary:** `cs.SE` — Software Engineering - **Cross-lists:** `cs.DC` — Distributed, Parallel, and Cluster Computing; `cs.CR` — Cryptography and Security The same classification is stated in the header comment of `main.tex`. ## Abstract (plain-text rendering) Multi-protocol LLM gateways promise to be many things at once: an OpenAI-compatible front door, a translator to vendor-native APIs, a failover layer, and a billing system. Each promise is a contract, and each contract is testable. This paper measures one such gateway, PLocalSwitch v0.2.27 at commit b9ce157 (8,642 Rust lines across 89 files), against the contracts it advertises, using a black-box harness that runs the shipped desktop binary against 27 local mock upstreams and exports the gateway's own SQLite ledger. The data plane is genuinely good: all 20/20 non-streaming combinations return byte-identical text. The control and envelope planes are not. Only 5/20 streaming responses use the envelope the client's protocol expects — Anthropic and Ollama clients receive OpenAI `data:` chunks, and a Gemini `streamGenerateContent` call receives a single non-streaming JSON body — and the `model` field is empty in 16/20 non-streaming responses. Tool calls vanish in both directions on the Gemini ingress. Configured retry policy is not consulted: a `false` switch still retries, and a sub-attempt budget of 6 becomes 24 upstream POSTs. A mid-stream premature close is reported as HTTP 200 with a synthetic `[DONE]` and zero billed tokens. Per-alias cache opt-out is ignored, cache purge is a no-op, and the cache statistics endpoint reports 0 entries while the cache is serving hits. The management plane answers 0 where the same process wrote 91 traces. We classify these into four kinds of drift — envelope, vocabulary, policy and telemetry — and report the negative results, including a headless build configuration that does not compile (12 errors). ## What is measured The harness starts 27 threaded mock upstreams on loopback (five protocol-faithful providers plus failure fixtures), generates a lab configuration from the shipped `gateway.yaml` template, launches the **release desktop binary** as a subprocess, issues requests with `urllib`, and then reads the gateway's own SQLite file. Every observation lands in `scripts/results/plocal_measure.json`; the paper's tables and figures are generated from that file. | Scenario | Fixture | What it answers | | --- | --- | --- | | `startup` | 27 idle mocks | spawn-to-ready, RSS, window handle, catalog size | | `auth` | — | which credential vocabularies the ingress accepts | | `non_stream_matrix` | 4 inbound × 5 upstream | text fidelity, `model` field survival | | `tool_matrix` | tools declared, 8 paths | does the provider see tools, does the client get an `id` | | `stream_matrix` | 4 inbound × 5 upstream | streaming envelope versus ingress protocol | | `retry` | 401 / 400 / all-500 / header-500 / premature EOF | policy switches, attempt budget, failover | | `sniff` | 404 generic vs 404 "unsupported protocol" | what triggers a protocol switch | | `hint_order` | hints `[openai, anthropic]`, OpenAI-only server | does the declared first hint win | | `cache_billing` | three identical requests + purge | cache opt-out, purge, stats, ledger rows | | `alias_collision` | two aliases, one `real_model` | which group wins | | `edges` | — | auxiliary endpoints, management answers, stubs | | `overhead` | 25 paired requests | gateway versus direct latency | ## File manifest | Path | Contents | | --- | --- | | `main.tex` | Paper source: 11 pages, two columns, five figures, twelve tables | | `refs.bib` | Bibliography database (all entries verified against the cited URL) | | `common.tex` | Shared preamble, copied from `papers/_shared/common.tex` | | `numbers.tex` | Generated `\newcommand` definitions for every inline number | | `tables/*.tex` | Generated table environments (setup, static, adapters, nsgrid, stream, tools, retry, cache, manage, edges, overhead, ledger) | | `figures/fig1_pipeline.pdf` | Harness and gateway request pipeline | | `figures/fig2_conformance.pdf` | Text, `model` field and streaming-envelope grids | | `figures/fig3_failover.pdf` | Attempt budget and error-text sensitivity | | `figures/fig4_ledger.pdf` | Management answers versus SQLite; cache billing rows | | `figures/fig5_overhead.pdf` | Loopback forwarding cost | | `scripts/gateway_lab.py` | Mock upstreams, config generation, process and Win32 helpers | | `scripts/measure.py` | Scenario driver, writes the JSON artifact | | `scripts/audit_repo.py` | Static audit (modules, adapters, routes, stub markers) | | `scripts/make_tables.py` | Generates `numbers.tex` and `tables/*.tex` from the JSON | | `scripts/make_figures.py` | Figure generator (matplotlib, PDF output) | | `scripts/results/plocal_measure.json` | Every measured number in the paper | | `scripts/results/headless_check.log` | `cargo check` output for the broken headless feature | | `scripts/results/run/gateway.log` | Start-up banner and warnings from the measured run | | `scripts/data/lab_config.json` | Sanitized lab configuration used by the run | | `scripts/data/audit.json` | Static audit artifact | | `README.md` | This file | The generated `scripts/results/run/` directory keeps only `gateway_config.json` and `gateway.log`; the SQLite file is exported to JSON and deleted by the harness, and no API key appears in any committed file. ## Build Requires a TeX distribution with `pdflatex`, `bibtex`, `natbib` and `abbrvnat.bst` (TeX Live or MiKTeX), or Tectonic. `xspace`, `xurl`, `enumitem` and `array` are used. ```bash tectonic main.tex # or pdflatex main && bibtex main && pdflatex main && pdflatex main ``` ## arXiv submission Upload the contents of `dist/plocal-switch.tar.gz` (built with `bash ../make_package.sh plocal-switch`) at . The archive contains `main.tex`, `refs.bib`, `common.tex`, `numbers.tex`, `tables/` and `figures/`; including the generated `main.bbl` is also fine. Choose `cs.SE` as the primary category and `cs.DC` plus `cs.CR` as cross-lists. `main.tex` compiles with both Tectonic and `pdflatex` + `bibtex`. ## Reproduce the measurements Requirements: Windows (the harness uses Win32 process and memory APIs), Python 3.13 with PyYAML and matplotlib, a Rust toolchain, and the PLocalSwitch repository built in release mode. ```bash # in the gateway repository cargo build --release # headless feature check (expected to fail; logged for the paper) cargo check --release --no-default-features --features gateway-server \ | tee ../papers/plocal-switch/scripts/results/headless_check.log # in this paper directory python scripts/audit_repo.py # -> scripts/data/audit.json python scripts/measure.py # -> scripts/results/plocal_measure.json python scripts/make_tables.py # -> numbers.tex, tables/*.tex python scripts/make_figures.py # -> figures/*.pdf ``` `measure.py` refuses to start if any of its 28 ports is busy or another `plocalswitch` instance is running, so a stray desktop gateway will not silently pollute a run. A full run takes well under a minute. ## Key results - Start-up: ready in 0.510 s, idle RSS 36.43 MB, a window handle is created, and `/v1/models` publishes 31 entries. - Non-streaming: 20/20 combinations return the exact upstream text; the `model` field survives in only 4/20 (the OpenAI ingress, minus the Gemini upstream). - Streaming: 5/20 responses use the ingress envelope. Anthropic and Ollama clients get OpenAI SSE; Gemini `streamGenerateContent` returns one `application/json` body. - Tools: 7/8 requests reach the provider with tools, 5/8 responses keep a usable call `id`; the Gemini ingress drops tools in both directions. - Retry: `auth_401_403 = false` and `bad_param_4xx = false` both retry anyway; an all-fail group issues 24 upstream POSTs against a configured cap of 6, touching 8 of 10 nodes in 25.26 ms, then returns HTTP 502. - Protocol switching keys on the error body: `404 page not found` ends the chain with HTTP 400, `unsupported protocol; use /v1/messages` switches and returns HTTP 200. - A node configured `protocol_hints: [openai, anthropic]` is addressed with Anthropic first, because the canonical token is `open_ai`; against an OpenAI-only server the request fails with HTTP 400. - Premature stream close: HTTP 200, one data chunk, a synthetic `[DONE]`, the second candidate never tried, 0 billed tokens. - Cache: an alias with `cache_enable: false` is cached; `/manage/cache/purge` is a no-op; `/manage/cache/stats` reports 0 entries while serving hits; cache-hit ledger rows keep 1,500 tokens, lose the model name and charge 0.000 CNY. - Control plane: 91 traces, 91 ledger rows, 124,500 billed tokens and 0.162 CNY in the database, against `/billing/summary` reporting 0 requests and 0 tokens, `/traces` and `/audit` returning `[]`, `/nodes` returning null quality for 27/27 nodes, and `/metrics` returning a placeholder string. - Auth: `Authorization: Bearer` works; Anthropic `x-api-key` and Gemini `?key=` both get HTTP 401. - Build: `cargo check --no-default-features --features gateway-server` exits 101 with 12 `E0433` errors, so every measurement is of the desktop binary. - Overhead: median 21.65 ms through the gateway versus 1.60 ms direct, paired delta 8.21 ms, `p95` 31.11 versus 24.77 ms. ## Documentation vs. measured behavior The start-up banner is treated in this paper as a specification, and three of its clauses are contradicted by measurement: "100% OpenAI v1 compatible" (the streaming envelope and the `model` field), "dual-ledger (client / upstream)" (one ledger table exists, and `usage_source` is hard-coded), and "tiktoken audit" (the tokenizer pool returns `None`). The same applies to configuration: `policy.retry_on.*`, `flex_adapter.global_max_sub_attempts`, and per-alias `cache_enable` are all present in the schema and all unenforced. Where the source itself documents the gap — 22 explicit `TODO`/`stub`/`placeholder` markers, 9 constant-response handlers — the paper cites the marker rather than inferring intent.