Contract Drift in a Multi-Protocol LLM Gateway: Black-Box Measurement of PLocalSwitch v0.2.27

Abstract

Multi-protocol LLM gateways promise to be many things at once: an OpenAI-compatible front door, a translator to vendor-native APIs, a failover layer, and a billing system. Each promise is a contract, and each contract is testable. This paper measures one such gateway, PLocalSwitch v0.2.27 at commit b9ce157 (8,642 Rust lines across 89 files), against the contracts it advertises, using a black-box harness that runs the shipped desktop binary against 27 local mock upstreams and exports the gateway's own SQLite ledger. The data plane is genuinely good: all 20/20 non-streaming combinations return byte-identical text. The control and envelope planes are not. Only 5/20 streaming responses use the envelope the client's protocol expects — Anthropic and Ollama clients receive OpenAI `data:` chunks, and a Gemini `streamGenerateContent` call receives a single non-streaming JSON body — and the `model` field is empty in 16/20 non-streaming responses. Tool calls vanish in both directions on the Gemini ingress. Configured retry policy is not consulted: a `false` switch still retries, and a sub-attempt budget of 6 becomes 24 upstream POSTs. A mid-stream premature close is reported as HTTP 200 with a synthetic `[DONE]` and zero billed tokens. Per-alias cache opt-out is ignored, cache purge is a no-op, and the cache statistics endpoint reports 0 entries while the cache is serving hits. The management plane answers 0 where the same process wrote 91 traces. We classify these into four kinds of drift — envelope, vocabulary, policy and telemetry — and report the negative results, including a headless build configuration that does not compile (12 errors).

Body

Imported entry paper (unverified hypothesis)

This paper was imported into jetessays as an entry paper from the author's local archive (plocal-switch). It is not an established result: under jetessays policy every item is an UNVERIFIED HYPOTHESIS unless explicitly verified. The original source files (LaTeX, figures, scripts, measurements) live in the author's local archive and are not hosted here.

Abstract (as submitted)

Multi-protocol LLM gateways promise to be many things at once: an OpenAI-compatible front door, a translator to vendor-native APIs, a failover layer, and a billing system. Each promise is a contract, and each contract is testable. This paper measures one such gateway, PLocalSwitch v0.2.27 at commit b9ce157 (8,642 Rust lines across 89 files), against the contracts it advertises, using a black-box harness that runs the shipped desktop binary against 27 local mock upstreams and exports the gateway's own SQLite ledger. The data plane is genuinely good: all 20/20 non-streaming combinations return byte-identical text. The control and envelope planes are not. Only 5/20 streaming responses use the envelope the client's protocol expects — Anthropic and Ollama clients receive OpenAI data: chunks, and a Gemini streamGenerateContent call receives a single non-streaming JSON body — and the model field is empty in 16/20 non-streaming responses. Tool calls vanish in both directions on the Gemini ingress. Configured retry policy is not consulted: a false switch still retries, and a sub-attempt budget of 6 becomes 24 upstream POSTs. A mid-stream premature close is reported as HTTP 200 with a synthetic [DONE] and zero billed tokens. Per-alias cache opt-out is ignored, cache purge is a no-op, and the cache statistics endpoint reports 0 entries while the cache is serving hits. The management plane answers 0 where the same process wrote 91 traces. We classify these into four kinds of drift — envelope, vocabulary, policy and telemetry — and report the negative results, including a headless build configuration that does not compile (12 errors).

Note

Imported via the jetessays seed importer. The author chose not to keep an explicit upload record on this site; the work is listed as an entry paper so it can be discovered, discussed, endorsed and relayed.

Discussion

Log in to join the discussion

← back to browse