# Privacy Action Benchmark: saved local and OpenRouter results

This report covers **47 evaluated configurations: 41 local checkpoint configurations and six OpenRouter services**. Each selected configuration completed the same four panels, totaling **9,306 selected decisions and 188 configuration × panel rows**. One additional local repeat is reported separately. This publication uses existing results; no new inference, fitting, or tuning was performed.

## What the results show

Mercury Decide and Jev 1.13 each returned 198 acceptable raw decisions out of 198, with no labeled unsafe actions. Both fell to 176 acceptable decisions after the fixed guard. Liquid D1 returned 196 acceptable raw decisions with no labeled unsafe actions. Solar Decide returned 184, including four labeled unsafe actions. The strongest local result by combined raw acceptable count was Mapika/decider-4b in its recorded NF4 configuration: 181/198 acceptable, zero labeled unsafe actions, and 158/198 acceptable after guarding. No local configuration matched the historical Mercury reference across all four panels.

These are descriptive observations on small, exposed, synthetic and template-correlated panels. They do not establish superiority on a population, calibrated privacy-risk estimates, legal compliance, anonymity, or deployment safety. The combined count is useful for navigation; per-panel results and interface differences remain necessary for interpretation.

## Full evaluated inventory

The table combines four correlated panels for each configuration. Revision and precision distinctions are retained in the JSON and interactive details. Local revisions are full pinned repository commits. Hosted revisions are provider-returned dated service identities, not independently verified weight commits. Exact hosted precision, quantization, and hardware were not disclosed.

| Model | Source | Scoring | Raw acceptable /198 | Guarded acceptable /198 | Unsafe raw /guarded | Guarded automated /198 | Acceptable automated /automated | Guard reviews (unnecessary) |
|---|---|---|---:|---:|---:|---:|---:|---:|
| `11wanqi/laya-command-safety-ft (float32 / none)` | Local | Native decision readout | 78 | 53 | 55/0 | 77 | 46/77 | 121 (114) |
| `3p3r/sentinel-laya (float32 / none)` | Local | Native decision readout | 79 | 45 | 59/0 | 51 | 37/51 | 147 (139) |
| `3p3r/sentinel-laya-multilingual (float32 / none)` | Local | Native decision readout | 65 | 19 | 72/0 | 13 | 5/13 | 185 (171) |
| `ameerhmz5/laya-modernbert-decision-90pct (float32 / none)` | Local | Native decision readout | 130 | 17 | 6/0 | 3 | 3/3 | 195 (181) |
| `Arc53/arc-decide-0.6b-v0.1 (float16 / none)` | Local | Native decision readout | 108 | 34 | 27/0 | 20 | 20/20 | 178 (164) |
| `bofenghuang/docto-decision-modernbert-fr-base-v0.1 (float32 / none)` | Local | Native decision readout | 65 | 27 | 8/0 | 20 | 14/20 | 178 (165) |
| `carrtesy/EXAONE-4.0-1.2B-JEV (bfloat16 / none)` | Local | Native decision readout | 116 | 96 | 47/6 | 119 | 90/119 | 79 (73) |
| `chaoliangUNSW/Jev-Style-0.8B-Decision-v3 (float16 / none)` | Local | Native decision readout | 107 | 89 | 50/3 | 109 | 83/109 | 89 (83) |
| `convaiinnovations/laya-typed-decisions (float32 / none)` | Local | Native decision readout | 63 | 19 | 126/0 | 11 | 5/11 | 187 (173) |
| `cross-encoder/nli-deberta-v3-base (float32 / none)` | Local | Adapted NLI | 48 | 25 | 126/0 | 11 | 11/11 | 187 (173) |
| `cross-encoder/nli-distilroberta-base (float32 / none)` | Local | Adapted NLI | 35 | 14 | 0/0 | 0 | 0/0 | 198 (184) |
| `cross-encoder/nli-MiniLM2-L6-H768 (float32 / none)` | Local | Adapted NLI | 43 | 15 | 122/0 | 8 | 1/8 | 190 (176) |
| `elnachto/laya-triage-en (float32 / none)` | Local | Native decision readout | 76 | 15 | 94/0 | 3 | 1/3 | 195 (181) |
| `facebook/bart-large-mnli (float32 / none)` | Local | Adapted NLI | 55 | 17 | 131/0 | 10 | 3/10 | 188 (174) |
| `GhostScientist/jev-decisions-v1-model (float16 / none)` | Local | Adapted label scores | 48 | 47 | 28/4 | 106 | 40/106 | 92 (85) |
| `gtm-k/laya-maintenance-decision-assist (float32 / none)` | Local | Native decision readout | 63 | 55 | 121/3 | 59 | 45/59 | 139 (129) |
| `gtm-k/laya-product-decision-assist (float32 / none)` | Local | Native decision readout | 70 | 72 | 107/5 | 91 | 63/91 | 107 (98) |
| `Horizon-Labs/multilingual-zeroshot-base (float32 / none)` | Local | Adapted NLI | 38 | 14 | 32/0 | 0 | 0/0 | 198 (184) |
| `Horizon-Labs/multilingual-zeroshot-large (float32 / none)` | Local | Adapted NLI | 31 | 16 | 111/0 | 9 | 2/9 | 189 (175) |
| `Horizon-Labs/multilingual-zeroshot-small (float32 / none)` | Local | Adapted NLI | 53 | 14 | 99/0 | 2 | 0/2 | 196 (182) |
| `HuggingFaceTB/SmolLM3-3B (bfloat16 / none)` | Local | Adapted label scores | 68 | 55 | 36/9 | 117 | 50/117 | 81 (76) |
| `impacte/laya-medical-escalation (float32 / none)` | Local | Native decision readout | 73 | 14 | 117/0 | 3 | 0/3 | 195 (181) |
| `impacte/mimir-laya-router (float32 / none)` | Local | Native decision readout | 68 | 63 | 108/6 | 74 | 52/74 | 124 (113) |
| `internlm/Intern-Decision-0.8B (bfloat16 / none)` | Local | Native decision readout | 137 | 90 | 2/0 | 94 | 82/94 | 104 (96) |
| `internlm/Intern-Decision-2B (float16 / none)` | Local | Native decision readout | 157 | 132 | 6/0 | 155 | 128/155 | 43 (39) |
| `internlm/Intern-Decision-4B (bfloat16 / nf4)` | Local | Native decision readout | 173 | 151 | 1/1 | 162 | 147/162 | 36 (32) |
| `JoshuaAshkinaze/policyfx-nli (float32 / none)` | Local | Adapted NLI | 46 | 16 | 152/0 | 10 | 2/10 | 188 (174) |
| `Mapika/decider-0.8b (float16 / none)` | Local | Native decision readout | 123 | 115 | 34/0 | 142 | 110/142 | 56 (51) |
| `Mapika/decider-2b (bfloat16 / none)` | Local | Native decision readout | 146 | 128 | 23/1 | 151 | 123/151 | 47 (42) |
| `Mapika/decider-4b (bfloat16 / nf4)` | Local | Native decision readout | 181 | 158 | 0/0 | 173 | 156/173 | 25 (23) |
| `MoritzLaurer/bge-m3-zeroshot-v2.0 (float32 / none)` | Local | Adapted NLI | 62 | 14 | 88/0 | 2 | 0/2 | 196 (182) |
| `MoritzLaurer/deberta-v3-base-zeroshot-v2.0 (float32 / none)` | Local | Adapted NLI | 43 | 14 | 35/0 | 0 | 0/0 | 198 (184) |
| `MoritzLaurer/deberta-v3-large-zeroshot-v2.0 (float32 / none)` | Local | Adapted NLI | 72 | 14 | 64/0 | 0 | 0/0 | 198 (184) |
| `MoritzLaurer/ModernBERT-base-zeroshot-v2.0 (float32 / none)` | Local | Adapted NLI | 42 | 14 | 71/0 | 0 | 0/0 | 198 (184) |
| `MoritzLaurer/ModernBERT-large-zeroshot-v2.0 (float32 / none)` | Local | Adapted NLI | 38 | 14 | 10/0 | 0 | 0/0 | 198 (184) |
| `MoritzLaurer/roberta-base-zeroshot-v2.0-c (float32 / none)` | Local | Adapted NLI | 40 | 16 | 158/0 | 10 | 2/10 | 188 (174) |
| `Prasanna85/laya-issue-triage (float32 / none)` | Local | Native decision readout | 103 | 40 | 48/0 | 37 | 29/37 | 161 (150) |
| `Qwen/Qwen3-0.6B (float16 / none)` | Local | Adapted label scores | 50 | 58 | 148/6 | 57 | 44/57 | 141 (127) |
| `Suarify/laya-pdpa-my-v3 (float32 / none)` | Local | Native decision readout | 96 | 86 | 19/0 | 141 | 84/141 | 57 (55) |
| `Suarify/laya-pdpa-my-v4 (float32 / none)` | Local | Native decision readout | 82 | 72 | 18/0 | 136 | 68/136 | 62 (58) |
| `winwinwinbb/soft-decider-421m (float32 / none)` | Local | Native decision readout | 57 | 28 | 139/0 | 20 | 14/20 | 178 (164) |
| `inception/mercury-decide:free` | OpenRouter | Native hosted Choice | 198 | 176 | 0/0 | 172 | 172/172 | 26 (22) |
| `liquid/d1` | OpenRouter | Native hosted Choice | 196 | 173 | 0/0 | 168 | 167/168 | 30 (24) |
| `respan/span-01` | OpenRouter | Adapted independent suitability | 166 | 143 | 12/3 | 146 | 136/146 | 52 (45) |
| `respan/span-01-lite` | OpenRouter | Adapted independent suitability | 164 | 142 | 12/3 | 146 | 135/146 | 52 (45) |
| `typesafe/jev-1.13` | OpenRouter | Native hosted Choice | 198 | 176 | 0/0 | 168 | 168/168 | 30 (22) |
| `upstage/solar-decide` | OpenRouter | Native hosted Choice | 184 | 163 | 4/4 | 162 | 152/162 | 36 (25) |

## Population and decision rubric

There are 99 underlying synthetic English case identities: 43 diagnostic cases and 56 challenge cases across eight correlated scenario families. Each appears in an original and a compact sample-free representation, producing 198 jobs per configuration. Four panels are retained separately: `diagnostic43-no_samples` (43), `diagnostic43-compact_no_samples` (43), `challenge56-no_samples` (56), and `challenge56-compact_no_samples` (56). Representations and repeated scenario templates are correlated, so 198 decisions are not 198 independent examples.

The six actions are `keep`, `drop`, `mask`, `pseudonymize`, `generalize`, and `review`. Acceptability means membership in the rubric's accepted action set. Unsafe actions are separately labeled; unsafe is not the complement of acceptable. Review receives credit only where the rubric accepts review. An incorrect action without an unsafe label is still an error.

The screen removes cell samples from model states. Oracle labels and scorer metadata were not model inputs. Public release contents are methodology and aggregate outcomes only: no state strings, per-case errors, raw requests or responses, cell values, private documents, authentication material, transport headers, machine identifiers, or personal paths.

Fixture provenance:

- `compact_fresh_test.json` SHA-256: `4692aa8c449d17b1b7bcab1593da32ec6436125dfb17ed70e46024b458012f9e`.
- `generality_challenge_v1.json` SHA-256: `4ae20dc31799e2cc40d2651edbeb9ff216666d48a6072e9fcb1b4e0cb6ac1f3d`.

## Interfaces and score semantics

The local inventory has 24 native decision readout configurations, 14 adapted NLI configurations, and three adapted label-score configurations. A native readout label describes the model interface; it does not establish privacy-task calibration. Publisher temperatures and the recorded inference/loading format remain fixed. FP32 stock-framework Laya readouts and the French-trained Docto transfer condition are disclosed in each configuration's metadata. Docto used unchanged English tasks, with no translation experiment. NF4 configurations are explicit and are not interchangeable with unquantized checkpoints.

The NLI controls normalize recorded entailment scores over the six actions. Causal label controls use conditional label-score softmax. These are task adapters, not native trained six-action privacy heads. GhostScientist/jev-decisions-v1-model also uses an adapted conditional label softmax at T=1; the publisher's greedy temperature-zero example is not a categorical probability contract. Failed or skipped runs do not establish architecture-wide inability.

Four hosted services—Mercury, Jev, Liquid D1, and Solar—received identical named six-option Choice questions, instructions, ordered criteria, and candidates. All six hosted runs used exactly matching serialized state strings. Full Span and Span Lite used six identical independent Noul suitability questions, one for each action and its frozen criterion. The adapter selects the greatest independent yes score; exact ties select review. Those six Bernoulli scores do not sum to a categorical distribution and were never normalized.

## The fixed guard and selective metrics

Raw and guarded outcomes are separate. The guard uses audience, field type, linkage need, supported transformations, and the unchanged 0.45 review threshold. Policy rules can override the raw choice before the threshold is considered. Native Choice guards use the selected action's probability; Span uses the maximum independent yes score. The numerical threshold is the same, but these signals estimate different quantities, so guarded comparisons describe whole systems rather than identical confidence experiments.

Automatic coverage is the fraction of decisions whose action is not review. Conditional correctness is acceptable automated decisions divided by automated decisions; it never replaces overall acceptability. Unnecessary reviews are reviews where the rubric does not accept review. The downloads retain raw and guarded acceptable, unsafe, review, coverage, conditional correctness, and unnecessary-review metrics. Local exports also retain per-action aggregate slices and confusion matrices.

## Probability and calibration diagnostics

Categorical NLL and Brier use singleton-gold rows: 38, 38, 56, and 56 per panel. Brier is the sum of squared differences over classes, averaged over those rows. The five set-valued diagnostic cases per representation are not assigned an invented categorical target. Their acceptable-set negative-log mass/event loss is retained separately, with zero-support counts where recorded. The current saved metrics contain no nonfinite losses. The exporter rejects nonfinite numbers rather than silently treating an infinite loss as missing; any future infinite loss would need an explicitly labeled status before release. JSON serialization forbids NaN and Infinity literals.

Local calibration exports retain both acceptable-set correctness and singleton-label correctness. The local primary equal-width diagnostic uses ten bins over [0,1], final upper boundary included. Local equal-mass bins follow floor rank boundaries at floor(i*n/k), with k=min(10,n//2). Hosted categorical calibration uses singleton labels and the historical Mercury audit's balanced larger-first equal-mass chunks, at least two rows per chunk. These equal-mass constructions are explicitly different; they are not silently pooled.

Span has no categorical NLL, multiclass Brier, or Choice ECE. Its independent-score sum ranges, zero/multiple yes-score counts, and suitability-score reliability are preserved under a separately named diagnostic in the JSON. They are not ranked as Choice calibration. No generic native temperature, provider field named confidence, or adapter normalization proves held-out calibration. Low ECE, especially for weak or nearly uniform predictions or entirely correct exposed panels, does not establish high accuracy or calibrated privacy risk.

## Timing and recorded cost

Local panel timing is serial synchronous scoring. The exact scope is retained per backend: it generally includes device transfer and forward/readout, excludes tokenization, and may include complete encoder/head computations. Runs used occupied workstations with resident services. Hardware and active load confound cross-model latency. Load time and peak allocated/reserved memory are preserved where recorded; no missing timing is reconstructed.

Hosted timing is client wall-clock, network-inclusive request roundtrip. No separate provider GPU/compute timing was returned. Hosted medians and nearest-rank p90 are retained. Local p95 uses linear interpolation at (n−1)*q. Local and hosted observations have different scopes and runs were not contemporaneous. The UI does not infer an overall local median from panel medians; it shows timing when a panel is selected. A recorded overall hosted median remains available in the combined view. D1 and Solar have separately recorded per-job bookkeeping walltime in the hosted metadata; that measurement is missing for other services.

All 1,188 hosted benchmark responses include usage and reported cost; none required a benchmark retry. Their combined benchmark-reported cost is **$0.018827174**. Known successful probes add **$0.000021630**. Three rejected Span interface probes have no usage, and earlier discovery spend is incompletely recorded. These are usage-reported amounts, not a reconciled account invoice or a claim about current pricing. Reported zero cost for the recorded Mercury/Lite runs is not a promise of current free availability. [The detailed hosted report](HOSTED_RESULTS.md) retains provider identities, timing, interface diagnostics, and billing coverage.

## Verification, repeats, and excluded work

The local publication reads the sealed completed aggregate with SHA-256 `df3e1c9c75d104f0ea22ba9efa05482f57e934761cedccd05893c52992a26b1c`. Every selected record has a complete 198-prediction receipt and four correctly sized panels. The preserved local count is 41 distinct configurations, 42 completed runs including one prior repeat, 8,118 selected predictions, and 164 panel rows. The 198-row Mapika/decider-4b NF4 repeat changed no choices and had maximum absolute saved probability difference 0.0. It is not an extra distinct configuration or independent set of cases.

The supplied hosted publication audit rejoined all 1,188 rows to frozen jobs, replayed guards, recomputed metrics, confirmed exact state/interface pairing, checked complete usage and model/provider identities, and verified no benchmark retries. Existing benchmark and analysis files remained unchanged. The audited hosted aggregate SHA-256 is `7cf0c8431b5212f699b2a350f154d319c3806284e093f27abcb6d2f34726649e`.

Cloudflare/clef-flash and Cloudflare/clef were prepared but not evaluated: temporary cloud provisioning did not establish access and the work was paused before inference. They have no benchmark metrics and are excluded from ranks and evaluated counts. Rejected Span Choice/Score interface probes are also listed separately from the successful Noul runs. This report is an inventory of completed saved results, not an exhaustive catalog of every searched or rejected repository lead.

## Interpretation and release format

Nominal local Wilson intervals are retained as binomial diagnostics, not strong population inference or superiority evidence. No significance test, privacy-specific temperature fitting, tuning, or new inference was performed for this report. A zero labeled unsafe count is not an anonymity, legal-compliance, malware-free, or behavioral-robustness certificate. Model admission/integrity checks and aggregate replay are not proof against all runtime vulnerabilities or model backdoors.

The report is a static HTML Space with no inference endpoint, runtime server, model download, or GPU provisioning. Filtering and sorting run in the visitor's browser over saved aggregate files. [Hugging Face documents static Spaces as directly served without compute](https://huggingface.co/docs/hub/spaces-sdks-static).

Release files: `index.html`, `styles.css`, `app.js`, `results.json`, `results.csv`, this `REPORT.md`, `HOSTED_RESULTS.md`, and the Space `README.md`. The JSON retains the most complete aggregate information; the CSV provides one row per configuration/panel for comparisons. Missing or semantically inapplicable metrics remain absent or null, never zero-filled as results.