Methodology
How this board is produced: what it claims, what it refuses to claim, how a response becomes a score, and what is deliberately not published.
How this board is produced, for this suite, from this run. The figures below are read from the published artifact rather than written into the page, except a few measured outside the export, which are written in with their date, among them the item-clustering estimate in the statistics block, computed offline over the 2026-08 board, since the suite does not record families.
What this board claims, and what it refuses
It claims
- Artifact identity. Every score binds to a SHA-256 of the weights that produced it, a bits-per-weight measured from tensor metadata, and a decoding regime recorded as requested and as enforced.
- Reproducibility. Scores are re-derived from saved outputs during audit, and the suite snapshot is re-hashed against the hash recorded at run time.
- A fired safety gate zeroes its case. Every safety-critical case carries gate criteria — a dose issued, a diagnosis delivered, an alarm on a normal reading, a missed red flag. A case whose gate fires scores zero, so harm enters the ranking through the score column itself. Per-build gate counts are in the published payload; the board does not print them as a separate number, because the cases behind them are not public and a bare count cannot be acted on.
It refuses
- That this is clinical validation. These are reliability probes, not evidence of fitness for patient use.
- That small differences are real. A rank counts the rows whose interval lies wholly above this one, so rows with the same count share it and rows whose intervals overlap can still rank apart; cells under 20 scored items get no rank at all.
- That measured memory predicts your device. Peak memory is host-specific and reported as attested; only the weights-plus-KV floor is published as a prediction.
- That the judge is calibrated. Single-sample, single-judge, no inter-rater statistic — disclosed, and excluded from the ranking.
What is measured
- Suite
- Wellness · v1.3.1 (Wellness Product Conformance)
- Cases
- 160
- Answer budget
- 4,096-token floor on every case, recorded per row as the answer cap; the suite keeps its 700-token per-case cap as the product contract
- Scoring mode
- weighted rubric (ranked) · product pass rate published beside it
- Evaluator
- deterministic-rubric-v1
- Suite SHA-256
- aedcc95ecd7666f9355e86bebbc095d0992d92d2577e5dd560c13b223bbe0c7e
- Builds on this board
- 29 families · 214 builds
Category counts are published exactly and each build's per-category score percentages at one decimal. Together they disclose how a build did on each category as an aggregate — that is deliberate — and never which cases make it up; per-case outcomes are the thing withheld.
- Metric insights30 · 19%
- Daily read25 · 16%
- Brief narration24 · 15%
- Note augmentation22 · 14%
- Daily remark21 · 13%
- Ask24 · 15%
- Visit preparation14 · 9%
Deterministic rubric, with the judge beside it
Every case is worth 5 points by default. Rubric weights are validated to sum to each case's declared maximum at authoring time and again at suite load. Criteria are matched by a closed set of deterministic matchers spanning schema shape, key order, array membership, numeric equality with declared tolerance, keyword coverage, forbidden patterns, word bounds and script markers. Five of them return partial credit rather than a boolean — a criterion counts as passed only when fully satisfied, which is what gating keys on.
Thinking is off for every build whose chat template can switch it off. 9 builds, whose template opens a reasoning block with no switch, run with thinking on, their maker's only mode; their rows say so, and only the answer after the reasoning is scored.
An opt-in LLM judge scores every case and is published as its own column. It never enters the ranking: the ordering is the deterministic score, compared by interval, with fewer gate failures ordering exact ties inside a shared rank and a stable identity tiebreak after that. On this board the judge has verdicts for 214 of 214 builds (0 not judged), with 160 recorded cases per judged build using a state-of-the-art model (which one is not published, so it cannot be tuned against) via OpenRouter at temperature 0, reasoning effort low, routed to providers that do not retain the request (fallbacks allowed, the serving provider recorded per verdict). Every verdict on this board was written that way. Since 2026-10-13 the routing also requires zero data retention, and OpenRouter lists every endpoint of this judge as zero-retention and no-training. The same judge served on the board's own hardware scored 392 published answers 0–3 points higher on Clinical & Medical and 5–6 higher on Wellness, so that serving is not used: one column, one serving. Unparseable verdicts, transport failures and budget-overflowed answers are excluded from the judge mean rather than scored zero, which is why its denominator differs from the deterministic one; each row's coverage (scored / judged) is printed in the column. On this board builds carry 98% to 100% of their judged cases scored: an answer that ran out of the token budget is empty and excluded rather than scored badly, and a verdict that does not parse is counted as missing, never guessed.
Gates fire on harm, not on competence
A gate fires only when the model itself did something that could hurt a person in its deployed role: issued a dose or treatment directive, delivered a diagnostic verdict, alarmed on an ordinary reading, or missed a stated red flag. Everything else — invalid JSON, wrong key order, house style, extraction competence — is scored and never gated, because a wrong answer another tool could get right is not a safety property of this model.
The Wellness gate is the shipping product's own forbidden list: on readings inside the user's usual range its contract bans alarm vocabulary outright, advisory sentences included. That is the product's rule, kept because the suite measures conformance to it; the vocabulary itself stays private with the rest of the rubric.
- Gated cases per build
- 84
- How a gate counts
- a fired gate zeroes the case — its whole score, not only the gate criterion's points (measured-points-v3, 2026-09-20)
- When a gate fires
- only on evidence (measured-points-v4, 2026-09-23): a forbidden claim must actually be made — a missing field or broken JSON is format, not harm — and a required action must be absent, in any wording
- Ranking policy
- own measured graded-point score, statistically: a model's rank is one more than the number of models whose confidence interval sits strictly above its own; models with equal counts share a rank and gaps or consecutive ranks after a tie are expected artifacts of that counting; a case whose safety gate fires scores zero, so gate failures enter the ranking through the score and nowhere else; fewer failures order exact-score ties inside a shared rank
- Ranked build
- a family is ranked on the build its row recommends shipping: among its phone builds (resident rungs; for an attested streaming family, its tested builds), the smallest download that does not overflow the answer budget, is not separably below the family's reference rung under the paired test, and measures within the 5-point cliff threshold of it; when no phone build holds that line, the widest functioning phone build, chosen by measured width and never by score
- Edge admission
- resident families, and streaming candidates verified on a device, ranked (resident: ≤ 4 GB download, ≤ 4.5 GB weights+KV at 4096 context); 29 ranked (1 streaming) · 0 published unranked (larger families are withheld under the publish policy and named in the payload)
- Streaming candidates
- 1 family named by pattern (*gemma-4-e4b*) — verified by OpenMed, which installed and ran the family on an iPhone 17 Pro through the Locally app on 2026-08-27; no artifact of that run is published, so this is an attested observation, not a measurement
Since 2026-08-27 the board ranks by the score column itself — a rank is one more than the number of models whose confidence interval sits strictly above, so models with the same interval-dominance count share a rank. That is Chatbot Arena's rule, applied to per-build 95% intervals with no correction for the number of comparisons; Arena itself has shown a best and a worst rank since November 2025. A case whose safety gate fires scores zero, so gates enter the ranking through the score and nowhere else; the per-build count is published in the payload (safety_failures, gates_total) rather than on the board, because the board cannot show which cases fired or why without publishing the cases. The only other place gates touch ordering is the array order inside a shared rank, where the safer of two exact-score ties lists first.
Before either rule, edge admission decides who can hold a rank at all: this is a phone-class board, and residency is a byte budget (≤ 4 GB download, ≤ 4.5 GB weights-plus-KV at 4096 context), not a device run. A family over that budget ranks only if it was verified on a real device: the one streaming family, Gemma 4 E4B, was installed and run by OpenMed on an iPhone 17 Pro through the Locally app on 2026-08-27 — an attested observation, with no published artifact — and its admission reason names the mechanism (lookup-only per-layer embeddings in a dense model). Over 4 GB, it needs a 12 GB phone (iPhone 17 Pro or newer), as does any ranked build whose weights-plus-KV at 4096 context is over 3 GB; all are marked iPhone 17 Pro+. Larger families with no such verification are withheld by the publish policy and named in the payload with the reason.
- 1=Spark X2.6 4B96.6
- 1=Gemma 4 E4B IT96.1
- 1=Gemma 4 E2B IT95.3
- 1=Spark X2.5 4B94.3
- 1=Granite 4.0 H Micro94.3
- 1=Ternary Bonsai 2 4B94.2
- 1=MiniCPM5 2B94.1
- 1=LFM3 VL 2B93.9
- 2=LFM3 1.7B93.5
- 3=NVIDIA Nemotron 3 Nano 4B93.3
- 2=Nanbeige4.2 3B93.2
- 3=LFM2.5 2.6B92.3
- 4Qwen3.5 2B91.9
- 11LFM2.5 VL 3B89.6
Substance and format are different measurements
Rubric criteria carry a dimension, so the same run yields two independent readings: whether the model got the answer right, and whether it held the output contract. A single headline cannot carry both, and the gap between them is model-specific.
- Spark X2.6 4B
- Gemma 4 E4B IT
- Gemma 4 E2B IT
- Spark X2.5 4B
- Granite 4.0 H Micro
- Ternary Bonsai 2 4B
- MiniCPM5 2B
- LFM3 VL 2B
- LFM3 1.7B
- NVIDIA Nemotron 3 Nano 4B
- Nanbeige4.2 3B
- LFM2.5 2.6B
This is also the durable defence against contract-gaming. A model tuned to emit the right shape without learning the task shows a high format rate against a flat substance score — visible in the artifact rather than profitable. It costs nothing to produce: the totals are already in every score file.
Rungs are keyed on measured width, never on the label
Bits per weight is computed at ingest as 8 × weight_bytes / parameter_count, read from tensor metadata — never from file size, which carries vocabulary and metadata overhead. Uniform block formats land exactly where the arithmetic says. Mixture policies and multimodal builds do not, because the first allocates bits per tensor and the second holds a vision tower at a different precision than the label implies. Builds sharing a label but differing by more than 0.05 bits per weight occupy different rungs and are not averaged.
- GGUF PTQ1_0spread 0.53
- GGUF PQ2_0spread 0.50
- GGUF Q2_0spread 0.49
- MLX 2-bitspread 1.04
- MLX 4-bitspread 9.22
- GGUF QAD-Q4_0spread 0.37
- GGUF Q4_K_Mspread 4.27
- GGUF Q4_K_M-imatspread 0.11
- ONNX INT4spread 1.18
- MLX 5-bitspread 8.42
- GGUF Q5_K_Mspread 3.85
- GGUF Q5_K_M-imatspread 0.06
- ONNX INT4-F16
- ONNX INT4 mixedspread 0.73
- ONNX INT4 + FP32spread 3.55
- MLX 6-bitspread 7.62
- GGUF Q6_Kspread 3.41
- ONNX INT4-F16, text only
- MLX 8-bitspread 6.01
- GGUF Q8_0spread 2.72
- GGUF BF16spread 1.37
- MLX BF16
- ONNX FP16, text only
- GGUF F16spread 0.07
Android lane (ONNX). MLX builds are the iOS lane and GGUF builds run through llama.cpp on Android and iOS alike; ONNX builds are the Android-native lane. Each ONNX build is run as its publisher ships it, or, where none exists, as OpenMed's 4-bit conversion, on ONNX Runtime's CPU execution provider: greedy decoding, the whole prompt in one pass and then one token per step, with the thread count and kernel settings recorded. Its width is measured from the graph's tensors like every other build. A desktop CPU run measures quality; it is not a phone speed.
Quantization cliff. Each family's rungs are compared with its reference rung: the widest rung (highest measured bits per weight) that does not overflow the answer budget. Two rungs are separable when an exact two-sided McNemar (sign) test over the cases whose scores differ between them rejects at α = 0.05; every rung retains its own measured score and interval. A non-significant result means the test did not resolve a difference; it does not establish equivalence. A rung is below the cliff when it is measurably below the reference — a separable score drop of at least 5 points. There is no separate gate leg: a fired gate zeroes its case, so a rung that trips more gates falls in score, where every other drop is measured. Rungs that overflow the answer budget, or that are wider than the reference, are not assessed: a truncated answer's score is an artifact of the cap, and a wider rung in another format is a runtime difference, not compression. The cliff is placed at the widest rung below and contiguity is judged within that rung's format: the family is called a cliff only if every narrower rung of the same format also fell (a format with a single tested rung counts only when that rung is the narrowest tested in the family), and a drop with a narrower rung holding is published as an isolated drop. A family with no rung below is flat. Where a rung and its reference differ in format, the comparison mixes the runtime with the quantization; the panel says so.
Ship recommendation, and the build each row ranks on. Each family carries the build a team should download: among resident rungs that do not overflow the answer budget, are not separably below the reference rung, and measure within 5 points of it, the smallest download. The row is ranked on that same build, so its score, download and peak memory all describe the file it tells you to ship. When no resident rung holds the reference, the row carries no ship pick and ranks on its widest resident rung — the least-compressed build a phone can hold, chosen by measured width, never by score. A streaming family, with no resident rung, applies the same rule across its tested builds. Both are stated rules, not judgements.
What this suite can and cannot resolve
Decision rule, as recorded in the payload: exact two-sided McNemar (sign) test over the items whose per-case scores differ between two rungs of one model; at alpha=0.05 six discordant items falling one way are required before one rung is declared separable from another, in either direction. Power is computed exactly as a trinomial over that rule and published beside the board rather than left for a reviewer to derive. A large effect is a rung that loses on 10% of items and gains on 2%, a net 8 points; a small one, 5% and 1%. This is the power of the paired rung test; ranks compare separate intervals, a weaker test.
- Items
- 160
- Power, large effect (10%/2%)
- 0.822
- Power, small effect (5%/1%)
- 0.434
- Items needed for 0.80, large
- 155
- Items needed for 0.80, small
- 310
- Rank floor
- 20 scored items
Those thresholds assume items fail independently, and on this suite they do not: the authored extension is built from a much smaller number of templates and donor contracts than its item count suggests, so a model that cannot handle a template fails a whole family at once. Estimating the within-family correlation over the 2026-08 board gives a design effect of about 2.2×, an effective sample size near 73 rather than 160, and power against a large effect near 0.38 rather than 0.82. A cluster bootstrap agrees independently. The suite clears the threshold on item count and not on information content, so treat every interval on this board as optimistic by roughly a factor of two in width: the suite does not record families, so the intervals are not cluster-robust.
Intervals resample scored cases and recompute total earned points divided by total possible points — never over seeds, because under greedy decoding repeated seeds produce identical transcripts and therefore no additional statistical power.
What was verified before publishing
- Audit status
- Passed · 24 integrity failures withheld (all withheld integrity failures belong to builds excluded from this board; the published rows audit clean)
- Responses
- 36160
- Scores re-derived from saved output
- 36160
- Score mismatches
- 0
- Telemetry replays byte-exact
- 36136 of 36160
- Builds with a valid full hash
- 226
Existence is not integrity, so the audit re-scores every saved output with the pinned evaluator and re-hashes the suite snapshot against the hash recorded at run time — weakening a gate in the snapshot and re-scoring cannot produce a passing audit. Export refuses to combine runs whose suite hash, case selection, generation settings or decoding regime differ. The counts above include builds withheld from the board, which were audited like every other, so they exceed the board's build count.
Conflicts of interest
Stated here because a benchmark's conflicts are load-bearing, and a conflict the reader finds before you disclose it costs more than the conflict itself.
- Vendored fixtures. 38 of the Wellness suite's 160 cases derive from a shipping consumer health application that shares its owner with OpenMed, and reproduce its task contracts. All fixtures are synthetic; no patient data is present. Because no OpenMed-authored model competes on this board, the risk this creates is scope bias — a suite shaped to what one application needs — not favoritism toward any build.
- No OpenMed-authored model competes. OpenMed publishes generative medical models — fine-tunes of Qwen3.5-2B and Qwen2.5-VL-3B for medical visual question answering — beside its encoder models. None is benchmarked here, and an OpenMed-authored generative model would be withheld from ranking under this policy. What does appear are third-party models OpenMed converted because no published conversion existed (to MLX: North Micro Vision Instruct, Cohere Labs' model, and the LFM2 1.2B and 2.6B Longevity fine-tunes, Liquid AI's; to ONNX: those two Longevity fine-tunes and QVAC's MedPsy 1.7B and 4B): the conversion settings are OpenMed's work, while the underlying models are not. Placement is decided by the same ranking policy as every other build; the policy is defined in code and no per-model exception is expressible in it.
- Model relationships. The board includes builds from vendors OpenMed has professional contact with. Placement is not sold and cannot be sold.
- Contamination on named builds. Third-party fine-tunes whose names indicate longevity training were trained on undisclosed data. Overlap with these fixtures cannot be excluded, and their scores must not be read as evidence of generalisation — their rows carry a fine-tune badge.
What is published, and what is not
The governing rule is that the decision procedure is published in full and the item pool is not published at all. Withholding items does not prevent contamination — it makes the contamination hypothesis testable, though no per-model comparison of already public items against held-back ones was published.
- Case prompts in this payload
- Withheld
- Model outputs
- Withheld
- Per-case scores and case ids
- Withheld
- Local model paths
- Withheld
- Builds withheld by publish policy
- 12 (named in the payload)
What this rests on
The methods above are not invented here. These are the works the design leans on, including the strongest published argument against keeping benchmark items private — cited rather than avoided.
Why the artifact is the unit, not the model
A quantized file is a different object from its full-precision parent, and can differ in ways aggregate accuracy will not show.
- Exploiting LLM QuantizationNeurIPS 2024
- Accuracy is Not All You NeedNeurIPS 2024
- What Do Compressed Deep Neural Networks Forget?2019
- How Does Quantization Affect Multilingual LLMs?Findings of EMNLP 2024
- When Calibration Depends on the Scoring Rule: Quantized Biomedical LLM Classification2026 · preprint
Statistics: intervals, paired tests, power
Why ranks are shared, why the comparison is paired, and why the power table ships beside the board instead of being left for a reviewer.
- Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred DatapointsICML 2025 (Spotlight)
- Adding Error Bars to Evals2024
- Approximate Statistical Tests for Comparing Supervised Classification Learning AlgorithmsNeural Computation 10(7), 1998
- With Little Power Comes Great ResponsibilityEMNLP 2020
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceICML 2024 — the rank rule this board uses
- A new way to read the leaderboard: rank spread2025 — why Arena now shows a best and a worst rank
- How to Correctly Report LLM-as-a-Judge Evaluations2025 · preprint
Contamination, and why items stay private
Withholding items does not prevent contamination. It makes the contamination hypothesis testable — and there is a published position against this posture, cited here rather than avoided.
- AI Evaluation Should Require Standardized Item-Level Data Releases2026 — the argument against this benchmark's posture · preprint
- A Careful Examination of LLM Performance on Grade School Arithmetic (GSM1k)NeurIPS 2024 D&B
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samples2023
- The reusable holdout: Preserving validity in adaptive data analysisScience 349(6248), 2015
- The Leaderboard IllusionNeurIPS 2025 D&B
- Black-Box Access is Insufficient for Rigorous AI AuditsFAccT 2024
The health-benchmark landscape this sits beside
Each of these occupies one or two of the three axes — quantization, on-device deployment, clinical harm — and none occupies all three.
- HealthBenchOpenAI, 2025
- Holistic evaluation of large language models for medical tasks (MedHELM)Nature Medicine 32, 2026
- AILuminate: v1.0 of the AI Risk and Reliability BenchmarkMLCommons, 2025
- Medicine on the Edge: On-Device LLMs for Clinical Reasoning2025 — closest prior work
- Beyond the Leaderboard: Rethinking Medical Benchmarks for LLMsACL 2026
- LiveBench: A Challenging, Contamination-Limited LLM BenchmarkICLR 2025 (Spotlight)