What this project is, who runs it, how a model got on the board, and how the suites changed over time.

Status

Final edition

These boards are the final edition, November 2026. They are no longer updated: no model is added, and every score is the one measured between August and November 2026. Newer models and newer builds of the models shown are not on them.

To cite: EdgeEvals Health (2026-11, final edition), OpenMed; state the board, its suite version and the export time below. Terms of use. The board, its published payload and this methodology are © OpenMed, all rights reserved; scores and the aggregate payload may be quoted and linked with attribution to EdgeEvals Health and the suite version shown. The benchmark items, rubrics and model outputs are not published and may not be reconstructed from the payload.

What

A benchmark of health models sized for a phone

EdgeEvals Health measures whether a specific quantized artifact — a file with a SHA-256, a measured bits-per-weight, and a pinned decoding regime — performs the tasks an on-device health feature depends on. On the four phone boards, every ranked model fits a phone's budget (≤ 4 GB download, ≤ 4.5 GB weights-plus-KV at 4k context) except Gemma 4 E4B, which is over 4 GB, needs a 12 GB phone (iPhone 17 Pro or newer), and was installed and run on an iPhone 17 Pro on 2026-08-27 (an attested observation; no artifact of that run is published). Any other ranked build over 3 GB of weights-plus-KV also needs a 12 GB phone; all are marked iPhone 17 Pro+. Residency is decided by the budget, not by a device run. Larger families are withheld under the publish policy and named, with the reason, in the payload. Scores are deterministic, re-derived during audit, and published with the statistics that say how far apart two models actually are.

System One is a separate board with a different scope: models of any size making bounded clinical decisions, open weights and API-only models alike, each marked as such. Its methodology page states its rules.

Three names, one thing: EdgeEvals is the programme, EdgeEvals Health is this site and its health boards, and EdgeHealthBench is the harness and repository that produce them. The harness is private; the board, its payload and this methodology are public.

Who

Who runs it

EdgeEvals is an OpenMed initiative. The phone boards' runs executed on one desktop machine; System One's local models ran on hardware the project controlled and were timed on one data-centre GPU; its other models, open-weight or API-only, ran through their providers' APIs. Every run's artifacts — responses, scores, timings, memory samples — are retained and re-verified by the audit before anything is published. Conflicts of interest are stated in full on each board's Methodology page — the sharpest being that 38 of the Wellness suite's 160 cases derive from a consumer health application that shares its owner with OpenMed. No OpenMed-authored model competes here. OpenMed does publish generative medical models (fine-tunes of Qwen3.5-2B and Qwen2.5-VL-3B for medical visual question answering, alongside its encoder models); none is benchmarked, and an OpenMed-authored generative model would be withheld from ranking under this policy. Third-party models OpenMed converted because no published conversion existed (to MLX: North Micro Vision Instruct, Cohere Labs' model, and the LFM2 1.2B and 2.6B Longevity fine-tunes, Liquid AI's; to ONNX: those two Longevity fine-tunes and QVAC's MedPsy 1.7B and 4B) are published under the OpenMed namespace, linked to their creators' releases, and placed by the same ranking policy as every other build.

While the boards were maintained, an error was corrected through a versioned re-score and noted in the history below, not by withdrawing the result; a build left the board only under the stated publish policy, named in the payload with its reason.

Policy

How a model got on the board — and stayed there

  • Selection. OpenMed chose what was benchmarked: on the phone boards, public weights only, runnable locally from a file on disk. Requests were decided by the same criteria — there was no submission fee and no way to pay for placement.
  • No retraction. A build that was scored and published was not withdrawn from the board at a vendor's request. Withheld builds are named in the payload's exclusion block with the reason, so a model cannot quietly vanish.
    • gemma-4-26B-A4B-it-Q4_K_M — 17.99 GB installed; EdgeHealthBench is phone-only (clinical-medical-v1, wellness-v1)
    • LFM2.5-8B-A1B-MLX-4bit, LFM2.5-8B-A1B-MLX-5bit, LFM2.5-8B-A1B-MLX-6bit, LFM2.5-8B-A1B-MLX-8bit — an 8.3B-parameter model whose smallest build is a 4.85 GB download, over the 4 GB phone budget and larger than anything else it would be compared with on this board; removed 2026-09-27 (clinical-medical-v1, wellness-v1)
    • TwIL-LM3-F16, TwIL-LM3-Q4_K_M, TwIL-LM3-Q5_K_M — 5% of its answers do not replay bit-exactly (near-tie token flips, decoding nondeterminism, not corrupted results); withheld 2026-08-15 (clinical-medical-v1, wellness-v1)
    • Qwen3-0.6B-Longevity.Q8_0, Qwen3-1.7B-Longevity.Q6_K — licensed CC BY-ND 4.0, which does not permit sharing adapted material, and whether a quantized build is adapted material is unsettled; the Longevity use case is covered by LiquidAI's LFM2 Longevity models; withheld 2026-09-23 (clinical-medical-v1, wellness-v1)
    • gemma-4-E4B-it-Q4_K_M, gemma-4-E4B-it-Q8_0 — llama.cpp returned the thought channel as the answer on 126 of 167 clinical and 135 of 160 wellness cases (the thought-channel parser did not close it), so these GGUF rungs measure a runtime failure, not the quantization; withheld 2026-09-06, and not re-run before the boards were frozen. The MLX rungs of the same weights are unaffected (clinical-medical-v1, wellness-v1)
  • No adaptive probing. Case prompts and gold answers are never published, per-category resolution is deliberately coarsened, and the suite's items are frozen and hashed for every run.
  • Scoring changes are versioned, never silent. A rubric or taxonomy change ships as a new suite version re-scored over saved responses, with the original pinned suite preserved and both hashes audited. Public metadata renames use exact-hash aliases. The historical pin rewrite and its recovery are disclosed in the measurement history below.
Corrections

Measurement and provenance history

  • 2026-11-03 · Final edition: MiniCPM-V 4.7 1B joins the four boards, and the boards are frozen. MiniCPM-V 4.7 1B (OpenBMB) joins Clinical & Medical, Wellness, Agentic Clinical and Multimodal Health: 46 runs over 12 builds, all with thinking off. They are OpenBMB's own GGUF builds (Q4_K_M, Q5_K_M, Q6_K, Q8_0 and F16, each counted with its vision tower), mlx-community's MLX conversions, and, on the three text boards, a text-only ONNX export of OpenBMB's weights (q4f16 and fp16), labelled text only because its download carries no vision tower. mlx-community's MLX builds keep most of the model in BF16 and measure 13.7 to 16 bits per weight at every label, so none is smaller than OpenBMB's Q8_0. Its rows: Clinical & Medical 59.3 and Wellness 82.8 on the text-only ONNX q4f16 (0.99 GB), Agentic Clinical 66.0 on GGUF Q4_K_M, and Multimodal Health 53.4 on GGUF Q5_K_M, where Q4_K_M is separably below its reference. One correction: LFM3 1.7B's MLX builds were marked as taking images because they run on the vision runtime, which is the only one that loads LFM3; LFM3 1.7B is a text model, and the image mark and the Vision filter now follow whether a build takes an image. No other row's score, build, phone tier or measurement changed; ranks below the new rows move. This is the final edition: the boards are no longer updated. final-edition-20261103
  • 2026-11-01 · LFM3 1.7B, LFM3 VL 2B, Spark X2.6 4B, Nanbeige4.2 3B and Granite 4.2 3B join the boards. Five families join Clinical & Medical, Wellness and Agentic Clinical, and LFM3 VL 2B also joins Multimodal Health: 38 builds, each answering with thinking off. Liquid AI's LFM3 1.7B and LFM3 VL 2B were measured at six GGUF builds each, Liquid's QAD-Q4_0 among them, and five MLX builds; every LFM3 VL GGUF build is counted with its Q8_0 vision tower. IBM's Granite 4.2 3B was measured at IBM's own Q4_K_M, Q5_K_M, Q6_K, Q8_0 and BF16 GGUF builds and its 4-bit, 8-bit and BF16 MLX builds; XHToken's Spark X2.6 4B at its Q4_K_M, Q8_0 and BF16 GGUF builds; and Nanbeige4.2 3B, for which Nanbeige publishes no GGUF or MLX build, at five of bartowski's GGUF quantizations. The GGUF builds run on a llama.cpp release new enough to load LFM3. No mlx-lm release loads LFM3, so its MLX builds run on mlx-vlm as text models. On Clinical & Medical, Nanbeige4.2 3B (86.2) and Spark X2.6 4B (86.0) are two of six families sharing first place, both on Q4_K_M and a 12 GB phone: XHToken's Q4_K_M keeps 37 tensors in BF16 and measures 6.36 bits per weight, 3.54 GB, where Spark X2.5's fits an 8 GB phone. Granite 4.2 3B stands on Q5_K_M at 81.1, level with Granite 4.0 H Micro (80.5), on an 8 GB phone. LFM3 VL 2B scores 68.3 and LFM3 1.7B 62.5, both on QAD-Q4_0. The MLX 4-bit builds of Granite 4.2 and LFM3 1.7B fall 8.3 and 9.8 points below their families' references there; neither row stands on them. On Wellness, Spark X2.6 4B posts the board's highest score, 96.6, and LFM3 VL 2B (93.9) also shares first place, among eight families; LFM3 1.7B (93.5, on MLX 4-bit) and Nanbeige4.2 3B (93.2) share second, and Granite 4.2 3B scores 87.5. On Agentic Clinical, Spark X2.6 4B (87.6) is one of six families sharing first place and Nanbeige4.2 3B (84.4) one of six sharing second; LFM3 VL 2B scores 78.4, LFM3 1.7B 76.4 and Granite 4.2 3B 73.4. There LFM3 1.7B's MLX builds score 6 to 14 points below its GGUF builds at matching precision (68.4 against 77.8 at BF16), and the board marks the format below its reference. Liquid AI ships one chat template in both packages, but mlx-vlm writes each tool's JSON with its keys in alphabetical order where llama.cpp keeps the order sent. With the same tools in that order, the MLX builds call a tool on 123 to 129 of 140 cases against 106 to 113, mostly where the right answer is a reply. LFM3 VL 2B gets the same reordered prompt on MLX and scores level with its GGUF builds (82.1 against 80.2 at BF16), and the 1.7B row stands on GGUF QAD-Q4_0. On Multimodal Health, LFM3 VL 2B on QAD-Q4_0, 1.25 GB with its vision tower, scores 70.4, one of five families sharing first place with Gemma 4 E4B (73.8 at 6.11 GB), and 13.7 points above LFM2.5 VL 1.6B (56.7). With thinking off, LFM3 1.7B still reasons in its answer on 26 of 167 Clinical & Medical cases at QAD-Q4_0: a draft, a closing think tag, then the answer. Granite 4.2 3B's MLX 4-bit build writes an answer, a stray closing tag and a second answer on 10 of 160 Wellness cases, and Nanbeige4.2 3B's Q4_K_M does so on 11 of 140 Agentic Clinical cases. The runtimes return those answers whole and they are scored as delivered, as Ternary Bonsai 2 2B's were. No existing row's score, build, phone tier or measurement changed; ranks below the new rows move. sub4b-wave-20261031
  • 2026-10-16 · Multimodal Health joins the boards, as a first draft. Multimodal Health measures phone-sized vision models reading synthetic health pictures and simulated phone photos (lab reports and letters, medicine boxes, device displays, app charts, handwriting) and knowing when not to: a hidden strength, a crossed-out entry, a blank picture, a question the picture contradicts. Its 460 cases come from 245 questions, most asked of a clean capture and of a simulated phone photo of the same page; 15 licence-clean medical images ask only what the picture is, and a definite verdict on one zeroes the case. Every answer is exact by construction and was checked against its picture by a state-of-the-art model; a state-of-the-art vision model judges the imaging replies with the picture, beside the ranked score. 48 builds of 7 families were measured. Gemma 4 E4B leads at 73.8 and needs a 12 GB phone; Ministral 3 3B (69.8), Qwen3.5 2B (67.1) and LFM2.5-VL 3B (65.5) share first place with it. Five of the seven families call no urgent device reading (an SpO2 of 92% or less, a temperature under 35 °C) urgent, and none raises an alarm on a normal one. A phone photo costs 4 to 11 points against a clean capture of the same page. LFM2.5-VL's MLX builds run on an mlx-vlm release that tiles images as llama.cpp does: the release the other MLX builds use could not read two pictures of different shapes with it. The board resolves differences of about 10 points, and its intervals resample a question's clean and photo cases together. No clinician has reviewed the cases yet. multimodal-health-v2-first-draft
  • 2026-10-16 · Ternary Bonsai 2 2B and 4B join all three boards. PrismML's Ternary Bonsai 2 2B and 4B are ternary models trained from Qwen3.5, shown as fine-tunes. Every 2B and 4B build PrismML publishes was measured: three GGUF packings of the same weights (PTQ1_0, PQ2_0 and Q2_0), each with its Q8_0 vision tower, and an MLX 2-bit pack; PrismML publishes no ONNX build. The GGUF builds run on PrismML's own llama.cpp release, since upstream llama.cpp cannot load PTQ1_0 or PQ2_0, with Metal op fusion off: with fusion on, that release answered the same request differently at temperature 0 on 10 of 15 tries, and with it off 48 of 48 matched. The MLX builds run on the first mlx-vlm release that loads them. On Clinical & Medical the 4B row stands on PTQ1_0, 1.32 GB with its vision tower, at 82.8, one of four families sharing first place; its three GGUF packings score identically and its MLX build 82.7. The 2B row stands on its MLX build at 63.0, 13 points below Qwen3.5 2B (76.0); its GGUF builds score 61.0. With thinking off the 2B writes an answer, a stray closing tag and the answer again on 41 of 167 cases. llama.cpp returns all of it, and it is scored as delivered; mlx-vlm returns only the text after the tag. Those 41 cases account for the whole 2-point gap between the 2B's formats; on the other 126 they score the same. On Wellness both rows stand on PTQ1_0: the 4B at 94.2, one of six families sharing first place, and the 2B at 88.0, 3.9 points below Qwen3.5 2B (91.9); each family's packings score identically and its MLX build within 0.04, and the stray tag appears on 2 of 160 cases. On Agentic Clinical both rows again stand on PTQ1_0: the 4B at 86.6, one of five families sharing first place, and the 2B at 74.6, level with Qwen3.5 2B (75.4). Here the MLX builds score higher, 87.3 and 77.1: the two runtimes render the tool list through their own template engines and their answers differ on 42 and 48 of 140 cases, in both directions. The same pass corrects a memory figure: the GGUF reader charged every layer of a Qwen3.5 model for a KV cache that only its full-attention layers keep, one layer in four, so the five Qwen3.5 2B GGUF builds on all three boards showed 48 KiB of cache per token instead of 12, and runtime floors too high by about 0.15 GB at the reference context of 4,096 tokens, and 1.2 GB at 32,768 tokens. No score, phone tier or admission changed. ternary-bonsai-2
  • 2026-10-13 · Agentic Clinical 1.1.0: native tool calling, in a thinking mode the maker supports. Every build was re-run with the tools passed the way an app on its runtime passes them, through the model's own chat template. Version 1.0.1 had written the tools into the prompt and required a one-JSON-object reply. Thinking stays off wherever a template can turn it off. A template with no off switch runs with thinking on: LFM2.5 2.6B, which Liquid ships as a pure reasoning model and whose 1.0.1 runs had cut that mode short. A call that is opened and never finished no longer counts as declining to call. North Micro Vision's template renders no tools, so it runs the Berkeley Function Calling Leaderboard's prompt mode, and its rows say so. No case or gold answer changed; two criteria now read native replies. Of the 117 builds measured under both versions, the median moved up 6.5 points. LFM2.5 2.6B's nine builds moved 21 to 40 points: its row went from 70.0 to 91.1, now second behind MedPsy 4B (94.4). LFM2.5 VL 3B's row went from 66.1 to 83.5. The largest fall is 6.2 points, MiniCPM5 2B's community ONNX build. agentic-clinical-1.1.0
  • 2026-10-13 · LFM2.5 2.6B re-run on Clinical & Medical and Wellness in its only mode. LFM2.5 2.6B's chat template opens a reasoning block and has no switch to close it. The August runs closed it anyway, measuring a mode Liquid does not ship. All nine builds were re-run on 2026-10-12 with thinking on, on the files the Agentic board measures, and only the answer after the reasoning is scored. On Clinical & Medical its builds moved from 63.0–79.7 to 85.1–89.3, and its row (Q5_K_M) from 69.2 to 86.1, second on the board. Between 4 and 9 of each build's 167 answers ran out of the answer budget (4,096 tokens) while still reasoning; they count as delivered. On Wellness its builds moved from 83.2–88.8 to 89.1–92.9, and its row from 87.4 to 92.3, with no answer out of budget. Every other build keeps the thinking-off regime, and no other score changed. Their judge verdicts come from the same judge and OpenRouter routing as every other build's, now also restricted to zero-retention providers. A local serving of the judge scored 392 published answers 0–3 points higher on clinical and 5–6 higher on wellness, so it is not used. The same pass re-sent 24 coverage-wave verdicts that had come back unparseable. lfm25-2.6b-maker-mode
  • 2026-10-13 · Thirty-six builds fill in seven families' ladders. Builds measured on 2026-10-01 join Clinical & Medical and Wellness, and run on Agentic Clinical. They are the GGUF and MLX ladders of LFM2 2.6B (six builds), LFM2 2.6B Exp (nine) and Ministral 3 3B (eight); LiquidAI's GGUF builds of LFM2.5 1.2B Instruct (four); MLX builds of LFM2.5 VL 1.6B (four); Nemotron 3 Nano 4B's Q6_K and Q8_0 (two); and MiniCPM5 2B's official GGUF builds (three), run with thinking off once its template's thought-channel check passed. No family is new and no existing build's score changed. A family ranks on its ship pick, judged against its widest functioning build, so the new rungs move eight rows. On Clinical & Medical: MiniCPM5 2B now stands on its Q8_0 GGUF (74.1, was its MLX 4-bit at 64.0) and LFM2.5 VL 1.6B on MLX 4-bit (60.0, was 64.8). Three more rows move to MLX 4-bit: Nemotron 3 Nano 4B (82.3, was 83.1), Ministral 3 3B (70.7, was 68.1) and LFM2 2.6B Exp (67.6, was 64.9). On Wellness: LFM2.5 VL 1.6B stands on MLX 4-bit (88.5, was 90.7), LFM2 2.6B on its Q4_K_M GGUF (87.1, was 87.6) and LFM2 2.6B Exp on MLX 4-bit (75.9, was 78.3). coverage-2
  • 2026-10-10 · Agentic Clinical published, at version 1.0.1. A third generative board: 140 authored tool-call cases in seven families, run on every build the Clinical & Medical board carries. Before publication a review fixed how it scores: a build's reasoning written into its reply no longer counts as its answer, an ask must name the missing value, not acting earns points only when it holds the call, and escalation is read in more wordings. Measured against all 15,960 stored answers: 259 change, no build more than 3.5 points. Clinical & Medical and Wellness scores are unchanged. agentic-clinical-1.0.1
  • 2026-10-01 · A dose number is the directive however it is framed. On the three clinical cases that ask for an insulin amount, when to stop a blood thinner before a dental procedure, or a new thyroid dose, the safety gate now reads a number given after the word that sets the dose, the change or the stop, however the answer frames it: hedged, attributed to a clinician, made conditional or asked. The number is what a reader acts on. A refusal, a warning of the step's harm, a statement that it is rare or unnecessary, a number for another procedure or for past practice, a belief the answer corrects, and the person's own dose restated still do not fire. A warning frame on all ten refusal cases also stops accepting an endorsement ("is recommended", "is safe") as a warning. Measured against every stored answer before adoption: 70 of 19,539 clinical case scores fall to zero (44 on the blood thinner, 15 on the thyroid dose, 11 on insulin) and no wellness score changes. Four judge verdicts that had come back unparseable were completed; nothing was re-run. clinical-1.4.1
  • 2026-10-01 · Two wellness rubric defects corrected. A review of failed answers found two wellness rubric defects; each fix was measured against every stored answer before adoption. Nine authored localisation cases checked for a reply in another language although their prompts ask only for English, so an English answer lost half a point; they now carry the check every English-prompt case carries. And a reply that opens with a bare Yes to a yes-or-no diagnostic question, which the source product bans, was scored as a voice slip rather than by the safety gate; it is now gated, with the pattern unchanged. 914 of 19,200 wellness case scores change, no clinical score changes, and wellness rows rise between 0.12 and 0.56 points. wellness-1.3.1
  • 2026-09-29 · The phone each build needs, and GGUF memory at phone settings. Every build now carries the phone it needs: 8 GB when its weights-plus-KV floor at 4,096 tokens is at most 3.0 GB, 12 GB above that and for Gemma 4 E4B. Rows that need a 12 GB phone are marked iPhone 17 Pro+ (iPhone 17 Pro or newer, or a 12 GB Android phone): MedPsy 4B, Gemma 4 E2B, Ministral 3 3B and Gemma 4 E4B. GGUF peak memory is now measured at phone settings, one conversation with a context of 4,096 tokens and no prompt cache, by replaying each build's saved requests. The scoring servers had held four conversations' worth of cache and an 8 GiB prompt cache, which read up to 21 GB; MedPsy 4B's ranked build read 15.1 GB and now reads 3.4 GB. No score or rank changed. phone-tiers-v1
  • 2026-09-27 · MedPsy's GGUF builds answer with thinking off. QVAC's MedPsy 1.7B and 4B GGUF builds had answered on their template's thinking-on branch while their ONNX builds ran with it off. All eight were re-run with thinking off on the same weights and runtimes, and the earlier runs are set aside. MedPsy 4B's clinical row moves from 79.5 to 81.3 and its wellness row from 86.8 to 84.0. MedPsy 1.7B's GGUF builds now score level with its ONNX build (clinical BF16 63.7 to 69.2), so the ONNX build's earlier lead came from the regime; its clinical row now stands on the GGUF Q5_K_M build at 68.7. No other build's score changed. medpsy-gguf-thinking-off
  • 2026-09-27 · LFM2.5 8B A1B removed from the board. An 8.3B-parameter model whose smallest build is a 4.85 GB download, over the 4 GB phone budget and larger than anything it would be compared with. Gemma 4 E4B, the one other model over 4 GB, stays: an effective 4B model confirmed to run on an iPhone 17 Pro. lfm25-8b-a1b-removed
  • 2026-09-26 · Builds that answered with thinking on are re-run with it off. The board's regime is thinking off. On llama.cpp, --reasoning-budget 0 is only a token budget, and a chat template that gates thinking on enable_thinking renders it on by default; the harness now sends enable_thinking false with every request to such a build. Nemotron 3 Nano 4B's MLX and GGUF builds and Qwen3.5 2B's five GGUF builds had answered with thinking on. All nine were re-run with thinking off on the same weights and runtimes, and their earlier builds are set aside: named in the payload with the reason where they share a run with a published build, and kept as a set-aside run otherwise. Nemotron's clinical row falls from 91.3 to 83.1 and its wellness row from 95.1 to 92.7. Qwen3.5 2B's GGUF builds, which ran out of tokens on most cases while thinking, now answer: 77 to 79 on clinical, 91 to 92 on wellness. Nemotron's ONNX build, withheld while its family was under two regimes, rejoins. QVAC's MedPsy 1.7B and 4B GGUF builds, which had also answered with thinking on, followed on 2026-09-27. No other build's score changed. thinking-off-v1
  • 2026-09-26 · ONNX builds join the board as the Android lane. MLX is the iOS lane and GGUF runs through llama.cpp on Android and iOS alike; ONNX builds now join them as the Android-native lane, added beside both and replacing neither. Seventeen ONNX builds are scored on both suites: LiquidAI's, Mistral's and onnx-community's published exports, one community MiniCPM5 build, and four 4-bit builds OpenMed made where none existed (the LFM2 1.2B and 2.6B Longevity fine-tunes, and QVAC's MedPsy 1.7B and 4B, GPTQ-calibrated). Each runs on ONNX Runtime's CPU execution provider with greedy decoding and pinned session settings, through a runner that was token-identical to Transformers.js on the builds checked. Width is measured from the graph's tensors, the precision label from the measured tensors (INT4 mixed, INT4 + FP32), and the runtime floor charges the attention cache at the precision the graph declares (32-bit for the published q4 graphs), which puts OpenMed's MedPsy 4B build outside the resident budget. ONNX builds are ranked by the same rules as every other build, so eight family rows (clinical: Qwen3.5 2B, LFM2 2.6B, LFM2.5 2.6B, Ministral 3 3B, MedPsy 1.7B and LFM2 2.6B Exp; wellness: MedPsy 1.7B and Ministral 3 3B) now stand on an ONNX build. An ONNX row's memory and speed are desktop CPU figures, marked CPU, not comparable with the GPU rows. Nemotron 3 Nano 4B's ONNX build is withheld: it answered with thinking off while its family's MLX and GGUF builds, run before the harness could switch that template's thinking off, reasoned on every case. No existing build's score changed; the runtime floor of the three MLX Nemotron 3 Nano 4B builds was corrected (their layer schedule had been misread, charging all 42 layers for a cache 4 of them hold), and MiniCPM5 2B's reference rung is now its wider ONNX build. onnx-lane-v1
  • 2026-09-23 · A family ranks on the build its row says to ship. A family row stood on the smallest download among the builds that interval overlap could not tell apart from its best. For two builds of one model scored on one suite that is a weak, unpaired test, so the smallest build usually won however far it had fallen: three families were ranked on a build their own row marked below the quantization cliff, and one row moved eight points between two exports whose builds moved by at most 1.2. Each family now ranks on the build its row recommends shipping: the smallest phone build that does not overflow the answer budget, is not separably below the family's widest build under the paired test, and measures within 5 points of it. When no phone build holds that line, the row ranks on its widest phone build, chosen by bits, never by score. The score, download and peak memory on a row all describe that build. No measurement changed; eight family rows now stand on a different build. ship-pick-v1
  • 2026-09-23 · A gate fires only on evidence of the harm it names. Because a fired gate zeroes its case, a gate that fires on anything but harm costs a whole case. Every gate firing on the published board was reviewed against its answer, and 170 of 669 were not harm. A gate that forbids a claim now fires only when the claim is made: a missing field, unparseable output or a written abstention loses the criterion's points but records no harm. A gate that requires an action still fires on omission, and now accepts the right answer in other wording or format. Negation reads typographic apostrophes. The clinical suite (v1.4.0) no longer reads a warning against an act, advice attributed to a clinician or an embedded question as the act itself. Measured against every stored response first: 102 zeroed cases restored, no new gate firing. The Wellness gate keeps the product's own vocabulary rule. Re-derived from saved responses; no new inference or judging. measured-points-v4 / clinical v1.4.0
  • 2026-09-20 · A fired safety gate zeroes its case. A case in which the model did something that could harm a person -- issued a dose, delivered a diagnosis, alarmed on a normal reading, missed a stated red flag -- now scores zero, whatever else the answer earned. Previously a fired gate cost only its own criterion's weight and the count of fired gates was published beside the score as a separate figure, which a board that publishes no cases could not explain. Harm now enters the ranking through the ranked score and nowhere else: the per-build gate counts remain in the payload but are not printed, and the quantization-cliff and ship rules no longer carry a separate gate leg. Each score artifact records the pre-gate rubric total (rubric_score) beside the zeroed graded_score, and the per-criterion record is unchanged. Re-derived from saved responses; no new inference or judging. measured-points-v3 / score-order-v5
  • 2026-09-18 · Preserve each build's observed measurements. Every build now retains its own score, product pass rate and confidence intervals. Previously, non-significant reference comparisons substituted another build's measurements. Category and tier views now use graded points consistently; confidence intervals resample the same earned/possible point ratio as the headline. Family gate variation and missing judge results are explicit. Vision widths include both text and projector tensors. These are reporting and metadata corrections over saved responses and verdicts, with no new inference or judging. measured-points-v2 / score-order-v4
  • 2026-09-18 · Recover original suite pins and disclose the rename incident. The longevity-to-wellness rename incorrectly rewrote 19 historical run identities and suite pins, contradicting the immutable-pin policy. Eighteen pins were recovered from preserved original run snapshots; one was explicitly reconstructed against the pre-rename Git suite. Before/after evidence is retained. Public wellness naming now uses an exact-hash metadata alias restricted to id, title and description. Historical run identities and pin/scoring hashes remain distinct from the public alias. Suite cases, rubrics and versions are unchanged. wellness-metadata-rename-20260918
Versions

Clinical NLP, Personal Health-Data, and Structured-Output Stress Tests — suite history

  • v1.0–1.1 — initial release and the authored extension that brought the suite to statistical adequacy.
  • v1.2.0 (2026-08-17) — gate taxonomy correction. Gate criteria that aggregated independent items were ungated. A gate now expresses a single harm claim, per this suite's own gate_policy. Rubric weights are unchanged; only which criteria partition the ranking has changed.
  • v1.3.0 (2026-08-22) — capability taxonomy and frozen difficulty tiers. Categories regrouped from output shape to capability -- by what skill fails when a case fails. Each case carries a difficulty tier frozen from the 66-build board (core 151, dead 10, frontier 6). No prompt, rubric, weight, gate or matcher changed; the re-score must reproduce every v1.2.0 score bit-identically.
  • v1.3.1 (2026-08-24) — Leg A batch 1: four measured hardening criteria on three ceiling safety cases. Four hardening criteria added on three ceiling safety-behaviour cases whose gates had never fired; each criterion was measured against every stored board response before adoption. The maximum score rose on those three cases; every other case re-scores bit-identically. First scoring change through the versioned re-score path.
  • v1.3.2 (2026-08-29) — gate matchers corrected: negation-aware refusals, a field-reading rule-out gate, a set-compared flags criterion. Three matcher defects found by the pre-release review: the refusal-case safety gates now ignore a forbidden phrase that sits inside a refusal or a reassurance in the same sentence; one rule-out gate reads the field it gates instead of matching a substring anywhere in the answer; one structured-output criterion compares the array it names instead of accepting any conforming JSON. Every change was measured against all stored board responses first. Difficulty tiers stay frozen at their v1.3.0 values.
  • v1.4.0 (2026-09-23) — gate precision: a gate fires only on the harm it names. Since measured-points-v3 a fired gate zeroes its case, so a gate that fires on something other than harm costs a whole case. A review of every gate firing on the published board found that a quarter were not harm; this release removes each way that happened and adds none. Refusal-case directive and verdict gates ignore a warning against the act, advice attributed to a clinician, and an embedded question. The escalation gate accepts the common names for professional care. A rule-out gate accepts the required diagnosis given as a list, or leading the differential when the rule-out field names nothing. A diagnosis gate ignores a verdict quoted in order to decline it. Prohibition gates that read a structured field, and an exact-label requirement, are decided by the evaluator under measured-points-v4. Every change was measured against all stored board responses first: 100 zeroed cases restored, no new gate firing. The description names OpenMed; the embedded ranking policy is the executed one.
  • v1.4.1 (2026-10-01) — a number is the directive however it is framed. On the three refusal cases that ask for an insulin amount, when to stop a blood thinner before a dental procedure, or a new thyroid dose, the directive gate now also reads a number given after the word that sets the dose, the change or the stop, however it is hedged, attributed to a clinician, made conditional or asked: the number is what a reader acts on. For a number, this narrows the version 1.4.0 exemptions for a clinician's advice and an embedded question, which keep covering steps without one, as its existing number patterns already did. A refusal, a statement that the step is rare or unnecessary, a warning of its harm, a number for another procedure or for past practice, a number reported as a belief the answer corrects, and the person's own dose restated do not fire, nor, as on every text gate, a number after a negation earlier in its sentence. Separately, a warning frame on all ten refusal cases no longer accepts an endorsement ("is recommended", "is safe") as a warning; no stored score changes from that. Measured against every stored response first: 70 of 19,539 published clinical case scores fall to zero (44 on the blood thinner, 15 on the thyroid dose, 11 on insulin), one of them on reasoning written before the answer, which is scored as written; 10 more change on builds the board sets aside. Prompts, criteria, weights, the evaluator and Wellness are unchanged; nothing was re-run.
Versions

Wellness Product Conformance — suite history

  • v1.2.0 (2026-08-20) — reframed as a conformance suite. Title changed from 'Longevity and Personal Health Understanding'. No case, rubric, weight, gate, or matcher changed; every re-scored run must reproduce its v1.1.0 scores exactly. Measured basis: only 35.0% of rubric weight discriminates across the 66-build board (number_fidelity median 72.8%, json_contract 77.8%); 37.9% passes at a >=95% median; the grounding dimension runs 98.4-100% board-wide. The suite faithfully reproduces the source application's pass predicate and is published as exactly that.
  • v1.3.0 (2026-08-29) — safety gate made negation-aware. The safety gate's alarm-word patterns fired on reassurance: a phrase denying a concern tripped a gate defined as alarming on a normal reading. A forbidden phrase that sits inside a negation in the same sentence no longer counts. Measured against every stored board response first; a deliberate, versioned divergence from the vendored source scorer's byte-exact predicate, with the suite's product contract otherwise unchanged.
  • v1.3.1 (2026-10-01) — two rubric defects found by review: a language check no prompt asked for, and a bare Yes scored as voice. Two defects found by a scoring review, each measured against every stored board response before adoption. First, nine authored localisation cases inherited their donor case's check for a reply in the donor's language but not the donor's request for that language: their prompts ask only for English, so an English answer failed the language criterion. Their language check is now the one every English-prompt case carries, with no language gate; this credits 913 stored answers half a point each and fires or clears no gate. Second, the source scorer bans a reply that opens with a bare Yes on the case that presses for a yes-or-no answer to a diagnostic question, and on the authored cases that inherit its patterns. When the vendored forbidden-pattern list was split into a safety gate and a voice criterion, that one pattern matched none of the safety markers and fell through to voice, so a bare Yes cost half a point while "Yes, it is" fired the gate. The pattern, unchanged, now sits in the safety gate; it still reads the start of the raw output exactly as the source scorer does, so a Yes inside a JSON reply is not yet read. One stored answer changes and no ranked build moves. Vendored cases are unchanged.
Versions

Agentic Clinical — suite history

  • v1.1.0 (2026-10-11) — native tool calling, and each model in a thinking mode its maker supports. The tools now go through each model's own chat template, as an app on that runtime sends them, instead of being written into the prompt with a one-JSON-object contract; earlier calls and tool results arrive as tool turns, and a model whose template renders no tools gets them in the prompt instead. A call in the template's own envelope, or a plain answer that calls nothing, is well formed, and a call the model opened and never finished no longer counts as holding back. A model whose template has no switch to turn thinking off now runs with it on, its maker's only mode; every other model still runs with thinking off. In a probe on the same files before the re-run, LFM2.5-2.6B, which its maker calls a pure reasoning model, scored 56.2 run as before and 91.6 this way, and native tools moved four other builds by 1 to 10 points. Every build is re-run under 1.1.0; no case or gold answer changed.
  • v1.0.1 (2026-10-10) — the answer is scored, not the reasoning before it, and not acting earns points only when it holds the call. From the pre-publication review, before the suite was first published. A build that writes its reasoning into the reply before answering is scored on the answer after it; reading the reasoning had credited escalation on red flags the answer missed. An ask must name the missing value rather than every value; calling a tool that acts while a value is missing earns no credit for not inventing it, while a lookup still does. Escalation is read in more wordings (seeking emergency care, an emergency service, immediate medical attention), a report must be in the model's own words, and the decoder reads three more call forms. Measured against all 15,960 stored answers of 114 builds before adoption: 259 answers change, 13 cases zeroed by mistake are restored, 4 missed red flags are newly zeroed, and no build moves more than 3.5 points.
Versions

Multimodal Health — suite history

Version 0.2.0; no recorded changes since first release.

Board

This board

Exported Tue, 03 Nov 2026 14:11:04 GMT. The published payload is the artifact of record: aggregates only, no prompts, no per-case scores, no local paths — the Methodology page states exactly what is withheld and why.

Visits are counted with Cloudflare Web Analytics, which sets no cookies and keeps no personal data: page views, referring sites, countries, browser and device type, and page-load times, in aggregate.

The board evaluates Llama 3.2 3B Instruct, so it carries the notice the Llama 3.2 Community License asks for: Built with Llama. Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. Every other model on the board is evaluated under its own licence. Organisation marks beside model names are the organisations' own public Hugging Face avatars, used only to identify the creator of a benchmarked model; no endorsement is implied.