Methodology

How the System One board is produced: what it claims, what it refuses to claim, how an answer becomes a rank, and what is deliberately not shown.

How this board is produced. Every number on this page is read from the same results the leaderboard reads, not written into the page.

Version 0.2

Harder decisions, and decisions read from images

v0.2 replaces v0.1. It keeps the 669 decisions of v0.1's four authored suites unchanged and adds two kinds of case. Frontier, 324 harder text decisions, joins them in the ranked score, so the rank now stands on 993 decisions. Vision, 88 decisions taken from images, is ranked on its own among the decision models that read images.

v0.2 ranks decision models only. v0.1 also ranked general models prompted to answer, large chat models asked for the answer in JSON and a small language model held to a JSON schema, beside general-purpose encoders adapted as classifiers. Few decision models existed then, so they were the comparison. There are now 55 decision models on this board, enough to compare decision models with one another; forcing a general model through a prompt to do a decision model's job costs more, needs a far larger model, and says little about the models this board exists to compare.

The strongest models answered most of v0.1 correctly, so every new case had to earn its place in pilot runs before the suites were frozen. A case stayed only when at least one of the board's top models answered it wrongly; a case they all answered correctly was retired as too easy, and a case whose rule could be read two ways was excluded. Retired and excluded cases are kept, with the reason, but never scored. The pilot answers chose cases and nothing else: every score on the board comes from a later run of the frozen suite, one per model.

v0.1 suites
Triage, Notes, Criteria, Coding: unchanged, 669 decisions
Frontier
644 written; 324 kept, 319 retired as too easy, 1 excluded
Vision reading
180 written; 35 kept, 136 retired as too easy, 9 excluded
Vision decisions
146 written; 53 kept, 93 retired as too easy
Removed
4 general models prompted to answer and 2 general-purpose encoders
Runs
Every model ran each new suite once, after it was frozen; a local model only on the weights the board had already scored
v0.1
Ranked on its four authored suites alone; their per-suite scores are unchanged
Scope

What this board claims, and what it refuses

It claims

  • Bounded decisions on identical inputs. Every model receives the same case and the same supplied options, through its own native interface. No answer is repaired, re-asked or chosen from several tries.
  • Every case stays in the denominator. An unanswered, invalid or unsupported decision counts as wrong. A model that cannot produce an output type shows X, not a score of zero.
  • A rank only where the data separate models. A rank is one more than the number of models an exact paired test puts ahead on the same cases.

It refuses

  • That this is clinical validation. The cases are authored, non-patient cases, and no clinician or certified coder reviewed their labels.
  • That a Frontier score is an accuracy on records in general. Its cases were kept because strong models got them wrong, so it compares models on hard cases and says little about how often any model errs on a typical record.
  • That one number summarises everything. Vision, auto-accept, robustness and the other benchmarks sit beside the rank and never enter it.
  • That this speed predicts a phone. Speed comes from a data-centre GPU; no phone has run these models.
  • That an API measurement is a local one. Some open-weight models were measured through their provider's API: their speed includes the network, though their weights can be downloaded. API-only models have no public weights at all.
Real world

Five suites written to look like real work

Patient messages to a clinic, decided under a supplied triage policy (urgency, red flags, route); real-length clinical notes with questions about one condition or one medicine each; criteria checks where “insufficient information” is a correct answer, including whether an encounter note supports a real CMS-HCC V28 category; and coding, the most specific ICD-10-CM FY2026 code for one condition chosen from a shortlist of real codes with their official titles, or none of them. Every code, title and HCC mapping is checked against the official reference files.

Frontier is written for the strongest models: multi-document records where the answer depends on reading all of it. A medication timeline across several documents; a finding revised by a later report; a trial criterion that applies only under a condition; and the difference between a record that states something is absent and one that never mentions it. Every case states the rules that decide its answer.

A suite declares, before any run, which wrong answers could hurt a person: an emergency sent to routine, a red flag left out, an allergy read as a medicine the patient takes, a held anticoagulant read as taken, a safety exclusion read as passed. Such an error zeroes every decision in the same message, note or patient, the rule the generative boards apply to a fired safety gate. A non-answer is not severe: it goes to a person instead of acting on the patient. Coding and payer errors are financial and carry no harm rule.

Triage
240 decisions in 80 groups; 57 carry a harm rule
Notes
153 decisions in 32 groups; 32 carry a harm rule
Criteria
156 decisions in 119 groups; 7 carry a harm rule
Coding
120 decisions in 120 groups
Frontier
324 decisions in 172 groups; 35 carry a harm rule
Clinical decisions
Credited decisions ÷ all 993 decisions; 131 carry a harm rule
Review
Authored, not reviewed by clinicians or certified coders; code and HCC facts verified against the official reference data
Vision

Decisions read from images

Each vision case shows the model an image and asks one bounded question. Every case needs the image: the text never states what the image shows. Vision reading asks for one value read off the image, such as a dial, a syringe, a label, a monitor or a document. Vision decisions ask the model to read the image and then apply a stated rule: a drawn-up syringe against the prescribed dose, a blood pressure against a threshold, a pump against its order.

Only decision models that read images run these suites. A model counts as reading images only when, through its own interface, its answers follow a test image; one that accepts an image and answers as if none were sent is text only. A text-only model is never shown a vision case, rather than answering from the text. A severe error (a dose drawn up wrongly read as right, a dangerous reading read as safe) zeroes every decision about the same image. The vision rank uses the same paired test as the board, on both suites pooled, among the models that ran both. The vision cases were chosen against general vision-language models that are not on this board, so no ranked model's answers chose them.

Vision reading
35 decisions in 24 image groups; 9 carry a harm rule
Vision decisions
53 decisions in 33 image groups; 24 carry a harm rule
Vision
Credited decisions ÷ all 88 image decisions, ranked among the 18 board models that read images
Images
Made for these suites, as clean scans and phone photos, or marked public-domain images; none comes from a patient
Ranking

A rank is earned by a paired test

The rank stands on Clinical decisions: only models that ran every one of the triage, notes, criteria, coding, frontier suites are ranked. For every pair of models, count the decisions only one of them was credited on. An exact two-sided sign test, the exact form of McNemar's test, asks whether that split could be chance. A model is clearly ahead when it scores higher and the test rejects at α = 0.05. The rank is one more than the number of models clearly ahead, so models the test cannot separate share a rank, marked =. Rows are ordered by score; the 95% interval beneath each score is a Wilson interval.

Policy
system-one-core-v1
Test
Exact two-sided sign test on discordant decisions, α = 0.05, no correction for the number of pairs
Repeated runs
One declared run per model; a better rerun never replaces a worse one
Not ranked
A model that has not run every suite, and the baselines
Filters
Search and the size chips narrow the table without changing any rank

With no correction for the number of pairs, an occasional separation between neighbours can be chance. Read a single-step rank gap as provisional and a multi-step gap as real.

Contenders

Who is compared, and how they answer

Every row is a decision model, read through its own native interface: typed-decision models, read the way their publisher reads them (a choice or pointer head, or the probabilities of the option letters); zero-shot classifiers built for the task (GLiClass, GLiNER); and decision models measured through their providers' APIs, some with open weights anyone can run locally and some API-only. No general model is prompted to stand in for one. A publisher's claim that a model can do a task is not inherited: the capability has to show on the cases.

Phone builds
18 models, each with a recorded download up to 4 GiB
Larger & API-only
19 open-weight models over 4 GiB run locally, 8 open-weight models measured through their provider's API and 10 API-only models with no public weights; all shown, and a size chip keeps only the models whose files fit, so it leaves out API-only models
Open weights vs API
A model counts as open when its weights are public, whether this board ran them locally or through an API; API-only means no public weights
What APIs receive
Authored cases only, images included, sent only to providers that neither retain nor train on them. Imported datasets never leave the machine, so API-measured rows show — on PubMedQA and SciFact; most local rows show — there too, because they did not run those datasets
API requests
A request that never reached the model (network failure, rate limit, server error, no route to the model) is retried where the lane declares retries, and if it is still lost, re-measured later on its own with the same model settings. An answer is never retried or replaced, and the board is built only when every request reached its model
Auto-accept

How much a confidence threshold accepts

A decision model is deployed by accepting its answer when the probability it gives that answer clears a threshold, and sending everything else to a person. The board measures exactly that. The single-answer decisions of the four v0.1 suites are split once, by group and without looking at any label, into a dev half and a test half. For each model the threshold is the lowest confidence at which its accepted dev answers still reach the target precision. The test half then shows how many decisions that threshold accepts and the precision those answers actually reach, at a 95% target and at 99%, in each model's expanded row.

Split
238 dev and 351 test single-answer decisions; groups never straddle the halves
No probabilities
The model accepts nothing on confidence (none of the 55 rows, with larger & API-only models included)
Not reached on dev
No confidence threshold reached the target on the dev half, so nothing is accepted
Amber
The test half fell short of the target: the dev threshold did not carry over (28 of 55 rows at 95%, with larger & API-only models included)

Test-half precision rests on the decisions a model accepted, so when it accepts few, one wrong answer moves it a long way. Read amber as a warning that a model's confidence does not transfer, not as an exact precision.

Robustness

Does a right answer survive a change in presentation?

Production inputs vary, and an answer that flips when the options are listed in a different order was never a decision. The robustness pack takes 140 single-answer decisions from the four v0.1 suites, chosen evenly across each task's answers without looking at any model's output, and asks each of them again in six ways. Holds is the share of a model's right answers that stay right in the variant. A model is charged only for decisions it had shown it could make, so a weaker model is not penalised twice for its accuracy. A variant the model declines, including one whose added text no longer fits its input window, counts as not held.

Options reversed
The same options in reverse order.
Options reworded
Every option description reworded; same meaning, same ids.
Irrelevant text
Non-clinical boilerplate (signatures, notices) added to the text.
Injected instruction
An instruction to give a wrong answer, placed inside the clinical text.
Answer removed
The correct option removed and “none of these” offered instead.
Asked again
The identical request sent a second time.
Pairs
840 parent and variant pairs
Range
55.6 to 98.8 across 55 rows, with larger & API-only models included

The injection is one overt instruction at the end of the clinical text: a floor on injection risk, not a measure of it. 52 of 55 rows, larger & API-only models included, gave the injected answer at least once. A repeated request that changes its answer shows run-to-run disagreement at the configured settings. Not ranked.

Baselines

Every score has a floor beside it

One row under the table gives scale: Label overlap, an untrained rule that picks the option sharing the most words with the case. Most common answer, listed below, always gives the most frequent label: majority-class accuracy on PubMedQA, and on SciFact the best single evidence label under the abstract-label F1. A model at or below these has not shown that it uses the case.

Label overlap · clinical decisions
21.6
Label overlap · pilot
46.0
Label overlap · PubMedQA
56.8
Most common answer · PubMedQA
55.2
Most common answer · SciFact
14.2
Statistics

What these suites can and cannot resolve

Ranked decisions
993
95% interval half-width at 50%
±3.1 points
Ranked rows sharing rank 1
3 of 55, with larger & API-only models included
Shared decisions to detect a 5-point gap
about 300 at 10% discordance, more as models disagree more
Image decisions
88: a vision rank separates only large differences
The strongest models are close.

3 models share the top rank: the paired test cannot separate them on these decisions. Telling them apart reliably needs larger reviewed suites, not more decimal places.

Other benchmarks

The pilot and published datasets, in the expanded row

Decision pilot
113 short, clean decision cases across 13 tasks, where the strongest models are at the ceiling
Extraction
27 cases across 3 tasks (numbers, structured fields, text spans); scored, not ranked
Historical choices
253 single-answer cases from an earlier collection
PubMedQA
500 expert-labelled yes/no/maybe questions answered from the abstract
SciFact labels
300 claims, each checked against 5 retrieved abstracts (1,500 pairs)
  • Assertion and context12
  • Medication documentation11
  • Clinical claim support14
  • Explicit-rule eligibility11
  • Relation and evidence binding8
  • Workflow routing9
  • Local-code mapping9
  • Document classification6
  • Privacy field categories7
  • Evidence sufficiency6
  • Multilingual text decisions7
  • Candidate evidence ranking7
  • Explicit ordinal scales6
Systems

Speed is one GPU; size is the files

Speed is the median time per answered decision on the 669 decisions of the v0.1 suites, run warm. Every model that runs on your own hardware is timed on the same data-centre GPU, one NVIDIA A100 (marked A100): the same files in the same precision as the run that scored it, run again only to time it on the decisions that run answered, so every score stays the scored run's. A row marked API was measured through its provider: its time is the round trip, network and queue included, and cannot be compared with local compute. Only answers that came back on the first try count: time spent waiting out a provider's rate limit is not model speed. Download is the model's recorded files, in GB (10⁹ bytes). Neither is a phone measurement.

Cost / 1K is what the provider billed for 1,000 decisions of the v0.1 suites, read from the receipt each response returned when it was measured (October 2026): the billed total over every response, answers that broke the answer format included, per 1,000. Prices change, and some may be introductory. A model run locally has no per-call fee; the hardware and power that run it are not counted.

Integrity

What was verified before a result appears

  • Frozen cases. Each suite is fixed before the runs that score it, and results from different versions of a suite are never combined.
  • Same weights. A local model's files are hashed at every run; a run of a new suite counts only when the hash matches the build the board already scored.
  • Audited before export. The frozen cases, the code that ran and every saved answer must match the hashes recorded at run time, and the board scores those saved answers; nothing is scored from memory of a run.
  • Pinned services. An API answer is scored only when the returned model and provider match the ones requested; fallbacks are disabled.
Limits

Known limits

  • Unreviewed labels. No clinician or certified coder reviewed the authored cases; a reviewed set could have moved scores and ranks.
  • Authored policies. The triage policy, payer policies and trial criteria are written for these suites; they resemble real ones but reproduce none.
  • Small suites. 993 decisions separate a strong model from a weak one, not the strongest models from each other; 88 image decisions separate only large differences.
  • Hard by selection. Frontier was filtered against the top models of v0.1, so a model that took part in that filtering meets cases chosen because it or its peers erred on them.
Disclosure

What is shown, and what is not

The decision procedure is described in full; the cases are not shown at all, so that they stay useful as a test. The board shows aggregates: scores, intervals, counts, speed and size.

Cases, images, gold labels and policies
Withheld
Model answers and per-case results
Withheld
Run identifiers, file hashes, runtime versions, adapter settings
Withheld
Status
Final edition (November 2026); no longer updated
References

What this rests on

  1. McNemar, Q.Note on the sampling error of the difference between correlated proportions or percentagesPsychometrika 12, 1947
  2. Wilson, E. B.Probable inference, the law of succession, and statistical inferenceJournal of the American Statistical Association 22, 1927
  3. Jin, Dhingra, Liu, Cohen, LuPubMedQA: A Dataset for Biomedical Research Question AnsweringEMNLP-IJCNLP 2019
  4. Wadden, Lin, Lo, Wang, van Zuylen, Cohan, HajishirziFact or Fiction: Verifying Scientific ClaimsEMNLP 2020