System One
v0.2Models making bounded clinical decisions: each answer is one of the options supplied with the case.
How to read this table
- Rank. Models clearly ahead, plus one; = is a tie. Clearly ahead means an exact paired test on the same cases tells them apart.
- Clinical decisions. Share of 993 decisions across Triage, Notes, Criteria, Coding, Frontier that are right with no severe error in the same message, note or patient. A severe error (an emergency sent to routine, a missed red flag, an allergy read as a medicine taken) zeroes its whole group. Unanswered cases count as wrong. The rank uses this column.
- Criteria, Coding, Frontier. The same score per suite. Not ranked separately. Triage and Notes, where the strongest models are near the ceiling, are in the expanded row.
- Frontier. Harder multi-document decisions, new in this version. A model without a Frontier score would have no score in the ranked column and be listed below the ranked models; every model on this board has one.
- Auto-accept, in the expanded row. Share of single-answer decisions accepted by a confidence threshold aimed at 95% or 99% precision. The threshold is set on one half of the cases and scored on the other; amber means the other half fell short of the aim. Not ranked.
- Holds, in the expanded row. Of the decisions a model got right, the share it still gets right when the same decision comes with the options reversed or reworded, irrelevant text, an injected instruction to answer wrongly, the correct option removed, or simply asked again. Not ranked.
- Baselines. Untrained word-overlap rule, for scale.
- Larger, API only. Larger means open weights over 4 GiB; “via API” means the result was measured through the provider's API; “local” beside it means the published weights run locally with the publisher's own code, and the score is still the API measurement. API only means there are no public weights at all.
- Speed and download. Median time per decision on Triage, Notes, Criteria and Coding, every model that runs on your own hardware timed on the same data-centre GPU, one NVIDIA A100 (API marks a round trip with the network), and every file the model ships, or the published files for a model measured via API. Not phone measurements.
- Cost / 1K. What the provider billed for 1,000 decisions on Triage, Notes, Criteria and Coding, read from each response's receipt when it was measured; prices change. A local model has no per-call fee (hardware and power are not counted).
- Expanded row. Every suite, how much it holds overall and under each robustness check, the earlier decision pilot (short, clean cases at the ceiling for the strongest models), auto-accept at 95% and 99%, and the build.
| Rank | Model | |||||||
|---|---|---|---|---|---|---|---|---|
| 1= | 88.286.1–90.152.19 GB | 95.5 | 93.3 | 69.4 | 623 msAPI | $0.051 | 52.19 GB | |
| 1= | 86.784.5–88.754.73 GB | 96.2 | 92.5 | 65.1 | 347 msA100 | $0local | 54.73 GB | |
| 1= | 86.684.3–88.652.22 GB | 91.7 | 93.3 | 67.0 | 493 msA100 | $0local | 52.22 GB | |
| 2 | 85.483.1–87.5API only | 95.5 | 91.7 | 60.5 | 434 msAPI | $0.048 | API only | |
| 4= | 84.381.9–86.452.21 GB | 94.9 | 92.5 | 58.6 | 434 msA100 | $0local | 52.21 GB | |
| 4= | 83.881.4–85.952.19 GB | 93.6 | 93.3 | 56.8 | 796 msAPI | $0.016 | 52.19 GB | |
| 5 | 82.880.3–85.052.20 GB | 92.9 | 91.7 | 54.9 | 246 msA100 | $0local | 52.20 GB | |
| 8= | 80.377.7–82.6API only | 88.5 | 88.3 | 50.9 | 210 msAPI | $0.014 | API only | |
| 8= | 80.177.5–82.4API only | 92.9 | 85.8 | 50.9 | 465 msAPI | $0.015 | API only | |
| 8= | 78.175.5–80.6API only | 87.2 | 82.5 | 50.9 | 417 msAPI | $0.031 | API only | |
| 10= | 76.974.2–79.554.03 GB | 89.7 | 86.7 | 41.7 | 351 msA100 | $0local | 54.03 GB | |
| 10= | 76.874.1–79.4API only | 89.1 | 83.3 | 41.4 | 358 msAPI | $0.044 | API only | |
| 10= | 76.673.9–79.218.84 GB | 90.4 | 82.5 | 39.8 | 160 msA100 | $0local | 18.84 GB | |
| 10= | 76.373.6–78.9API only | 93.6 | 87.5 | 37.3 | 490 msAPI | $0.090 | API only | |
| 10= | 76.273.5–78.816.82 GB | 89.7 | 82.5 | 42.9 | 87 msA100 | $0local | 16.82 GB | |
| 10= | 75.672.9–78.2API only | 82.1 | 81.7 | 44.8 | 1.1 sAPI | $0.087 | API only | |
| 11= | 75.472.7–78.056.05 GB | 91.7 | 84.2 | 38.0 | 198 msA100 | $0local | 56.05 GB | |
| 11= | 75.172.3–77.7API only | 89.7 | 85.0 | 36.4 | 345 msAPI | $0.044 | API only | |
| 11= | 74.872.0–77.454.99 GB | 90.4 | 81.7 | 37.7 | 630 msAPI | $0.217 | 54.99 GB | |
| 13 | 74.271.4–76.8API only | 82.7 | 85.8 | 37.7 | 358 msAPI | $0.046 | API only | |
| 18 | 72.769.9–75.471.93 GB | 82.1 | 84.2 | 34.3 | 107 msA100 | $0local | 71.93 GB | |
| 20 | 72.069.1–74.79.11 GB | 86.5 | 66.7 | 38.3 | 76 msA100 | $0local | 9.11 GB | |
| 21= | 71.468.5–74.117.94 GB | 82.7 | 80.8 | 34.6 | 587 msAPI | $0.029 | 17.94 GB | |
| 21= | 70.367.4–73.19.10 GB | 85.3 | 75.0 | 34.9 | 133 msA100 | $0local | 9.10 GB | |
| 22 | 69.766.8–72.517.99 GB | 83.3 | 84.2 | 34.3 | 171 msA100 | $0local | 17.99 GB | |
| 23 | 68.765.7–71.570.76 GB | 83.3 | 75.0 | 33.0 | 289 msAPI | $0.136 | 70.76 GB | |
| 24= | 68.065.0–70.89.47 GB | 84.6 | 82.5 | 27.5 | 150 msA100 | $0local | 9.47 GB | |
| 25= | 67.664.6–70.419.53 GB | 84.6 | 82.5 | 25.6 | 574 msA100 | $0local | 19.53 GB | |
| 24= | 67.664.6–70.43.07 GB | 82.7 | 73.3 | 34.0 | 299 msA100 | $0local | 3.07 GB | |
| 25= | 67.564.5–70.39.11 GB | 82.7 | 75.0 | 31.5 | 451 msA100 | $0local | 9.11 GB | |
| 26= | 66.763.7–69.519.08 GB | 80.1 | 77.5 | 22.2 | 443 msAPI | $0.081 | 19.08 GB | |
| 26= | 66.263.2–69.03.01 GB | 80.1 | 70.0 | 28.1 | 253 msA100 | $0local | 3.01 GB | |
| 28 | 65.262.1–68.12.73 GB | 82.1 | 74.2 | 28.1 | 128 msA100 | $0local | 2.73 GB | |
| 31 | 64.061.0–67.09.50 GB | 85.3 | 81.7 | 24.4 | 621 msAPI | $0.030 | 9.50 GB | |
| 32= | 63.560.5–66.54.45 GB | 69.2 | 71.7 | 31.5 | 59 msA100 | $0local | 4.45 GB | |
| 32= | 63.360.3–66.3API only | 66.7 | 76.7 | 26.2 | 906 msAPI | $0.087 | API only | |
| 34= | 62.559.5–65.59.34 GB | 83.3 | 75.0 | 24.1 | 447 msAPI | $0.054 | 9.34 GB | |
| 34= | 61.458.4–64.45.47 GB | 55.8 | 73.3 | 29.0 | 117 msA100 | $0local | 5.47 GB | |
| 37 | 59.856.7–62.83.85 GB | 63.5 | 74.2 | 24.7 | 35 msA100 | $0local | 3.85 GB | |
| 40= | 56.753.6–59.74.35 GB | 59.6 | 72.5 | 28.1 | 32 msA100 | $0local | 4.35 GB | |
| 40= | 56.052.9–59.12.03 GB | 59.6 | 65.0 | 26.2 | 78 msA100 | $0local | 2.03 GB | |
| 40= | 55.252.1–58.30.89 GB | 60.9 | 70.0 | 26.2 | 70 msA100 | $0local | 0.89 GB | |
| 42 | 52.549.4–55.64.64 GB | 57.1 | 60.0 | 25.3 | 174 msA100 | $0local | 4.64 GB | |
| 43= | 51.748.6–54.82.01 GB | 51.3 | 66.7 | 25.9 | 131 msA100 | $0local | 2.01 GB | |
| 43= | 50.447.2–53.51.52 GB | 52.6 | 62.5 | 29.3 | 105 msA100 | $0local | 1.52 GB | |
| 43= | 49.946.8–53.11.73 GB | 51.9 | 55.8 | 24.4 | 58 msA100 | $0local | 1.73 GB | |
| 44 | 48.345.2–51.41.73 GB | 51.3 | 53.3 | 31.5 | 185 msA100 | $0local | 1.73 GB | |
| 45 | 47.744.6–50.81.52 GB | 44.2 | 53.3 | 26.2 | 36 msA100 | $0local | 1.52 GB | |
| 49 | 41.438.4–44.51.22 GB | 45.5 | 44.2 | 24.1 | 33 msA100 | $0local | 1.22 GB | |
| 50= | 27.925.2–30.81.84 GB | 41.7 | 39.2 | 21.9 | 108 msA100 | $0local | 1.84 GB | |
| 50= | 24.922.3–27.70.30 GB | 30.1 | 37.5 | 27.2 | 34 msA100 | $0local | 0.30 GB | |
| 51 | 22.520.0–25.20.84 GB | 34.0 | 15.0 | 24.7 | 41 msA100 | $0local | 0.84 GB | |
| 53= | 17.415.2–19.90.35 GB | 34.6 | 39.2 | 7.7 | 14 msA100 | $0local | 0.35 GB | |
| 53= | 16.013.9–18.40.85 GB | 32.7 | 25.0 | 8.3 | 44 msA100 | $0local | 0.85 GB | |
| 55 | 4.93.8–6.51.95 GB | 19.9 | 15.0 | X | 47 msA100 | $0local | 1.95 GB | |
| Baselines | ||||||||
| — | Label overlapUntrained word-overlap rule | 21.6 | 30.1 | 26.7 | 24.4 | — | — | — |
Score against size and cost
Clinical decisions by parameter count, on a log scale. The solid step follows the best score at or below each size. Models with no public weights have no size; the best of them is the dashed line. Hover a point to see it.
- Phone builds (up to 4 GiB)
- Larger, open weights
- Best API only