System One

v0.2

Models making bounded clinical decisions: each answer is one of the options supplied with the case.

How to read this table
  • Rank. Models clearly ahead, plus one; = is a tie. Clearly ahead means an exact paired test on the same cases tells them apart.
  • Clinical decisions. Share of 993 decisions across Triage, Notes, Criteria, Coding, Frontier that are right with no severe error in the same message, note or patient. A severe error (an emergency sent to routine, a missed red flag, an allergy read as a medicine taken) zeroes its whole group. Unanswered cases count as wrong. The rank uses this column.
  • Criteria, Coding, Frontier. The same score per suite. Not ranked separately. Triage and Notes, where the strongest models are near the ceiling, are in the expanded row.
  • Frontier. Harder multi-document decisions, new in this version. A model without a Frontier score would have no score in the ranked column and be listed below the ranked models; every model on this board has one.
  • Auto-accept, in the expanded row. Share of single-answer decisions accepted by a confidence threshold aimed at 95% or 99% precision. The threshold is set on one half of the cases and scored on the other; amber means the other half fell short of the aim. Not ranked.
  • Holds, in the expanded row. Of the decisions a model got right, the share it still gets right when the same decision comes with the options reversed or reworded, irrelevant text, an injected instruction to answer wrongly, the correct option removed, or simply asked again. Not ranked.
  • Baselines. Untrained word-overlap rule, for scale.
  • Larger, API only. Larger means open weights over 4 GiB; “via API” means the result was measured through the provider's API; “local” beside it means the published weights run locally with the publisher's own code, and the score is still the API measurement. API only means there are no public weights at all.
  • Speed and download. Median time per decision on Triage, Notes, Criteria and Coding, every model that runs on your own hardware timed on the same data-centre GPU, one NVIDIA A100 (API marks a round trip with the network), and every file the model ships, or the published files for a model measured via API. Not phone measurements.
  • Cost / 1K. What the provider billed for 1,000 decisions on Triage, Notes, Criteria and Coding, read from each response's receipt when it was measured; prices change. A local model has no per-call fee (hardware and power are not counted).
  • Expanded row. Every suite, how much it holds overall and under each robustness check, the earlier decision pilot (short, clean cases at the ceiling for the strongest models), auto-accept at 95% and 99%, and the build.
System One decision leaderboard
RankModel
1=88.286.1–90.152.19 GB95.593.369.4623 msAPI$0.05152.19 GB
1=86.784.5–88.754.73 GB96.292.565.1347 msA100$0local54.73 GB
1=86.684.3–88.652.22 GB91.793.367.0493 msA100$0local52.22 GB
285.483.1–87.5API only95.591.760.5434 msAPI$0.048API only
4=84.381.9–86.452.21 GB94.992.558.6434 msA100$0local52.21 GB
4=83.881.4–85.952.19 GB93.693.356.8796 msAPI$0.01652.19 GB
582.880.3–85.052.20 GB92.991.754.9246 msA100$0local52.20 GB
8=80.377.7–82.6API only88.588.350.9210 msAPI$0.014API only
8=80.177.5–82.4API only92.985.850.9465 msAPI$0.015API only
8=78.175.5–80.6API only87.282.550.9417 msAPI$0.031API only
10=76.974.2–79.554.03 GB89.786.741.7351 msA100$0local54.03 GB
10=76.874.1–79.4API only89.183.341.4358 msAPI$0.044API only
10=76.673.9–79.218.84 GB90.482.539.8160 msA100$0local18.84 GB
10=76.373.6–78.9API only93.687.537.3490 msAPI$0.090API only
10=76.273.5–78.816.82 GB89.782.542.987 msA100$0local16.82 GB
10=75.672.9–78.2API only82.181.744.81.1 sAPI$0.087API only
11=75.472.7–78.056.05 GB91.784.238.0198 msA100$0local56.05 GB
11=75.172.3–77.7API only89.785.036.4345 msAPI$0.044API only
11=74.872.0–77.454.99 GB90.481.737.7630 msAPI$0.21754.99 GB
1374.271.4–76.8API only82.785.837.7358 msAPI$0.046API only
1872.769.9–75.471.93 GB82.184.234.3107 msA100$0local71.93 GB
2072.069.1–74.79.11 GB86.566.738.376 msA100$0local9.11 GB
21=71.468.5–74.117.94 GB82.780.834.6587 msAPI$0.02917.94 GB
21=70.367.4–73.19.10 GB85.375.034.9133 msA100$0local9.10 GB
2269.766.8–72.517.99 GB83.384.234.3171 msA100$0local17.99 GB
2368.765.7–71.570.76 GB83.375.033.0289 msAPI$0.13670.76 GB
24=68.065.0–70.89.47 GB84.682.527.5150 msA100$0local9.47 GB
25=67.664.6–70.419.53 GB84.682.525.6574 msA100$0local19.53 GB
24=67.664.6–70.43.07 GB82.773.334.0299 msA100$0local3.07 GB
25=67.564.5–70.39.11 GB82.775.031.5451 msA100$0local9.11 GB
26=66.763.7–69.519.08 GB80.177.522.2443 msAPI$0.08119.08 GB
26=66.263.2–69.03.01 GB80.170.028.1253 msA100$0local3.01 GB
2865.262.1–68.12.73 GB82.174.228.1128 msA100$0local2.73 GB
3164.061.0–67.09.50 GB85.381.724.4621 msAPI$0.0309.50 GB
32=63.560.5–66.54.45 GB69.271.731.559 msA100$0local4.45 GB
32=63.360.3–66.3API only66.776.726.2906 msAPI$0.087API only
34=62.559.5–65.59.34 GB83.375.024.1447 msAPI$0.0549.34 GB
34=61.458.4–64.45.47 GB55.873.329.0117 msA100$0local5.47 GB
3759.856.7–62.83.85 GB63.574.224.735 msA100$0local3.85 GB
40=56.753.6–59.74.35 GB59.672.528.132 msA100$0local4.35 GB
40=56.052.9–59.12.03 GB59.665.026.278 msA100$0local2.03 GB
40=55.252.1–58.30.89 GB60.970.026.270 msA100$0local0.89 GB
4252.549.4–55.64.64 GB57.160.025.3174 msA100$0local4.64 GB
43=51.748.6–54.82.01 GB51.366.725.9131 msA100$0local2.01 GB
43=50.447.2–53.51.52 GB52.662.529.3105 msA100$0local1.52 GB
43=49.946.8–53.11.73 GB51.955.824.458 msA100$0local1.73 GB
4448.345.2–51.41.73 GB51.353.331.5185 msA100$0local1.73 GB
4547.744.6–50.81.52 GB44.253.326.236 msA100$0local1.52 GB
4941.438.4–44.51.22 GB45.544.224.133 msA100$0local1.22 GB
50=27.925.2–30.81.84 GB41.739.221.9108 msA100$0local1.84 GB
50=24.922.3–27.70.30 GB30.137.527.234 msA100$0local0.30 GB
5122.520.0–25.20.84 GB34.015.024.741 msA100$0local0.84 GB
53=17.415.2–19.90.35 GB34.639.27.714 msA100$0local0.35 GB
53=16.013.9–18.40.85 GB32.725.08.344 msA100$0local0.85 GB
554.93.8–6.51.95 GB19.915.0X47 msA100$0local1.95 GB
Baselines
—Label overlapUntrained word-overlap rule21.630.126.724.4———

Score against size and cost

Clinical decisions by parameter count, on a log scale. The solid step follows the best score at or below each size. Models with no public weights have no size; the best of them is the dashed line. Hover a point to see it.

  • Phone builds (up to 4 GiB)
  • Larger, open weights
  • Best API only
0204060801000.025B0.05B0.1B0.25B0.5B1B2B4B8B16B32Bd1-ultra · API onlyDrex Maxelephant-1d3-miniMedDecider-4Bd3-nanod1-2Bdecider-2bGLiClass v4 EdgeJulia-1GLiClass Instruct Edgeparameters (log scale)