Performance Benchmarks

Every number on this page is produced by benchmarks, whose results are committed to the repository. python -m benchmarks.run reproduces them, and the docs build reads the committed CSVs rather than re-running anything.

Note

This page previously carried a star-rating table scoring the calibrators on “calibration error”, “granularity preservation”, “speed” and “robustness”. Those ratings had no provenance — no script produced them and no measurement backed them. They were removed, and this is what replaced them.

The design

Thirty seeds over ten methods and ten dataset/model cells. Within a cell the calibrator is the only thing that varies: the out-of-fold scores and the test scores are computed once and shared, so a difference between two calibrators cannot be resampling noise. The test split is touched exactly once — nothing is selected, tuned or inspected on it.

Calibrators fit on out-of-fold scores from cross_val_predict, because a model’s scores on its own training rows are already too good and a calibrator fitted there learns the wrong correction.

Library defaults only. Tuning calibre’s methods against an untuned isotonic baseline would settle the comparison by construction. One asymmetry is worth naming rather than hiding: SplineCalibrator, and the "auto" defaults of the relaxed and regularized calibrators, choose their own hyperparameters by internal cross-validation. That is a real advantage over a fixed competitor, and it is paid for in the fit time the benchmark also records.

Everything that could be tuned to flatter calibre — datasets, seeds, model and calibrator settings, the baseline, which metrics are primary — lives in benchmarks/config.py, so the choices are visible in one diff.

Results

The overconfident design: a model reporting 1.8 * z for true log-odds z. Lower Brier is better. “vs known truth” is available because the design is synthetic, and it is the strongest evidence on the page. “Wins” counts seeds where the method beat sklearn_isotonic on held-out Brier.

Method	Brier	dBrier	smECE	Distinct	vs known truth	Wins
sklearn_platt	0.1521	+0.0082	0.0251	1599	0.0064	26/30
sklearn_temperature	0.1522	+0.0082	0.0251	1599	0.0040	26/30
calibre_spline	0.1524	+0.0079	0.0259	1588	0.0169	28/30
calibre_regularized	0.1525	+0.0079	0.0264	1596	0.0181	24/30
calibre_centered	0.1527	+0.0076	0.0284	1514	0.0205	25/30
calibre_relaxed_pava	0.1530	+0.0073	0.0270	1356	0.0254	28/30
calibre_isotonic	0.1530	+0.0073	0.0270	49	0.0255	0/30
sklearn_isotonic	0.1530	+0.0073	0.0270	49	0.0255	baseline
calibre_nearly_isotonic	0.1531	+0.0072	0.0270	51	0.0262	4/30
uncalibrated	0.1604	–	0.0835	1594	0.0808	0/30

Three things to read off it.

The Brier gains over isotonic are small. The large win is the distinct-value column: around 1400–1600 values instead of 49, at a Brier difference in the fourth decimal. RelaxedPAVACalibrator is the cleanest case — it beats isotonic on 28 of 30 seeds by an average of 0.00001, which is to say it costs nothing, and keeps 28 times the resolution.

scikit-learn’s parametric methods win this design outright. Both score better than anything in calibre and land four times closer to the known truth. That is not an artefact: the distortion here is a pure temperature change, so a one-parameter model is exactly specified and a non-parametric one is paying for flexibility it does not need. This is a regime where calibre loses, and it is a real one.

smECE barely separates the methods, because it is a calibration measure and resolution is not miscalibration. No single number settles this comparison, which is why the page shows several and refuses to combine them.

What resolution actually looks like

One thin tick per distinct output value, one strip per method, drawn across the input range. The number of ticks is the number of distinct values.

The obvious objection is that the extra values might be noise. If they were, the methods keeping them would sit higher on the score axis:

They do not. The frontier is flat: two clusters more than a decade apart in resolution, at the same height.

Paired differences against the baseline

Seed variance dwarfs the effect being measured, so levels would invite reading noise as a result. Differencing within a seed removes the dataset draw, the model fit and the split, leaving only the calibrator. The interval resamples seeds, because the seed is the unit of replication.

An interval that spans zero is drawn spanning zero. Across all 90 method-cells, 45 beat sklearn_isotonic with an interval clear of zero.

Where calibre loses

Named rather than buried, because a benchmark that only reports wins is not a benchmark.

• ``overconfident`` — Platt and temperature scaling beat every calibre method, as above.

• ``breast_cancer/logreg`` — not calibrating at all beats isotonic by 0.0013 Brier, interval [0.0003, 0.0025], on 22 of 30 seeds. Logistic regression is already close to calibrated there and the test half is only about 228 rows, so pooling costs more than it buys.

• ``NearlyIsotonicCalibrator`` at its defaults is close to plain isotonic on these designs — 51 distinct values against 49. Its resolution frontier is dominated by centered isotonic regression, which reaches more distinct values at a better score, so no default was invented to hide this. See the class docstring.

nonmonotone was built expecting calibre to lose, since no monotone calibrator can express a non-monotone truth. It did not: RegularizedIsotonicCalibrator scores 0.2156 against Platt’s 0.2224, because the parametric methods cannot follow the dip either and give up more. That is what measuring is for.

What keeps it honest

Guards, not promises. Each of these fails loudly:

• calibre_isotonic must reproduce sklearn_isotonic to 1e-12 on every row. calibre’s wrapper is a thin layer over scikit-learn’s, so any divergence is a bug — and a benchmark that hid it would be reporting calibre’s advantage over its own baseline.

• aggregate.py refuses to summarise a cell missing any of its seeds, naming the offenders. A dataset that errored on half its seeds would otherwise be averaged over whatever survived.

• No composite score. Score and resolution stay on separate axes. Folding them into one number is where a thumb goes on the scale.

• Figures are drawn through calibre.plots, so a regression in the plotting layer breaks the benchmark build instead of quietly producing a wrong picture here.

netcal is optional and off by default, enabled with --include-netcal. It is not a hard dependency: the moment it lags a Python release, a required import would make the whole harness un-runnable — which silently stops the benchmark being re-run, the failure this design exists to prevent.