Calibration Report¶

One call that gathers the CORP decomposition, three calibration-error estimators that disagree in instructive ways, the bias, and the resolution the forecasts retain. Nothing here is new: it is assembled from CORP Evaluation and Calibration Metrics.

calibre.calibration_report(y_true, y_pred, n_bins=15, ci=False, level=0.95, n_resamples=1000, random_state=0, ci_method='bc')[source]¶

Summarise the calibration of one set of probabilities.

Gathers the CORP decomposition, three calibration-error estimators that disagree in instructive ways, and the resolution the forecasts retain.

Parameters:

y_true (ndarray) – Ground truth values (0 or 1).
y_pred (ndarray) – Predicted probabilities.
n_bins (int) – Bin count for the two fixed-bin estimators. The sweep chooses its own and smECE needs none.
ci (bool) – Whether to bootstrap confidence intervals for brier, smece and debiased_ece. Off by default because it costs n_resamples recomputations of each. MCB and DSC are excluded on purpose: the naive bootstrap is inconsistent for functionals of an isotonic fit, and would report an interval that can sit above the estimate. Use consistency_bands() or confidence_bands() for those.
level (float) – Nominal coverage for those intervals.
n_resamples (int) – Bootstrap resamples.
random_state (int | None) – Seed.
ci_method (str) – Interval method, passed to bootstrap_ci(). Defaults to "bc", which is bias-corrected; the plain percentile interval under-covers badly here, for reasons that function documents.

Returns:

The summary. Print it, or read fields off it.

Return type:

CalibrationReport

Raises:

ValueError – If the arrays disagree in length or n_bins is below 1.

Warning

Run this on held-out predictions. On the data a calibrator was fitted to, any isotonic-family method reports MCB of exactly zero by construction – the calibrator and this diagnostic are the same PAV projection, and PAV is idempotent – no matter how badly the model generalises. Use cross_val_calibrate() for out-of-fold probabilities.

Examples

>>> import numpy as np
>>> from calibre import calibration_report
>>> rng = np.random.default_rng(0)
>>> p = rng.uniform(0, 1, 2000)
>>> y = rng.binomial(1, p).astype(float)
>>> report = calibration_report(y, p)
>>> report.n
2000

These are calibrated by construction, so miscalibration is small next to the discrimination the forecasts earn:

>>> bool(report.mcb < 0.1 * report.dsc)
True

And the uncorrected estimator reports more error than the corrected one:

>>> bool(report.plugin_ece >= report.debiased_ece)
True

class calibre.CalibrationReport(n, base_rate, mean_prediction, bias, brier, mcb, dsc, unc, smece, smece_sigma, debiased_ece, plugin_ece, sweep_ece, sweep_bins, n_bins, n_distinct, distinct_ratio, intervals=<factory>)[source]¶

Everything worth knowing about one set of probabilities.

Parameters:

n (int)
base_rate (float)
mean_prediction (float)
bias (float)
brier (float)
mcb (float)
dsc (float)
unc (float)
smece (float)
smece_sigma (float)
debiased_ece (float)
plugin_ece (float)
sweep_ece (float)
sweep_bins (int)
n_bins (int)
n_distinct (int)
distinct_ratio (float)
intervals (dict[str, dict[str, float]])

n¶

Number of observations.

Type:: int

base_rate¶

Observed event frequency.

Type:: float

mean_prediction¶

Mean forecast. Compare with base_rate: the gap is bias.

Type:: float

bias¶

Calibration in the large, |mean_prediction - base_rate|.

Type:: float

brier¶

Brier score. The proper scoring rule to optimise.

Type:: float

mcb¶

Miscalibration, from the CORP decomposition. What recalibration recovers.

Type:: float

dsc¶

Discrimination. What the forecasts buy over predicting the base rate.

Type:: float

unc¶

Uncertainty. The difficulty of the problem; no forecaster changes it.

Type:: float

smece¶

Smooth calibration error, with no bin count and no bandwidth to choose.

Type:: float

smece_sigma¶

The bandwidth smECE selected.

Type:: float

debiased_ece¶

Bias-corrected binned error at n_bins.

Type:: float

plugin_ece¶

Uncorrected binned error at n_bins, on the same bins. The gap between this and debiased_ece is the bias you would have reported.

Type:: float

sweep_ece¶

Binned error at the bin count the monotone sweep selected.

Type:: float

sweep_bins¶

That bin count.

Type:: int

n_bins¶

The bin count used for debiased_ece and plugin_ece.

Type:: int

n_distinct¶

Distinct forecast values. Isotonic regression collapses this; the point of most of this package is not to.

Type:: int

distinct_ratio¶

n_distinct / n.

Type:: float

intervals¶

Bootstrap confidence intervals, empty unless ci=True was passed. Each holds lower, upper, bias and degenerate.

Type:: dict[str, dict[str, float]]

to_dict()[source]¶

Return the report as a plain dictionary.

Returns:: Every field, suitable for a DataFrame row or JSON.
Return type:: dict

Confidence intervals¶

calibre.bootstrap_ci(metric, y_true, y_pred, level=0.95, n_resamples=1000, random_state=0, method='bc')[source]¶

Put a confidence interval on any calibration metric.

Resamples observations with replacement, recomputes the metric on each resample, and turns the resulting draws into an interval. Works with any callable of (y_true, y_pred), so it covers the estimators in calibre.metrics and anything you write yourself.

Calibration metrics are usually reported as bare numbers, which invites reading a difference of 0.002 between two models as real. On a few thousand observations the interval is often wider than that.

Parameters:

metric (Callable[[np.ndarray, np.ndarray], float]) – Callable taking (y_true, y_pred) and returning a float.
y_true (np.ndarray) – Ground truth values.
y_pred (np.ndarray) – Predicted probabilities.
level (float) – Nominal coverage in (0, 1). Defaults to 0.95.
n_resamples (int) – Number of bootstrap resamples. Defaults to 1000.
random_state (int | None) – Seed. Defaults to 0 so results are reproducible.
method (str) –
How to build the interval from the draws:
- "bc" (default) – bias-corrected percentile. Shifts the quantiles by how far the draws sit above the estimate. Costs nothing extra and, being a percentile method, can never return a bound outside the range of the statistic.
- "bca" – adds Efron’s acceleration for skewness. Costs n extra evaluations of metric for the jackknife, so it is offered rather than defaulted.
- "basic" – the reverse-percentile interval 2*theta - quantiles. Corrects the bias but routinely returns a negative lower bound for a non-negative statistic.
- "percentile" – the raw quantiles. Documented, and wrong here; see below.

Returns:

estimate (the metric on the observed data), lower,: upper, level, n_resamples, method, bias (the bootstrap mean minus the estimate, so the distortion is visible rather than hidden), and degenerate (whether the interval collapsed to a point).

Return type:

dict

Raises:

ValueError – If level is outside (0, 1), n_resamples is below 2, the arrays disagree in length, or method is unknown.

Notes

Why the default is not the percentile interval.

The bootstrap resamples from the empirical measure, so E[F*] = F. What happens to an estimator theta = g(F) is then decided entirely by the shape of g:

g linear in F – a plain mean, such as the Brier score – gives E[g(F*)] = g(F) exactly, by Jensen with equality.
g convex in F gives E[g(F*)] >= g(F), strictly. Every calibration error is convex: each bin’s contribution is an absolute value of a linear functional of F, and norms of those stay convex.

The size of the gap is set by curvature at F, which is unbounded at the kink ||delta|| = 0 and negligible far from it. So the distortion is worst exactly when the model is well calibrated – the case the user most wants an honest answer for. Measured (see experiments/bootstrap_bias/investigate.py), bootstrap mean over observed:

statistic	calibrated data	miscalibrated data
Brier score (linear)	1.00x	1.00x
plugin ECE (convex)	1.42x	1.01x
smECE	1.33x	1.04x
`MCB`	1.52x	1.09x

The plugin figure of 1.42 is the predicted sqrt(2): the observed value is ||delta|| for sampling noise delta, while the resample gives ||delta + eps|| with eps of comparable variance, doubling the variance inside the norm. The effect does not shrink with sample size – measured at 1.43, 1.44, 1.44, 1.42 for n of 250, 1000, 4000 and 16000 – because both terms scale as 1/sqrt(n). More data will not save you; a better interval will.

Coverage of a true calibration error of exactly zero, at a nominal 95%, using debiased_calibration_error() (whose estimand really is the true error):

method	coverage
`percentile`	77%
`basic`	98%
`bc`	95%

Hence the default, which is also 3.6 times tighter: mean width 0.017 against the percentile interval’s 0.063. basic over-covers at 98% and returns negative lower bounds for a non-negative quantity.

One caveat on the default. bc reads the bias off how many draws fall below the estimate, so it degenerates when the statistic has an atom at the estimate. debiased_calibration_error() floors at zero and returns exactly zero on 59% of well-calibrated samples, and in 30% of those the interval collapses to [0, 0]; the returned degenerate flag says when that happened. No such collapse occurs for the plugin error, smECE, the Brier score or MCB, none of which are censored. If you want an interval that stays non-degenerate near zero, measure with smooth_calibration_error(), which never floors.

A separate point about plugin estimators. The uncorrected binned error is biased, so its estimand is E[plugin] > 0 rather than the true error. An interval for it correctly excludes zero, and no choice of interval method changes that. If you want an interval that can cover zero, measure with an estimator that targets zero – debiased_calibration_error() or smooth_calibration_error().

``MCB`` and ``DSC`` carry an extra problem. They are functionals of an isotonic fit, and a resample leaves only about 63% of rows distinct (measured: 0.630 against a theoretical 0.632), so PAV overfits the duplicates. Their inflation tracks effective sample size rather than convexity alone: subsampling without replacement gives MCB of 0.0155, 0.0088 and 0.0056 at m of 200, 500 and 1000 against 0.0036 observed at n = 2000. Prefer consistency_bands() or confidence_bands() there, which resample outcomes rather than rows.

Examples

>>> import numpy as np
>>> from calibre.metrics import debiased_calibration_error
>>> rng = np.random.default_rng(0)
>>> p = rng.uniform(0, 1, 2000)
>>> y = rng.binomial(1, p).astype(float)
>>> ci = bootstrap_ci(debiased_calibration_error, y, p, n_resamples=200)

The data are calibrated by construction, so the interval should reach zero:

>>> bool(ci["lower"] <= 0.001)
True

The reported bias is how far the resampling pushed the statistic up:

>>> bool(ci["bias"] > 0.0)
True

A percentile interval on the same draws sits higher:

>>> percentile = bootstrap_ci(
...     debiased_calibration_error, y, p, n_resamples=200, method="percentile"
... )
>>> bool(percentile["lower"] >= ci["lower"])
True

Usage¶

import numpy as np

from calibre import calibration_report

rng = np.random.default_rng(0)
p = rng.uniform(0, 1, 3000)
y = rng.binomial(1, p).astype(float)
overconfident = np.clip(1.8 * (p - 0.5) + 0.5, 0, 1)

print(calibration_report(y, overconfident))

CalibrationReport  n=3,000  base rate 0.4933

  Brier            0.1849
    = MCB          0.0183   (recalibration recovers this)
    - DSC          0.0834   (earned by the forecasts)
    + UNC          0.2500   (irreducible)

  bias             0.0039   (mean forecast 0.4973)
  smECE            0.1173   (bandwidth 0.1182, chosen)
  debiased ECE     0.1240   (15 bins)
  plugin ECE       0.1267   (15 bins, uncorrected)
  sweep ECE        0.1180   (6 bins, chosen)

  distinct values  1,657 of 3,000 (55.2%)

Read MCB first: it is what recalibration would recover, and here it is a fifth of what the forecasts earn in DSC. The three error estimators agree on the magnitude but not the number, which is the point of showing all three.

Warning