Calibration Report¶

One call that gathers the CORP decomposition, four calibration-error estimators, and prediction granularity. It summarizes independent held-out evaluation rows; it does not perform a hypothesis test or issue a calibration verdict. Nothing here is new: it is assembled from CORP Evaluation and Calibration Metrics.

calibre.calibration_report(y_true, y_pred, *, n_bins=15, include_brier_interval=False, interval_level=0.95, interval_n_resamples=1000, random_state=0, interval_method='bca')[source]¶

Summarize the calibration of one set of probabilities.

Gathers the CORP decomposition, four calibration-error estimators, and prediction granularity. It does not perform a hypothesis test or issue a calibrated/not-calibrated verdict.

Parameters:

y_true (ndarray) – Ground truth values (0 or 1).
y_pred (ndarray) – Predicted probabilities.
n_bins (int) – Bin count for the two fixed-bin estimators. The sweep chooses its own and smECE needs none.
include_brier_interval (bool) – Whether to bootstrap an interval for the mean Brier score. Calibration-error and CORP-component intervals are omitted because the ordinary row bootstrap is not generally valid for those non-smooth estimators.
interval_level (float) – Nominal coverage for the Brier interval.
interval_n_resamples (int) – Number of bootstrap resamples.
random_state (int | None) – Bootstrap random seed.
interval_method (str) – Method passed to bootstrap_ci(). Defaults to "bca".

Returns:

The summary. Print it, or read fields off it.

Return type:

CalibrationReport

Raises:

ValueError – If the evaluation data or n_bins are invalid.
TypeError – If include_brier_interval is not boolean.

Warning

Run this on independent, held-out predictions. On the data a calibrator was fitted to, any isotonic-family method reports MCB of exactly zero by construction – the calibrator and this diagnostic are the same PAV projection, and PAV is idempotent – no matter how badly the model generalizes. Use cross_val_calibrate() for out-of-fold probabilities.

Sweep calibration error assumes a non-decreasing population calibration curve. It can be near zero for strongly nonmonotone miscalibration; compare it with the other diagnostics rather than treating it as a general verdict.

Examples

>>> import numpy as np
>>> from calibre import calibration_report
>>> rng = np.random.default_rng(0)
>>> p = rng.uniform(0, 1, 2000)
>>> y = rng.binomial(1, p).astype(float)
>>> report = calibration_report(y, p)
>>> report.n_observations
2000

These are calibrated by construction, so miscalibration is small next to the discrimination the forecasts earn:

>>> bool(report.miscalibration < 0.1 * report.discrimination)
True

And the uncorrected estimator reports more error than the corrected one:

>>> bool(
...     report.plugin_calibration_error
...     >= report.debiased_calibration_error
... )
True

class calibre.CalibrationReport(n_observations, base_rate, mean_prediction, mean_calibration_error, brier_score, miscalibration, discrimination, uncertainty, smooth_calibration_error, smooth_calibration_bandwidth, debiased_calibration_error, plugin_calibration_error, sweep_calibration_error, sweep_n_bins, n_bins, n_unique_predictions, unique_prediction_ratio, intervals=<factory>)[source]¶

Held-out diagnostics for one set of probability forecasts.

Parameters:

n_observations (int)
base_rate (float)
mean_prediction (float)
mean_calibration_error (float)
brier_score (float)
miscalibration (float)
discrimination (float)
uncertainty (float)
smooth_calibration_error (float)
smooth_calibration_bandwidth (float)
debiased_calibration_error (float)
plugin_calibration_error (float)
sweep_calibration_error (float)
sweep_n_bins (int)
n_bins (int)
n_unique_predictions (int)
unique_prediction_ratio (float)
intervals (Mapping[str, Mapping[str, Any]])

n_observations¶

Number of observations.

Type:: int

base_rate¶

Observed event frequency.

Type:: float

mean_prediction¶

Mean forecast probability.

Type:: float

mean_calibration_error¶

Absolute gap between the mean forecast and base rate.

Type:: float

brier_score¶

Mean Brier score.

Type:: float

miscalibration¶

CORP miscalibration component (MCB).

Type:: float

discrimination¶

CORP discrimination component (DSC).

Type:: float

uncertainty¶

CORP uncertainty component (UNC).

Type:: float

smooth_calibration_error¶

Smooth calibration error (smECE).

Type:: float

smooth_calibration_bandwidth¶

Bandwidth selected for smECE.

Type:: float

debiased_calibration_error¶

Bias-corrected binned error at n_bins.

Type:: float

plugin_calibration_error¶

Uncorrected binned error on the same bins.

Type:: float

sweep_calibration_error¶

Monotonic sweep calibration error. Its interpretation assumes a non-decreasing population calibration curve.

Type:: float

sweep_n_bins¶

Bin count selected by the monotonic sweep.

Type:: int

n_bins¶

Bin count used for the debiased and plugin errors.

Type:: int

n_unique_predictions¶

Number of distinct forecast probabilities.

Type:: int

unique_prediction_ratio¶

n_unique_predictions / n. This describes prediction granularity, not calibration or statistical resolution.

Type:: float

intervals¶

Read-only bootstrap intervals, empty unless include_brier_interval=True was passed. Only brier_score is intervalled.

Type:: Mapping[str, Mapping[str, Any]]

to_dict()[source]¶

Return the report as a plain dictionary.

Returns:: Every field, suitable for a DataFrame row or JSON.
Return type:: dict

Confidence intervals¶

calibre.bootstrap_ci(metric, y_true, y_pred, *, level=0.95, n_resamples=1000, random_state=0, method='bca')[source]¶

Bootstrap an interval for a regular scalar evaluation metric.

Observation pairs are sampled with replacement and passed to metric. Interval construction delegates to scipy.stats.bootstrap() with paired resampling. BCa is the standard default; percentile and basic intervals are also available.

Parameters:

metric (Callable[[np.ndarray, np.ndarray], float]) – Callable taking (y_true, y_pred) and returning a finite scalar.
y_true (np.ndarray) – Binary outcomes in {0, 1}.
y_pred (np.ndarray) – Forecast probabilities in [0, 1].
level (float) – Nominal coverage in (0, 1). Defaults to 0.95.
n_resamples (int) – Number of bootstrap resamples. Defaults to 1000.
random_state (int | None) – Seed. Defaults to 0 so results are reproducible.
method (str) – "bca" (default), "basic" or "percentile".

Returns:

estimate (the metric on the observed data), lower,: upper, level, n_resamples, method and bias (the bootstrap mean minus the estimate).

Return type:

dict

Raises:

ValueError – If the data, bootstrap arguments, method, or a metric result is invalid.
TypeError – If metric is not callable.
RuntimeError – If the resampling distribution is degenerate or SciPy cannot define finite endpoints.

Warning

Rows must be independent held-out evaluation units. Resample clusters or time blocks instead when observations are dependent; this function does not implement those designs.

Ordinary bootstrap intervals need a regular statistic. Calibration-error metrics are non-smooth at perfect calibration, and percentile, basic and BCa intervals need not attain nominal coverage there. Use this function for regular quantities such as held-out mean proper scores, not as a test that calibration error equals zero.

Notes

BCa follows Efron (1987), “Better Bootstrap Confidence Intervals”. The resampling and interval endpoints are computed by SciPy rather than a package-local implementation.

Examples

>>> import numpy as np
>>> from calibre.metrics import brier_score
>>> rng = np.random.default_rng(0)
>>> p = rng.uniform(0, 1, 500)
>>> y = rng.binomial(1, p).astype(float)
>>> ci = bootstrap_ci(brier_score, y, p, n_resamples=200)
>>> bool(ci["lower"] <= ci["estimate"] <= ci["upper"])
True

Usage¶

import numpy as np

from calibre import calibration_report

rng = np.random.default_rng(0)
p = rng.uniform(0, 1, 3000)
y = rng.binomial(1, p).astype(float)
overconfident = np.clip(1.8 * (p - 0.5) + 0.5, 0, 1)

print(calibration_report(y, overconfident))

CalibrationReport  n=3,000  base rate 0.4933

  Brier            0.1849
    = MCB          0.0183   (recalibration recovers this)
    - DSC          0.0834   (earned by the forecasts)
    + UNC          0.2500   (irreducible)

  mean cal. error  0.0039   (mean forecast 0.4973)
  smECE            0.1173   (bandwidth 0.1182, chosen)
  debiased ECE     0.1240   (15 bins)
  plugin ECE       0.1267   (15 bins, uncorrected)
  sweep ECE        0.1222   (6 bins; assumes a monotone calibration curve)

  prediction granularity  1,657 of 3,000 values unique (55.2%)

Read MCB first: it is what recalibration would recover, and here it is a fifth of what the forecasts earn in DSC. The four error estimators agree on the magnitude but not the number, which is the point of showing all four.

Warning

Sweep ECE assumes a non-decreasing population calibration curve. It can be near zero for strongly nonmonotone miscalibration, so compare it with the other diagnostics rather than treating it as a general verdict.