Calibration Metrics¶

This module provides metrics for evaluating calibration quality.

For the CORP reliability diagram and the MCB/DSC/UNC score decomposition — which need no bin count and cannot be tuned in your favor — see CORP Evaluation.

Bias-aware calibration error¶

The plugin binned estimator is biased upward: part of each bin’s gap is sampling noise in the label mean rather than miscalibration, and the bias grows with the bin count. These two estimators correct for that, and are the ones to reach for when the number will be reported.

Smooth Calibration Error (smECE)¶

calibre.smooth_calibration_error(y_true: ndarray, y_pred: ndarray, *, bandwidth: float | None = None, return_bandwidth: Literal[False] = False) → float[source]¶

calibre.smooth_calibration_error(y_true: ndarray, y_pred: ndarray, *, bandwidth: float | None = None, return_bandwidth: Literal[True]) → tuple[float, float]

Calculate the smooth calibration error (smECE).

Binned calibration error has no consistent limit: refine the bins and the estimate keeps climbing on data that is perfectly calibrated. smECE replaces the bins with a Gaussian kernel and, by default, chooses its own bandwidth. The default estimator therefore has no bin-count or bandwidth choice for the caller to tune.

\[\mathrm{smECE}_\sigma(f, y) = \frac{\int \left| (K_\sigma \star \nu)(t) \right| \, dt} {\int (K_\sigma \star \rho)(t) \, dt}, \qquad \nu = \sum_i (f_i - y_i)\, \delta_{f_i}, \quad \rho = \sum_i \delta_{f_i}\]

The bandwidth is the fixed point \(\sigma = \mathrm{smECE}_\sigma\), found by bisection. Below that width the kernel is resolving noise; above it, it is smoothing away real miscalibration.

This is a consistent calibration measure in the sense of Blasiok, Gopalan, Hu and Nakkiran (2023): it is bounded above and below by polynomial functions of the true distance to the nearest perfectly calibrated predictor. Binned ECE is not, which is why it can report a large error for a predictor that is almost calibrated and a small one for a predictor that is not.

Evaluate smECE on held-out predictions. The automatic bandwidth is selected from that evaluation sample as part of the estimator; it is not a calibrator fit and does not require a separate tuning split. To assess recalibration, pair smECE with held-out proper scores and resolution or discrimination diagnostics rather than selecting a model from calibration error alone.

Parameters:

y_true (ndarray) – Binary outcomes in {0, 1}.
y_pred (ndarray) – Predicted probabilities in [0, 1].
bandwidth (float | None) – Kernel bandwidth, at least 0.001. When None, the fixed point above is used, which is the recommended behavior and what makes the estimator hyperparameter-free. A fixed bandwidth is useful when its scale has a substantive interpretation, with weaker theoretical guarantees than the automatic choice.
return_bandwidth (bool) – Also return the bandwidth used. Worth reporting: it is an interpretable scale, roughly the resolution at which miscalibration is detectable.

Returns:

The smooth calibration error, and: the bandwidth when return_bandwidth.

Return type:

float or tuple of (float, float)

Raises:

ValueError – If inputs are empty, multidimensional, non-finite, inconsistent in shape, outside their documented domains, or non-numeric; or if bandwidth is not a finite number at least 0.001.
TypeError – If return_bandwidth is not boolean.

See also

debiased_calibration_error() – bias-corrected, but still needs a bin count.
calibre.evaluation.score_decomposition() – avoids binning by using isotonic regression, and decomposes the score rather than summarizing the error.

References

Błasiok & Nakkiran (2024), “Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing”, ICLR. Błasiok, Gopalan, Hu & Nakkiran (2023), “A Unifying Theory of Distance from Calibration”, STOC. The numerical discretization and bandwidth search follow Błasiok and Nakkiran’s Apple relplot reference implementation: https://github.com/apple/ml-calibration. See THIRD_PARTY_NOTICES.md.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> p = rng.uniform(0, 1, 2000)
>>> y = rng.binomial(1, p).astype(float)

Calibrated by construction, so the error is near zero:

>>> bool(smooth_calibration_error(y, p) < 0.03)
True

An overconfident predictor is caught:

>>> squashed = np.clip(2.0 * (p - 0.5) + 0.5, 0, 1)
>>> bool(smooth_calibration_error(y, squashed) > 0.05)
True

Unlike a binned estimator there is no bin count to justify; the bandwidth is chosen by the data:

>>> error, width = smooth_calibration_error(y, p, return_bandwidth=True)
>>> bool(0.0 < width <= 1.0)
True

Unlike everything below it, the default smECE has no bin count and selects its own bandwidth. It is a consistent measure of distance from calibration and is the one to reach for when the number will be quoted without qualification.

Debiased Calibration Error¶

calibre.debiased_calibration_error(y_true, y_pred, *, n_bins=15, squared=False)[source]¶

Calculate the debiased \(\ell_2\) calibration error.

The plugin binned estimator mistakes sampling noise in each bin’s outcome rate for miscalibration. This estimator subtracts that Bernoulli-noise term from the squared gap before aggregating bins. It removes this finite-sample component of bias; it does not remove the approximation error caused by representing a continuous calibration curve with finitely many bins.

\[\widehat{\mathrm{CE}}^2 = \sum_k \frac{n_k}{n} \left[ (\bar{f}_k - \bar{y}_k)^2 - \frac{\bar{y}_k (1 - \bar{y}_k)}{n_k - 1} \right]\]

Parameters:

y_true (ndarray) – Ground truth values (0 or 1).
y_pred (ndarray) – Predicted probabilities.
n_bins (int) – Requested number of equal-mass bins, capped at the sample size. Tied predictions are never split, so fewer bins may be used. The conventional default of 15 follows the experiments in Roelofs et al.; it is not universally optimal.
squared (bool) – Return the estimate of the squared error instead, without the square root or the floor at zero. This is the quantity the correction actually makes unbiased, and it may legitimately come out negative – see Notes.

Returns:

Debiased calibration error. Floored at zero: the correction can: drive the sum negative on well-calibrated data, which is evidence of no detectable miscalibration rather than of negative error. With squared=True the unfloored sum is returned instead.

Return type:

float

Raises:

ValueError – If inputs are empty, multidimensional, non-finite, inconsistent in length, or outside their documented domains; if n_bins is not a positive integer; or if squared is not boolean.

Notes

This is the \(\ell_2\) error, so it is not comparable in magnitude to expected_calibration_error(), which is \(\ell_1\).

The correction operates on the squared scale, not the root scale. On well-calibrated samples, the corrected squared estimate fluctuates around zero and can legitimately be negative. The default floor and square root produce a reportable non-negative error but reintroduce upward bias near zero. For averaging or model comparison, use squared=True and aggregate on that scale.

sample_weight is deliberately unsupported. The cited correction is derived for individual independent observations; treating arbitrary weights as replication or importance weights would change that statistical claim.

References

Bröcker (2012), “Estimating Reliability and Resolution of Probability Forecasts through Decomposition of the Empirical Score”; Ferro & Fricker (2012), “A Bias-Corrected Decomposition of the Brier Score”. The formula follows Kumar, Liang & Ma’s unbiased_square_ce and unbiased_l2_ce reference implementation for “Verified Uncertainty Calibration” (NeurIPS 2019). Equal-mass binning follows the empirical recommendation of Roelofs, Cain, Shlens & Mozer (2022), “Mitigating Bias in Calibration Error Estimation”.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> p = rng.uniform(0, 1, 4000)
>>> y = rng.binomial(1, p).astype(float)

These predictions are calibrated, so the matched \(\ell_2\) plugin estimator reports finite-sample noise that the correction largely removes:

>>> plugin = plugin_calibration_error(y, p, n_bins=15, norm=2)
>>> debiased = debiased_calibration_error(y, p, n_bins=15)
>>> bool(debiased < plugin)
True

See also

sweep_calibration_error : Chooses the bin count instead of fixing it. calibre.evaluation.score_decomposition : Avoids binning altogether.

Sweep Calibration Error¶

calibre.sweep_calibration_error(y_true: ndarray, y_pred: ndarray, *, norm: int = 2, return_n_bins: Literal[False] = False) → float[source]¶

calibre.sweep_calibration_error(y_true: ndarray, y_pred: ndarray, *, norm: int = 2, return_n_bins: Literal[True]) → tuple[float, int]

Calculate the monotonic sweep calibration error (ECE_sweep).

Fixing the bin count is the weak point of binned calibration error: too few bins hide miscalibration, too many measure noise, and the best choice depends on the sample size and the score distribution. This estimator chooses instead.

The method assumes a non-decreasing population calibration curve, as is often expected for likelihood-trained models. Bins are added while the observed bin heights stay monotone, and the sweep stops at the largest bin count for which they do. Under that assumption, non-monotonicity signals that the bins have become fine enough to read sampling noise. For a genuinely nonmonotone calibration curve, the method can stop too early and understate error.

Parameters:

y_true (ndarray) – Ground truth values (0 or 1).
y_pred (ndarray) – Predicted probabilities.
norm (int) – Positive integer norm. The default is 2, matching the paper’s experiments and the authors’ reference implementation. Set to 1 for the weighted mean absolute gap.
return_n_bins (bool) – Also return the bin count the sweep settled on. That number is half of what the estimator has to say – it is the sweep’s answer to “how fine can these data support?” – and reporting only the error hides it.

Returns:

Binned calibration error at the: selected bin count, and that bin count when return_n_bins is True. The count is the number of bins actually occupied, which ties can hold below the number the sweep reached.

Return type:

float or tuple of (float, int)

Raises:

ValueError – If inputs are empty, multidimensional, non-finite, inconsistent in length, or outside their documented domains; if norm is not a positive integer; or if return_n_bins is not boolean.

Notes

This function evaluates held-out predictions; it does not fit a calibrator. The selected bin count and error are properties of this evaluation sample.

sample_weight is deliberately unsupported. The cited algorithm defines equal mass as equal numbers of independent observations; arbitrary weights require a separate definition of both bin mass and monotonicity.

References

Roelofs, Cain, Shlens & Mozer (2022), “Mitigating Bias in Calibration Error Estimation”, AISTATS. The implementation follows Algorithm 1 and the authors’ Google Research em_monotonic_sweep reference, with the additional guarantee that tied predictions are never split.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> p = rng.uniform(0, 1, 4000)
>>> y = rng.binomial(1, p).astype(float)
>>> float(sweep_calibration_error(y, p)) < 0.05
True

See also

debiased_calibration_error : Fixes the bin count and corrects the bias. calibre.evaluation.score_decomposition : Lets isotonic regression bin.

Plugin Calibration Error¶

calibre.plugin_calibration_error(y_true, y_pred, *, n_bins=15, norm=2, sample_weight=None)[source]¶

Calculate the uncorrected \(\ell_p\) binned calibration error.

\[\widehat{\mathrm{CE}}_p = \left[ \sum_k \frac{W_k}{W} \left| \bar{f}_k - \bar{y}_k \right|^p \right]^{1/p}\]

Here \(W_k\) is the total evaluation weight in bin \(k\); with unit weights, \(W_k/W = n_k/n\).

This is the plain plugin estimator: the quantity debiased_calibration_error() corrects and sweep_calibration_error() chooses a bin count for. It exists so those three can be compared on equal terms.

That comparison is otherwise a trap. expected_calibration_error() is \(\ell_1\) on uniform-width bins, while debiased_calibration_error() and sweep_calibration_error() default to \(\ell_2\) on equal-mass bins. This function takes both the norm and bin count as arguments and uses the same equal-mass, tie-safe binning as the bias-aware estimators, so callers can compare matched quantities.

Parameters:

y_true (ndarray) – Ground truth values (0 or 1).
y_pred (ndarray) – Predicted probabilities.
n_bins (int) – Number of equal-mass bins. Fewer are used when ties prevent it.
norm (int) – Norm. 1 gives the familiar weighted mean absolute gap; 2 matches debiased_calibration_error().
sample_weight (ndarray | None) – Non-negative evaluation weights. Zero-weight rows are excluded before validation and binning.

Returns:

The uncorrected calibration error. Biased upward, and: increasingly so as n_bins grows.

Return type:

float

Raises:

ValueError – If inputs are empty, multidimensional, non-finite, inconsistent in length, outside their documented domains, or carry invalid weights; or if n_bins or norm is not a positive integer.

References

Kumar, Liang & Ma (2019), “Verified Uncertainty Calibration”, NeurIPS 32. The formula follows the authors’ plugin_ce reference implementation; the 15-bin default follows their lower_bound_scaling_ce interface. Roelofs, Cain, Shlens & Mozer (2022), “Mitigating Bias in Calibration Error Estimation”, AISTATS, PMLR 151:4036–4054.

See also

debiased_calibration_error : The same quantity at norm=2, bias-corrected. sweep_calibration_error : Chooses n_bins rather than fixing it.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> p_hat = rng.uniform(0, 1, 4000)
>>> y = rng.binomial(1, p_hat).astype(float)

These are calibrated by construction, so the true error is zero and whatever the plugin reports is bias – which grows with the bin count:

>>> coarse = plugin_calibration_error(y, p_hat, n_bins=5)
>>> fine = plugin_calibration_error(y, p_hat, n_bins=50)
>>> bool(fine > coarse)
True

Debiasing removes it:

>>> bool(debiased_calibration_error(y, p_hat, n_bins=50) < fine)
True

Calibration Error Metrics¶

Mean Calibration Error¶

calibre.mean_calibration_error(y_true, y_pred, *, sample_weight=None)[source]¶

Calculate the absolute calibration-in-the-large error.

\[\left| \frac{\sum_i w_i \hat p_i}{\sum_i w_i} - \frac{\sum_i w_i y_i}{\sum_i w_i} \right|\]

This measures whether predictions are right on average. It can be zero when positive and negative errors cancel, even for completely reversed predictions, so use it with a proper score and a calibration curve rather than as a complete calibration assessment.

Parameters:

y_true (ndarray) – Ground truth values (0 or 1 for binary classification).
y_pred (ndarray) – Predicted probabilities in [0, 1].
sample_weight (ndarray | None) – Non-negative evaluation weights. Zero-weight observations are ignored. A common rescaling does not change the result.

Returns:

Absolute difference between the mean prediction and the base rate.

Return type:

float

References

Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016). A calibration hierarchy for risk models was defined: from utopia to empirical data. Journal of Clinical Epidemiology, 74, 167–176. https://doi.org/10.1016/j.jclinepi.2015.12.005

Examples

>>> import numpy as np
>>> y_true = np.array([0, 1, 1, 0, 1])
>>> y_pred = np.array([0.2, 0.7, 0.8, 0.4, 0.6])
>>> round(mean_calibration_error(y_true, y_pred), 4)   # mean 0.54 vs base 0.6
0.06

A perfectly calibrated predictor scores zero, however unsharp it is:

>>> y = np.array([0, 0, 1, 1])
>>> mean_calibration_error(y, np.full(4, 0.5))
0.0

Root Mean Squared Calibration Error¶

calibre.root_mean_squared_calibration_error(y_true: ndarray, y_pred: ndarray, *, n_bins: int = 10, strategy: Literal['uniform', 'quantile'] = 'uniform', sample_weight: ndarray | None = None, return_details: Literal[False] = False) → float[source]¶

Calculate the mass-weighted root mean squared calibration error.

\[\operatorname{RMSCE} = \left[\sum_k \frac{w_k}{\sum_j w_j} (\bar p_k - \bar y_k)^2\right]^{1/2}\]

Unlike an unweighted average across occupied bins, this definition gives a bin influence proportional to its evaluation-sample mass. Uniform bins divide the probability domain [0, 1] into equal-width intervals. Quantile bins are approximately equal in weighted mass and never split tied predictions.

This is a plugin estimator: its value depends on the binning choice and is biased upward by finite-sample variation in bin outcome rates. Use the same held-out evaluation observations for comparisons between prediction systems; do not evaluate a calibrator on the observations used to fit it.

Parameters:

y_true (ndarray) – Binary outcomes in {0, 1}.
y_pred (ndarray) – Predicted probabilities in [0, 1].
n_bins (int) – Requested number of bins, capped at the number of positive-weight observations. Quantile binning can use fewer bins when ties prevent a split.
strategy (Literal['uniform', 'quantile']) – "uniform" for fixed-width probability bins or "quantile" for weighted equal-mass, tie-preserving bins.
sample_weight (ndarray | None) – Non-negative evaluation weights. Zero-weight observations are ignored. A common rescaling does not change the result.
return_details (bool) – Return the score and occupied-bin summaries when true.

Returns:

RMSCE when return_details is false. dict: RMSCE, counts, weights, prediction ranges, and weighted bin means when return_details is true.

Return type:

float

Raises:

ValueError – If inputs are empty, multidimensional, non-finite, inconsistent in length, outside their documented domains, or carry invalid weights; if n_bins is not a positive integer; or if strategy is unknown.
TypeError – If return_details is not boolean.

References

Guo, Pleiss, Sun & Weinberger (2017), “On Calibration of Modern Neural Networks”, ICML, PMLR 70:1321–1330. Kumar, Liang & Ma (2019), “Verified Uncertainty Calibration”, NeurIPS 32. The formula matches the norm="l2" calibration error implemented by TorchMetrics.

Examples

>>> import numpy as np
>>> y_true = np.array([0, 0, 1, 1])
>>> y_pred = np.array([0.25, 0.25, 0.75, 0.75])
>>> root_mean_squared_calibration_error(y_true, y_pred, n_bins=2)
0.25

Expected Calibration Error¶

calibre.expected_calibration_error(y_true, y_pred, *, n_bins=10, sample_weight=None)[source]¶

Calculate uniform-bin expected calibration error (ECE).

\[\operatorname{ECE} = \sum_k \frac{w_k}{\sum_j w_j} \left|\bar p_k - \bar y_k\right|\]

The probability interval [0, 1] is divided into equal-width bins, and each occupied bin’s absolute calibration gap is weighted by its evaluation mass. This is the conventional plugin ECE estimator, matching the norm="l1" definition in TorchMetrics.

ECE is sensitive to the bin count and is not a consistent measure of all calibration departures: overprediction and underprediction inside one bin can cancel exactly. Finite-sample variation in bin outcome rates also biases the score upward. Use smooth_calibration_error(), debiased_calibration_error(), or sweep_calibration_error() when a less arbitrary headline estimate is required.

Parameters:

y_true (ndarray) – Binary outcomes in {0, 1}.
y_pred (ndarray) – Predicted probabilities in [0, 1].
n_bins (int) – Requested number of equal-width bins, capped at the number of positive-weight observations.
sample_weight (ndarray | None) – Non-negative evaluation weights. Zero-weight observations are ignored. A common rescaling does not change the result.

Returns:

Sample-mass-weighted absolute calibration gap.

Return type:

float

Raises:

ValueError – If inputs are empty, multidimensional, non-finite, inconsistent in length, outside their documented domains, or carry invalid weights; or if n_bins is not a positive integer.

References

Naeini, Cooper & Hauskrecht (2015), “Obtaining Well Calibrated Probabilities Using Bayesian Binning”, AAAI 29. Guo, Pleiss, Sun & Weinberger (2017), “On Calibration of Modern Neural Networks”, ICML, PMLR 70:1321–1330. Roelofs, Cain, Shlens & Mozer (2022), “Mitigating Bias in Calibration Error Estimation”, AISTATS, PMLR 151:4036–4054.

Examples

>>> import numpy as np
>>> y_true = np.array([0, 0, 1, 1, 1])
>>> y_pred = np.array([0.25, 0.25, 0.55, 0.75, 0.75])
>>> round(expected_calibration_error(y_true, y_pred, n_bins=2), 2)
0.29

Maximum Calibration Error¶

calibre.maximum_calibration_error(y_true, y_pred, *, n_bins=10, sample_weight=None)[source]¶

Calculate uniform-bin maximum calibration error (MCE).

\[\operatorname{MCE} = \max_k \left|\bar p_k - \bar y_k\right|\]

The probability interval [0, 1] is divided into equal-width bins. MCE is the largest absolute gap between the weighted mean prediction and weighted event rate among the occupied bins. This is the conventional infinity-norm plugin estimator, matching the norm="max" definition in TorchMetrics.

MCE gives the same influence to every occupied bin regardless of its evaluation mass: one low-mass bin can determine the entire score. It is also sensitive to the bin count, can hide equal and opposite errors within a bin, and selects the largest finite-sample fluctuation. Treat it as a worst-bin diagnostic rather than a standalone estimate of population calibration. Use held-out observations and inspect the bin counts in a reliability diagram.

Parameters:

y_true (ndarray) – Binary outcomes in {0, 1}.
y_pred (ndarray) – Predicted probabilities in [0, 1].
n_bins (int) – Requested number of equal-width bins, capped at the number of positive-weight observations.
sample_weight (ndarray | None) – Non-negative evaluation weights used to calculate each bin’s means. Zero-weight observations are ignored. A common rescaling does not change the result.

Returns:

Largest occupied-bin absolute calibration gap.

Return type:

float

Raises:

ValueError – If inputs are empty, multidimensional, non-finite, inconsistent in length, outside their documented domains, or carry invalid weights; or if n_bins is not a positive integer.

References

Examples

>>> import numpy as np
>>> y_true = np.array([0, 0, 1, 1, 1])
>>> y_pred = np.array([0.25, 0.25, 0.55, 0.75, 0.75])
>>> round(maximum_calibration_error(y_true, y_pred, n_bins=2), 4)
0.3167

Note

These estimators are not interchangeable. By default, expected_calibration_error() is \(\ell_1\), while debiased_calibration_error() and sweep_calibration_error() are \(\ell_2\). expected_calibration_error() uses uniform-width bins, while the two bias-aware estimators use equal-mass bins. Compare like with like by setting the same norm where the APIs permit it.

Scoring Metrics¶

Brier Score¶

calibre.brier_score(y_true, y_pred, *, sample_weight=None)[source]¶

Calculate the binary Brier score.

\[\operatorname{BS} = \frac{\sum_i w_i (p_i-y_i)^2}{\sum_i w_i}\]

The Brier score is a strictly proper scoring rule: in expectation, reporting the true event probability uniquely minimises it. Lower is better, with zero for perfect binary probability forecasts and one for forecasts that are wrong with certainty.

Brier score measures overall probabilistic performance, not calibration alone. Its Murphy decomposition separates miscalibration, discrimination, and outcome uncertainty. Compare systems on the same held-out observations; do not evaluate a calibrator on the observations used to fit it.

Parameters:

y_true (ndarray) – Binary outcomes in {0, 1}.
y_pred (ndarray) – Predicted probabilities in [0, 1].
sample_weight (ndarray | None) – Non-negative evaluation weights. Zero-weight observations are ignored. A common rescaling does not change the result.

Returns:

Weighted mean squared probability error.

Return type:

float

References

Brier (1950), “Verification of Forecasts Expressed in Terms of Probability”, Monthly Weather Review 78:1–3. Murphy (1973), “A New Vector Partition of the Probability Score”, Journal of Applied Meteorology 12:595–600. The calculation delegates to sklearn.metrics.brier_score_loss().

Examples

>>> import numpy as np
>>> y_true = np.array([0, 1, 1, 0, 1])
>>> y_pred = np.array([0.2, 0.7, 0.8, 0.4, 0.6])
>>> brier_score(y_true, y_pred)
0.098

Calibration Curve¶

calibre.calibration_curve(y_true, y_pred, *, n_bins=10, strategy='uniform', sample_weight=None)[source]¶

Compute occupied-bin summaries for a binary reliability diagram.

Uniform bins divide the probability domain [0, 1] into equal-width intervals. Quantile bins have approximately equal evaluation mass and never split tied predictions. Empty bins are omitted: returning them as zero-valued points would make missing data look like observed perfect predictions.

The bin means use sample_weight when supplied. bin_counts remains the number of positive-weight observations in each occupied bin, so it describes empirical support rather than weighted mass. Use held-out evaluation data; computing this curve on the observations used to fit a calibrator gives an optimistic picture of its performance.

Parameters:

y_true (ndarray) – Binary outcomes in {0, 1}.
y_pred (ndarray) – Predicted probabilities in [0, 1].
n_bins (int) – Requested number of bins, capped at the number of positive-weight observations. Quantile binning can use fewer bins when ties prevent a split.
strategy (Literal['uniform', 'quantile']) – "uniform" for fixed-width probability bins or "quantile" for weighted equal-mass, tie-preserving bins.
sample_weight (ndarray | None) – Non-negative evaluation weights. Zero-weight observations are ignored. A common rescaling changes neither the bin assignments nor the bin means, including with quantile binning.

Returns:

Weighted event rate in each occupied bin. prob_pred: Weighted mean predicted probability in each occupied bin. bin_counts: Number of positive-weight observations in each occupied bin.

Return type:

prob_true

Raises:

ValueError – If inputs are empty, multidimensional, non-finite, inconsistent in length, outside their documented domains, or carry invalid weights; if n_bins is not a positive integer; or if strategy is unknown.

References

Niculescu-Mizil & Caruana (2005), “Predicting Good Probabilities with Supervised Learning”, ICML, section 4. Guo, Pleiss, Sun & Weinberger (2017), “On Calibration of Modern Neural Networks”, ICML, PMLR 70:1321–1330. The unweighted API and occupied-bin convention follow sklearn.calibration.calibration_curve().

Examples

>>> import numpy as np
>>> y_true = np.array([0, 0, 1, 1])
>>> y_pred = np.array([0.1, 0.2, 0.8, 0.9])
>>> prob_true, prob_pred, counts = calibration_curve(
...     y_true, y_pred, n_bins=2
... )
>>> prob_true
array([0., 1.])
>>> prob_pred
array([0.15, 0.85])
>>> counts
array([2, 2])

Statistical Metrics¶

Correlation Metrics¶

calibre.correlation_metrics(y_true, y_pred, *, input_scores=None, original_predictions=None)[source]¶

Calculate descriptive Spearman rank correlations.

These correlations diagnose monotonic association and rank preservation; they do not measure probability calibration. In particular, every strictly increasing transformation of a prediction vector has the same Spearman correlation even when the transformed probabilities are badly miscalibrated.

A correlation is undefined when either member of a pair is constant. The corresponding value is NaN without emitting SciPy’s constant-input warning. Weighted Spearman correlation is intentionally unsupported because SciPy has no standard weighted definition.

Parameters:

y_true (ndarray) – Observed outcomes or continuous targets.
y_pred (ndarray) – Predicted or calibrated values on the same observations.
input_scores (ndarray | None) – One-dimensional scores supplied to the calibrator.
original_predictions (ndarray | None) – Predictions before calibration.

Returns:

Spearman correlations to the outcomes and any supplied comparison vectors.

Return type:

dict

References

Spearman (1904), “The Proof and Measurement of Association between Two Things”, American Journal of Psychology 15:72–101. The calculation delegates to scipy.stats.spearmanr().

Examples

>>> import numpy as np
>>> y_true = np.array([0, 1, 1, 0, 1])
>>> y_pred = np.array([0.2, 0.7, 0.8, 0.4, 0.6])
>>> original = np.array([0.1, 0.6, 0.9, 0.3, 0.5])
>>> corr = correlation_metrics(
...     y_true, y_pred, original_predictions=original
... )
>>> sorted(corr)
['spearman_corr_to_original_predictions', 'spearman_corr_to_y_true']
>>> round(float(corr["spearman_corr_to_y_true"]), 4)
0.866
>>> round(float(corr["spearman_corr_to_original_predictions"]), 4)
1.0

Granularity Metrics¶

These measure what a calibrator did to the resolution of your scores — the thing isotonic regression quietly destroys. No other calibration package reports them.

Unique Value Counts¶

calibre.unique_value_counts(predictions, *, original_predictions=None)[source]¶

Count exact distinct values before and after calibration.

This is a structural granularity diagnostic, not a measure of calibration or statistical resolution. Counts are exact: the function does not silently merge nearby floating-point values. Apply domain-specific rounding explicitly before calling if operationally equivalent values should be grouped.

When original predictions are supplied, both vectors must describe the same observations. The ratio is the calibrated count divided by the original count; values below one indicate that calibration produced fewer distinct outputs.

Parameters:

predictions (ndarray) – Predicted or calibrated values.
original_predictions (ndarray | None) – Predictions before calibration on the same rows.

Returns:

Exact distinct-value counts and, when available, their ratio.

Return type:

dict

References

Counting delegates to numpy.unique(). Statistical resolution should instead be evaluated through a proper-score decomposition such as brier_decomposition().

Examples

>>> import numpy as np
>>> predictions = np.array([0.2, 0.7, 0.7, 0.2, 0.7])
>>> original = np.array([0.1, 0.6, 0.9, 0.2, 0.5])
>>> counts = unique_value_counts(
...     predictions, original_predictions=original
... )
>>> counts["n_unique_predictions"], counts["n_unique_original_predictions"]
(2, 5)
>>> counts["unique_prediction_ratio"]
0.4

Tie Preservation Score¶

calibre.tie_preservation_score(original_predictions, calibrated_predictions)[source]¶

Measure pairwise agreement between original and calibrated tie partitions.

Each distinct prediction value defines a group. The returned Rand index is the fraction of observation pairs that are either tied in both vectors or distinct in both vectors. It is one when calibration leaves every pair’s tie status unchanged and zero when every pair’s status changes.

Equality is exact, matching unique_value_counts(). This is a structural diagnostic, not a calibration or forecast-quality metric: prediction vectors with the same tie partition can have very different proper scores.

Parameters:

original_predictions (ndarray) – Predictions before calibration.
calibrated_predictions (ndarray) – Calibrated predictions on the same rows.

Returns:

Rand index between zero and one.

Return type:

float

References

Rand (1971), “Objective Criteria for the Evaluation of Clustering Methods”, Journal of the American Statistical Association 66(336), 846–850. The implementation delegates to sklearn.metrics.rand_score().

Examples

>>> import numpy as np
>>> original = np.array([0.1, 0.2, 0.3, 0.4])
>>> calibrated = np.array([0.1, 0.1, 0.3, 0.4])
>>> tie_preservation_score(original, calibrated)
0.8333333333333334

Usage Examples¶

Basic Evaluation¶

import numpy as np

from calibre import (
    brier_score,
    expected_calibration_error,
    mean_calibration_error,
)

y_true = np.array([0, 0, 1, 1, 1])
y_pred = np.array([0.1, 0.3, 0.6, 0.8, 0.9])

print(f"Brier score {brier_score(y_true, y_pred):.4f}")
print(f"ECE         {expected_calibration_error(y_true, y_pred, n_bins=5):.4f}")
print(f"bias        {mean_calibration_error(y_true, y_pred):.4f}")

brier_score is a proper scoring rule and the one to optimize. mean_calibration_error is calibration in the large, |mean(prediction) − base rate|. It accepts sample_weight; because opposing errors can cancel, pair it with a proper score and calibration curve.

Reporting an honest calibration error¶

On data that is calibrated by construction the true error is zero, so whatever the plugin estimator reports is bias:

import numpy as np

from calibre import debiased_calibration_error, sweep_calibration_error
from calibre.metrics import expected_calibration_error

rng = np.random.default_rng(0)
p = rng.uniform(0, 1, 4000)
y = rng.binomial(1, p).astype(float)

print(f"plugin ECE  {expected_calibration_error(y, p, n_bins=15):.4f}")
print(f"debiased    {debiased_calibration_error(y, p, n_bins=15):.4f}")
print(f"sweep       {sweep_calibration_error(y, p):.4f}")

Measuring what calibration cost you in resolution¶

import numpy as np

from calibre import (
    CenteredIsotonicCalibrator,
    IsotonicCalibrator,
    unique_value_counts,
)

rng = np.random.default_rng(0)
scores = rng.uniform(0, 1, 2000)
labels = rng.binomial(1, scores).astype(float)

for name, cal in (
    ("isotonic", IsotonicCalibrator()),
    ("centered", CenteredIsotonicCalibrator()),
):
    out = cal.fit(scores, labels).transform(scores)
    counts = unique_value_counts(out, original_predictions=scores)
    print(f"{name:9s} {counts['n_unique_predictions']:5d} distinct values")

Both are well calibrated. Only one of them still tells you which of two cases is the riskier bet.