Calibration Metrics

This module provides metrics for evaluating calibration quality.

For the CORP reliability diagram and the MCB/DSC/UNC score decomposition — which need no bin count and cannot be tuned in your favour — see CORP Evaluation.

Bias-aware calibration error

The plugin binned estimator is biased upward: part of each bin’s gap is sampling noise in the label mean rather than miscalibration, and the bias grows with the bin count. These two estimators correct for that, and are the ones to reach for when the number will be reported.

Smooth Calibration Error (smECE)

calibre.smooth_calibration_error(y_true: ndarray, y_pred: ndarray, sigma: float | None = None, return_sigma: Literal[False] = False) → float

calibre.smooth_calibration_error(y_true: ndarray, y_pred: ndarray, sigma: float | None = None, *, return_sigma: Literal[True]) → tuple[float, float]

Calculate the smooth calibration error (smECE).

Binned calibration error has no consistent limit: refine the bins and the estimate keeps climbing on data that is perfectly calibrated. smECE replaces the bins with a Gaussian kernel and, crucially, chooses its own bandwidth, so there is no knob at all – not a bin count, not a bandwidth.

\[\mathrm{smECE}_\sigma(f, y) = \frac{\int \left| (K_\sigma \star \nu)(t) \right| \, dt} {\int (K_\sigma \star \rho)(t) \, dt}, \qquad \nu = \sum_i (f_i - y_i)\, \delta_{f_i}, \quad \rho = \sum_i \delta_{f_i}\]

The bandwidth is the fixed point \(\sigma = \mathrm{smECE}_\sigma\), found by bisection. Below that width the kernel is resolving noise; above it, it is smoothing away real miscalibration.

This is a consistent calibration measure in the sense of Blasiok, Gopalan, Hu and Nakkiran (2023): it is bounded above and below by polynomial functions of the true distance to the nearest perfectly calibrated predictor. Binned ECE is not, which is why it can report a large error for a predictor that is almost calibrated and a small one for a predictor that is not.

Parameters:

• y_true (ndarray) – Ground truth values (0 or 1).

• y_pred (ndarray) – Predicted probabilities in [0, 1].

• sigma (float | None) – Kernel bandwidth. When None, the fixed point above is used, which is the recommended behaviour and what makes the estimator hyperparameter-free.

• return_sigma (bool) – Also return the bandwidth used. Worth reporting: it is an interpretable scale, roughly the resolution at which miscalibration is detectable.

Returns:

The smooth calibration error, and

the bandwidth when return_sigma.

Return type:

float or tuple of (float, float)

Raises:

ValueError – If the arrays disagree in length, y_pred falls outside [0, 1], or sigma is not positive.

See also

• debiased_calibration_error() – bias-corrected, but still needs a bin count.

• calibre.evaluation.score_decomposition() – avoids binning by using isotonic regression, and decomposes the score rather than summarising the error.

References

Blasiok & Nakkiran (2024), “Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing”, ICLR. Blasiok, Gopalan, Hu & Nakkiran (2023), “A Unifying Theory of Distance from Calibration”, STOC.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> p = rng.uniform(0, 1, 2000)
>>> y = rng.binomial(1, p).astype(float)

Calibrated by construction, so the error is near zero:

>>> bool(smooth_calibration_error(y, p) < 0.03)
True

An overconfident predictor is caught:

>>> squashed = np.clip(2.0 * (p - 0.5) + 0.5, 0, 1)
>>> bool(smooth_calibration_error(y, squashed) > 0.05)
True

Unlike a binned estimator there is no bin count to justify; the bandwidth is chosen by the data:

>>> error, width = smooth_calibration_error(y, p, return_sigma=True)
>>> bool(0.0 < width <= 1.0)
True

Unlike everything below it, smECE has no bin count and no bandwidth to choose, and it is a consistent measure of distance from calibration. It is the one to reach for when the number will be quoted without qualification.

Debiased Calibration Error

calibre.debiased_calibration_error(y_true, y_pred, n_bins=15, squared=False)

Calculate the debiased \(\ell_2\) calibration error.

The plugin binned estimator is biased upward: each bin contributes the squared gap between mean prediction and mean label, and part of that gap is sampling noise in the label mean rather than miscalibration. The bias is roughly n_bins / n, so it grows as bins are added – which is exactly when a finer picture of the calibration curve is wanted. Subtracting the per-bin Bernoulli variance removes it.

\[\widehat{\mathrm{CE}}^2 = \sum_k \frac{n_k}{n} \left[ (\bar{f}_k - \bar{y}_k)^2 - \frac{\bar{y}_k (1 - \bar{y}_k)}{n_k - 1} \right]\]

Parameters:

• y_true (ndarray) – Ground truth values (0 or 1).

• y_pred (ndarray) – Predicted probabilities.

• n_bins (int) – Number of equal-mass bins. Defaults to 15, following Guo et al. (2017) as used by Roelofs et al.

• squared (bool) – Return the estimate of the squared error instead, without the square root or the floor at zero. This is the quantity the correction actually makes unbiased, and it may legitimately come out negative – see Notes.

Returns:

Debiased calibration error. Floored at zero: the correction can

drive the sum negative on well-calibrated data, which is evidence of no detectable miscalibration rather than of negative error. With squared=True the unfloored sum is returned instead.

Return type:

float

Raises:

ValueError – If the arrays disagree in length or n_bins is below 1.

Notes

This is the \(\ell_2\) error, so it is not comparable in magnitude to expected_calibration_error(), which is \(\ell_1\).

The correction is unbiased on the squared scale, not on the error scale. Measured on 400 perfectly calibrated samples of 1500 observations, where the true error is exactly zero (tests/test_monte_carlo.py):

quantity	mean	distance from zero
squared=True (the sum itself)	+4.5e-05	1.3 standard errors
default (sqrt of the floored sum)	+0.0106	15.7 standard errors

The sum is unbiased, exactly as intended, and comes out negative on 53% of calibrated samples – what an unbiased estimate of zero should do. The floor then discards that half, and no amount of data removes the resulting upward bias in the reported error. (The square root pulls the other way, being concave: E[sqrt(W)] of 0.0106 against sqrt(E[W]) of 0.0172.)

So: to report an error, use the default, and read a small positive value on well-calibrated data as the floor rather than as evidence. To average across folds, compare two models, or do anything else that assumes unbiasedness, use squared=True and take the square root at the very end, if at all.

References

Bröcker (2012); Ferro & Fricker (2012); Kumar, Liang & Ma (2019), “Verified Uncertainty Calibration”, NeurIPS.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> p = rng.uniform(0, 1, 4000)
>>> y = rng.binomial(1, p).astype(float)

These predictions are calibrated, so the plugin estimator reports error that is not there while the debiased one does not:

>>> plugin = expected_calibration_error(y, p, n_bins=15)
>>> debiased = debiased_calibration_error(y, p, n_bins=15)
>>> bool(plugin > 0.01), bool(debiased < 0.01)
(True, True)

See also

sweep_calibration_error : Chooses the bin count instead of fixing it. calibre.evaluation.score_decomposition : Avoids binning altogether.

Sweep Calibration Error

calibre.sweep_calibration_error(y_true: ndarray, y_pred: ndarray, p: int = 1, return_n_bins: Literal[False] = False) → float

calibre.sweep_calibration_error(y_true: ndarray, y_pred: ndarray, p: int = 1, *, return_n_bins: Literal[True]) → tuple[float, int]

Calculate the monotonic sweep calibration error (ECE_sweep).

Fixing the bin count is the weak point of binned calibration error: too few bins hide miscalibration, too many measure noise, and the best choice depends on the sample size and the score distribution. This estimator chooses instead.

A true calibration curve is non-decreasing – a model’s accuracy should not fall as its confidence rises. So bins are added while the observed bin heights stay monotone, and the sweep stops at the largest bin count for which they do. Non-monotonicity is the signal that the bins have become fine enough to be reading noise.

Parameters:

• y_true (ndarray) – Ground truth values (0 or 1).

• y_pred (ndarray) – Predicted probabilities.

• p (int) – Norm. 1 gives the familiar weighted mean absolute gap.

• return_n_bins (bool) – Also return the bin count the sweep settled on. That number is half of what the estimator has to say – it is the sweep’s answer to “how fine can these data support?” – and reporting only the error hides it.

Returns:

Binned calibration error at the

selected bin count, and that bin count when return_n_bins is True. The count is the number of bins actually occupied, which ties can hold below the number the sweep reached.

Return type:

float or tuple of (float, int)

Raises:

ValueError – If the arrays disagree in length or p is below 1.

References

Roelofs, Cain, Shlens & Mozer (2022), “Mitigating Bias in Calibration Error Estimation”, AISTATS. Algorithm 1.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> p = rng.uniform(0, 1, 4000)
>>> y = rng.binomial(1, p).astype(float)
>>> float(sweep_calibration_error(y, p)) < 0.05
True

See also

debiased_calibration_error : Fixes the bin count and corrects the bias. calibre.evaluation.score_decomposition : Lets isotonic regression bin.

Plugin Calibration Error

calibre.plugin_calibration_error(y_true, y_pred, n_bins=15, p=2)

Calculate the uncorrected \(\ell_p\) binned calibration error.

\[\widehat{\mathrm{CE}}_p = \left[ \sum_k \frac{n_k}{n} \left| \bar{f}_k - \bar{y}_k \right|^p \right]^{1/p}\]

This is the plain plugin estimator: the quantity debiased_calibration_error() corrects and sweep_calibration_error() chooses a bin count for. It exists so those three can be compared on equal terms.

That comparison is otherwise a trap. expected_calibration_error() is \(\ell_1\) on uniform-width bins, debiased_calibration_error() is \(\ell_2\) on equal-mass bins, and sweep_calibration_error() is \(\ell_1\) on equal-mass bins by default – so plotting them against each other compares three different quantities and reads as disagreement between estimators. This function takes both the norm and the bin count as arguments and uses the same equal-mass, tie-safe binning as the bias-aware estimators, so the only thing that differs is the bias correction.

Parameters:

• y_true (ndarray) – Ground truth values (0 or 1).

• y_pred (ndarray) – Predicted probabilities.

• n_bins (int) – Number of equal-mass bins. Fewer are used when ties prevent it.

• p (int) – Norm. 1 gives the familiar weighted mean absolute gap; 2 matches debiased_calibration_error().

Returns:

The uncorrected calibration error. Biased upward, and

increasingly so as n_bins grows.

Return type:

float

Raises:

ValueError – If the arrays disagree in length, n_bins is below 1, or p is below 1.

See also

debiased_calibration_error : The same quantity at p=2, bias-corrected. sweep_calibration_error : Chooses n_bins rather than fixing it.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> p_hat = rng.uniform(0, 1, 4000)
>>> y = rng.binomial(1, p_hat).astype(float)

These are calibrated by construction, so the true error is zero and whatever the plugin reports is bias – which grows with the bin count:

>>> coarse = plugin_calibration_error(y, p_hat, n_bins=5)
>>> fine = plugin_calibration_error(y, p_hat, n_bins=50)
>>> bool(fine > coarse)
True

Debiasing removes it:

>>> bool(debiased_calibration_error(y, p_hat, n_bins=50) < fine)
True

Calibration Error Metrics

Mean Calibration Error

calibre.mean_calibration_error(y_true, y_pred)

Calculate the mean calibration error: the bias of the predictions.

\[\left| \mathbb{E}[\hat{p}] - \mathbb{E}[y] \right|\]

This is calibration in the large – whether the predictions are right on average. It is zero for any predictor whose mean matches the base rate, and it says nothing about calibration within subgroups; for that use expected_calibration_error().

Parameters:

• y_true (ndarray) – Ground truth values (0 or 1 for binary classification).

• y_pred (ndarray) – Predicted probabilities.

Returns:

Absolute difference between the mean prediction and the base rate.

Return type:

float

Raises:

ValueError – If arrays have different shapes.

Notes

Changed in version 0.7.0: Previously this returned mean(|y_pred - y_true|), which is mean absolute error, not a calibration error at all: it is minimised by hard 0/1 predictions and is nonzero for a perfectly calibrated model – a perfectly calibrated constant predictor of 0.5 scored 0.5. Use sklearn.metrics.mean_absolute_error() if you want the old quantity.

Examples

>>> import numpy as np
>>> y_true = np.array([0, 1, 1, 0, 1])
>>> y_pred = np.array([0.2, 0.7, 0.8, 0.4, 0.6])
>>> round(mean_calibration_error(y_true, y_pred), 4)   # mean 0.54 vs base 0.6
0.06

A perfectly calibrated predictor scores zero, however unsharp it is:

>>> y = np.array([0, 0, 1, 1])
>>> mean_calibration_error(y, np.full(4, 0.5))
0.0

Binned Calibration Error

calibre.binned_calibration_error(y_true, y_pred, x=None, n_bins=10, strategy='uniform', return_details=False)

Calculate binned calibration error.

Parameters:

• y_true (ndarray) – Ground truth values.

• y_pred (ndarray) – Predicted values.

• x (ndarray | None) – Input features for binning. If None, y_pred is used for binning.

• n_bins (int) – Number of bins.

• strategy (str) –

  Strategy for binning:

  • ’uniform’: Bins with uniform widths.

  • ’quantile’: Bins with approximately equal counts.

• return_details (bool) – If True, return bin details (bin centers, counts, mean predictions, mean truths).

Returns:

Binned calibration error. If return_details is True, returns a

dictionary with BCE and bin details.

Return type:

bce

Raises:

ValueError – If arrays have different lengths or unknown binning strategy.

Examples

>>> import numpy as np
>>> y_true = np.array([0, 1, 1, 0, 1])
>>> y_pred = np.array([0.2, 0.7, 0.8, 0.4, 0.6])
>>> binned_calibration_error(y_true, y_pred, n_bins=2)
0.3

Expected Calibration Error

calibre.expected_calibration_error(y_true, y_pred, n_bins=10)

Calculate Expected Calibration Error (ECE).

The ECE is a weighted average of the absolute calibration error across bins, where each bin’s weight is proportional to the number of samples in the bin.

Parameters:

• y_true (ndarray) – Ground truth values (0 or 1 for binary classification).

• y_pred (ndarray) – Predicted probabilities.

• n_bins (int) – Number of bins for discretizing predictions.

Returns:

Expected Calibration Error.

Return type:

ece

Raises:

ValueError – If arrays have different lengths.

Examples

>>> import numpy as np
>>> y_true = np.array([0, 1, 1, 0, 1])
>>> y_pred = np.array([0.2, 0.7, 0.8, 0.4, 0.6])
>>> float(expected_calibration_error(y_true, y_pred, n_bins=2))
0.3

Maximum Calibration Error

calibre.maximum_calibration_error(y_true, y_pred, n_bins=10)

Calculate Maximum Calibration Error (MCE).

The MCE is the maximum absolute difference between the average predicted probability and the fraction of positive samples in any bin.

Parameters:

• y_true (ndarray) – Ground truth values (0 or 1 for binary classification).

• y_pred (ndarray) – Predicted probabilities.

• n_bins (int) – Number of bins for discretizing predictions.

Returns:

Maximum Calibration Error.

Return type:

mce

Raises:

ValueError – If arrays have different lengths.

Examples

>>> import numpy as np
>>> y_true = np.array([0, 1, 1, 0, 1])
>>> y_pred = np.array([0.2, 0.7, 0.8, 0.4, 0.6])
>>> round(float(maximum_calibration_error(y_true, y_pred, n_bins=2)), 4)
0.3

Note

These estimators are not interchangeable, and their magnitudes are not comparable. expected_calibration_error() and sweep_calibration_error() are \(\ell_1\); debiased_calibration_error() is \(\ell_2\). expected_calibration_error() uses uniform-width bins, while the two bias-aware estimators use equal-mass bins. Compare like with like.

Scoring Metrics

Brier Score

calibre.brier_score(y_true, y_pred)

Calculate the Brier score.

The Brier score is a proper scoring rule that measures the mean squared difference between predicted probabilities and the actual outcomes.

Parameters:

• y_true (ndarray) – Ground truth values (0 or 1 for binary classification).

• y_pred (ndarray) – Predicted probabilities.

Returns:

Brier score (lower is better).

Return type:

score

Raises:

ValueError – If arrays have different lengths.

Examples

>>> import numpy as np
>>> y_true = np.array([0, 1, 1, 0, 1])
>>> y_pred = np.array([0.2, 0.7, 0.8, 0.4, 0.6])
>>> brier_score(y_true, y_pred)
0.098

Calibration Curve

calibre.calibration_curve(y_true, y_pred, n_bins=10, strategy='uniform')

Compute the calibration curve for binary classification.

Parameters:

• y_true (ndarray) – Ground truth values (0 or 1 for binary classification).

• y_pred (ndarray) – Predicted probabilities.

• n_bins (int) – Number of bins for discretizing predictions.

• strategy (str) –

  Strategy for binning:

  • ’uniform’: Bins with uniform widths.

  • ’quantile’: Bins with approximately equal counts.

Returns:

The true fraction of positive samples in each bin. prob_pred: The mean predicted probability in each bin. counts: The number of samples in each bin.

Return type:

prob_true

Raises:

ValueError – If arrays have different lengths or unknown binning strategy.

Examples

>>> import numpy as np
>>> y_true = np.array([0, 1, 1, 0, 1, 0, 1, 0, 1, 0])
>>> y_pred = np.array([0.1, 0.9, 0.8, 0.3, 0.7, 0.2, 0.6, 0.4, 0.9, 0.1])
>>> prob_true, prob_pred, counts = calibration_curve(y_true, y_pred, n_bins=5)

Statistical Metrics

Correlation Metrics

calibre.correlation_metrics(y_true, y_pred, x=None, y_orig=None)

Calculate correlation metrics between various signals.

Parameters:

• y_true (ndarray) – Ground truth values.

• y_pred (ndarray) – Predicted/calibrated values.

• x (ndarray | None) – Input features.

• y_orig (ndarray | None) – Original uncalibrated predictions.

Returns:

Dictionary of correlation metrics.

Return type:

correlations

Examples

>>> import numpy as np
>>> y_true = np.array([0, 1, 1, 0, 1])
>>> y_pred = np.array([0.2, 0.7, 0.8, 0.4, 0.6])
>>> y_orig = np.array([0.1, 0.6, 0.9, 0.3, 0.5])
>>> corr = correlation_metrics(y_true, y_pred, y_orig=y_orig)
>>> sorted(corr)
['spearman_corr_orig_to_calib', 'spearman_corr_to_y_orig',
 'spearman_corr_to_y_true']
>>> round(float(corr["spearman_corr_to_y_true"]), 4)
0.866
>>> round(float(corr["spearman_corr_to_y_orig"]), 4)
1.0

Granularity Metrics

These measure what a calibrator did to the resolution of your scores — the thing isotonic regression quietly destroys. No other calibration package reports them.

Unique Value Counts

calibre.unique_value_counts(y_pred, y_orig=None, precision=6)

Count unique values in predictions.

Parameters:

• y_pred (ndarray) – Predicted/calibrated values.

• y_orig (ndarray | None) – Original uncalibrated predictions.

• precision (int) – Decimal precision for rounding.

Returns:

Dictionary with counts of unique values.

Return type:

counts

Examples

>>> import numpy as np
>>> y_pred = np.array([0.2, 0.7, 0.8, 0.2, 0.7])
>>> y_orig = np.array([0.1, 0.6, 0.9, 0.2, 0.5])
>>> unique_value_counts(y_pred, y_orig)
{'n_unique_y_pred': 3, 'n_unique_y_orig': 5, 'unique_value_ratio': 0.6}

Calibration Diversity Index

calibre.calibration_diversity_index(y_calibrated, reference_diversity=None)

Measure granularity preservation in calibrated predictions.

Parameters:

• y_calibrated (ndarray) – Calibrated predictions.

• reference_diversity (float | None) – Reference diversity to compare against (e.g., diversity of original predictions). If None, returns absolute diversity.

Returns:

Diversity index. Higher values indicate more granular

predictions. If reference_diversity is provided, returns relative diversity.

Return type:

diversity

Examples

>>> import numpy as np
>>> y_cal = np.array([0.1, 0.2, 0.3, 0.4, 0.5])
>>> diversity = calibration_diversity_index(y_cal)
>>> diversity > 0
True

Tie Preservation Score

calibre.tie_preservation_score(y_original, y_calibrated, tolerance=1e-10)

Measure how well calibration preserves genuine ties while removing spurious ones.

Parameters:

• y_original (ndarray) – Original predicted probabilities before calibration.

• y_calibrated (ndarray) – Calibrated probabilities.

• tolerance (float) – Tolerance for considering values as tied.

Raises:

ValueError – If arrays have different lengths.

Returns:

Tie preservation score between 0 and 1. Higher values indicate

better preservation of meaningful ties.

Return type:

score

Examples

>>> import numpy as np
>>> y_orig = np.array([0.1, 0.15, 0.2, 0.6, 0.65, 0.7])
>>> y_cal = np.array([0.1, 0.15, 0.2, 0.65, 0.65, 0.65])
>>> score = tie_preservation_score(y_orig, y_cal)
>>> 0 <= score <= 1
True

Plateau Quality Score

calibre.plateau_quality_score(X, y, y_calibrated)

Overall quality score for plateaus in calibrated predictions.

Parameters:

• X (ndarray) – Input features.

• y (ndarray) – True target values.

• y_calibrated (ndarray) – Calibrated predictions.

Raises:

ValueError – If arrays have different lengths.

Returns:

Quality score between 0 and 1. Higher values indicate better

plateau quality.

Return type:

score

Examples

>>> import numpy as np
>>> X = np.array([0.1, 0.2, 0.3, 0.4, 0.5])
>>> y = np.array([0, 0, 1, 1, 1])
>>> y_cal = np.array([0.1, 0.25, 0.25, 0.4, 0.6])
>>> score = plateau_quality_score(X, y, y_cal)
>>> bool(0 <= score <= 1)
True

Progressive Sampling Diversity

calibre.progressive_sampling_diversity(X, y, sample_sizes=None, n_trials=10, random_state=None)

Compute diversity vs sample size curve for progressive sampling analysis.

Parameters:

• X (ndarray) – Input features.

• y (ndarray) – Target values.

• sample_sizes (list[int] | None) – Sample sizes to test. If None, uses default range.

• n_trials (int) – Number of trials per sample size.

• random_state (int | None) – Random state for reproducibility.

Raises:

ValueError – If X and y have different lengths.

Returns:

Sample sizes tested. diversities: Average diversity at each sample size.

Return type:

sizes

Examples

>>> import numpy as np
>>> X = np.linspace(0, 1, 100)
>>> y = np.random.binomial(1, X, 100)
>>> sizes, divs = progressive_sampling_diversity(
...     X, y, sample_sizes=[20, 50, 80]
... )
>>> len(sizes) == len(divs) == 3
True

Usage Examples

Basic Evaluation

import numpy as np

from calibre import (
    brier_score,
    expected_calibration_error,
    mean_calibration_error,
)

y_true = np.array([0, 0, 1, 1, 1])
y_pred = np.array([0.1, 0.3, 0.6, 0.8, 0.9])

print(f"Brier score {brier_score(y_true, y_pred):.4f}")
print(f"ECE         {expected_calibration_error(y_true, y_pred, n_bins=5):.4f}")
print(f"bias        {mean_calibration_error(y_true, y_pred):.4f}")

brier_score is a proper scoring rule and the one to optimise. mean_calibration_error is calibration in the large, |mean(prediction) − base rate|.

Reporting an honest calibration error

On data that is calibrated by construction the true error is zero, so whatever the plugin estimator reports is bias:

import numpy as np

from calibre import debiased_calibration_error, sweep_calibration_error
from calibre.metrics import expected_calibration_error

rng = np.random.default_rng(0)
p = rng.uniform(0, 1, 4000)
y = rng.binomial(1, p).astype(float)

print(f"plugin ECE  {expected_calibration_error(y, p, n_bins=15):.4f}")
print(f"debiased    {debiased_calibration_error(y, p, n_bins=15):.4f}")
print(f"sweep       {sweep_calibration_error(y, p):.4f}")

Measuring what calibration cost you in resolution

import numpy as np

from calibre import (
    CenteredIsotonicCalibrator,
    IsotonicCalibrator,
    unique_value_counts,
)

rng = np.random.default_rng(0)
scores = rng.uniform(0, 1, 2000)
labels = rng.binomial(1, scores).astype(float)

for name, cal in (
    ("isotonic", IsotonicCalibrator()),
    ("centered", CenteredIsotonicCalibrator()),
):
    out = cal.fit(scores, labels).transform(scores)
    counts = unique_value_counts(out, y_orig=scores)
    print(f"{name:9s} {counts['n_unique_y_pred']:5d} distinct values")

Both are well calibrated. Only one of them still tells you which of two cases is the riskier bet.