Multiclass Calibration

There is no single best multiclass calibration method. There are two regimes with different winners, and picking wrong costs roughly a factor of six. Measured against known true probabilities over 12 seeds on 5 classes:

miscalibration	uncalibrated	temperature	per-class (CIR)
global	0.0821	0.0025	0.0165
class-dependent	0.1043	0.0849	0.0173
class-dependent + shift	0.0373	0.0276	0.0176

The winner took 12/12 seeds in every row. So measure before you choose.

Scope is deliberate: only class-wise calibration is targeted. Canonical calibration is infeasible to verify beyond four or five classes.

The Diagnostic

calibre.miscalibration_profile(P, y)

Report how miscalibration is distributed across classes.

This is the diagnostic worth running before choosing a multiclass method. If every class is miscalibrated by about the same amount, a single global correction such as temperature scaling can capture it. If the miscalibration is concentrated in particular classes, no one-parameter method can express the fix and per-class calibration is needed.

Parameters:

• P (ndarray) – Predicted probabilities, shape (n_samples, n_classes).

• y (ndarray) – Integer class labels.

Returns:

mcb (per-class miscalibration), spread (coefficient of

variation of mcb), worst_classes (indices ordered by descending mcb), and reading (a plain-language interpretation).

Return type:

dict

Notes

Calibrated on synthetic data where the true regime is known, spread is about 0.13 when the distortion is global and 0.38-0.92 when it is class-dependent. The 0.25 threshold used for reading sits between those, but it is a rule of thumb from one study design, not a test with a calibrated false-positive rate. Treat a borderline value as “try both”.

Measure this on out-of-fold predictions. Miscalibration estimated on the data a calibrator was fit to is not merely optimistic; for an isotonic-family calibrator it is identically zero.

Examples

>>> import numpy as np
>>> from calibre.multiclass import miscalibration_profile
>>> rng = np.random.default_rng(0)
>>> truth = rng.dirichlet(np.ones(4) * 0.7, size=3000)
>>> y = np.array([rng.choice(4, p=t) for t in truth])

Distort each class differently – no single temperature can undo this:

>>> skewed = truth ** np.array([0.6, 1.2, 1.8, 2.4])
>>> P = skewed / skewed.sum(axis=1, keepdims=True)
>>> profile = miscalibration_profile(P, y)
>>> bool(profile["spread"] > 0.25)
True
>>> "per-class" in profile["reading"]
True

See also

TemperatureScaler : The method to reach for when the spread is small.

Class-wise Evaluation

calibre.classwise_decomposition(P, y, score='brier')

Decompose the score of each class one-vs-rest.

Runs the CORP decomposition (calibre.evaluation.score_decomposition()) separately on each column, treating class k against all others. Each entry therefore carries the same guarantees as the binary case: the identity mean_score = MCB - DSC + UNC is exact, and MCB and DSC are non-negative.

Parameters:

• P (ndarray) – Predicted probabilities, shape (n_samples, n_classes).

• y (ndarray) – Integer class labels.

• score (str) – Proper scoring rule: "brier" (default) or "log".

Returns:

One decomposition per class, in class order. Each has

mean_score, MCB, DSC, UNC.

Return type:

list of dict

Notes

This is class-wise calibration, the standard relaxation of the multiclass problem. It says nothing about whether whole probability vectors are jointly calibrated.

Examples

>>> import numpy as np
>>> from calibre.multiclass import classwise_decomposition
>>> rng = np.random.default_rng(0)
>>> truth = rng.dirichlet(np.ones(3), size=1500)
>>> y = np.array([rng.choice(3, p=t) for t in truth])
>>> parts = classwise_decomposition(truth, y)
>>> len(parts)
3

The identity holds for every class:

>>> all(
...     abs(d["mean_score"] - (d["MCB"] - d["DSC"] + d["UNC"])) < 1e-12
...     for d in parts
... )
True

See also

miscalibration_profile : Reads these to say which calibration method to use.

calibre.classwise_ece(P, y, n_bins=15, estimator='debiased')

Average one-vs-rest calibration error across classes.

Parameters:

• P (ndarray) – Predicted probabilities, shape (n_samples, n_classes).

• y (ndarray) – Integer class labels.

• n_bins (int) – Bins per class, used by the "debiased" estimator.

• estimator (str) – "debiased" (default) subtracts the per-bin Bernoulli variance; "sweep" chooses the bin count by monotonicity instead.

Returns:

Mean per-class calibration error.

Return type:

float

Raises:

ValueError – If estimator is unknown.

Notes

Built on the bias-aware estimators in calibre.metrics, so the plugin bias that grows with the bin count is corrected, and no bin edge ever splits a group of tied predictions.

Examples

>>> import numpy as np
>>> from calibre.multiclass import classwise_ece
>>> rng = np.random.default_rng(0)
>>> truth = rng.dirichlet(np.ones(3), size=2000)
>>> y = np.array([rng.choice(3, p=t) for t in truth])
>>> bool(classwise_ece(truth, y) < 0.05)
True

calibre.top_label_ece(P, y, n_bins=15, estimator='debiased')

Calibration error of the predicted class’s confidence.

Asks whether, among the cases predicted with confidence c, the model is right about c of the time. This is the weakest and most commonly reported notion of multiclass calibration; a model can score perfectly here while being badly miscalibrated on every non-predicted class.

Parameters:

• P (ndarray) – Predicted probabilities, shape (n_samples, n_classes).

• y (ndarray) – Integer class labels.

• n_bins (int) – Bins, used by the "debiased" estimator.

• estimator (str) – "debiased" (default) or "sweep".

Returns:

Calibration error of the top-label confidence.

Return type:

float

Raises:

ValueError – If estimator is unknown.

Examples

>>> import numpy as np
>>> from calibre.multiclass import top_label_ece
>>> rng = np.random.default_rng(0)
>>> truth = rng.dirichlet(np.ones(3), size=2000)
>>> y = np.array([rng.choice(3, p=t) for t in truth])
>>> bool(top_label_ece(truth, y) < 0.05)
True

See also

classwise_ece : The stronger notion, averaging over every class.

calibre.classwise_reliability(P, y)

Build a CORP reliability diagram for each class, one-vs-rest.

Parameters:

• P (ndarray) – Predicted probabilities, shape (n_samples, n_classes).

• y (ndarray) – Integer class labels.

Returns:

One diagram per class, in class order. No

bin count to choose.

Return type:

list of ReliabilityDiagram

Examples

>>> import numpy as np
>>> from calibre.multiclass import classwise_reliability
>>> rng = np.random.default_rng(0)
>>> truth = rng.dirichlet(np.ones(3), size=1000)
>>> y = np.array([rng.choice(3, p=t) for t in truth])
>>> diagrams = classwise_reliability(truth, y)
>>> len(diagrams)
3
>>> all(np.all(np.diff(d.cep) >= -1e-12) for d in diagrams)
True

Calibrators

class calibre.TemperatureScaler(max_log_temperature=3.0)

Divide the logits by one fitted constant.

The strongest method available when miscalibration is global: on synthetic data with a single global distortion it beat per-class calibration by a factor of six, winning all 12 seeds. It has one parameter, so it cannot overfit a calibration set, and it is monotone in the logits, so the predicted class never changes – accuracy is exactly preserved.

That guarantee is also its ceiling. Because the same temperature is applied to every class, it cannot correct a distortion that differs by class; in that regime it barely improved on doing nothing. Run miscalibration_profile() first.

Parameters:

max_log_temperature (float) – The search runs over log(T) in [-b, b]. Widen only if the fitted temperature lands on a bound.

temperature_

Fitted temperature. Above 1 softens the predictions, below 1 sharpens.

n_features_in_

Number of classes seen during fit.

Notes

Temperature scaling preserves the class ordering within each row, but not the ordering of people within a class: the softmax denominator makes each calibrated probability depend on the whole row. Measured at 49.6% of adjacent within-class pairs inverted – worse than one-vs-rest calibration followed by normalisation. If you rank people by their probability of a given class, that reordering is real and no standard calibration metric reveals it.

Examples

>>> import numpy as np
>>> from calibre.multiclass import TemperatureScaler
>>> rng = np.random.default_rng(0)
>>> truth = rng.dirichlet(np.ones(4) * 0.7, size=2000)
>>> y = np.array([rng.choice(4, p=t) for t in truth])

An overconfident model, sharpened globally:

>>> sharp = truth ** 2.2
>>> P = sharp / sharp.sum(axis=1, keepdims=True)
>>> scaler = TemperatureScaler().fit(P, y)

It recovers a temperature above 1, softening the predictions back:

>>> bool(scaler.temperature_ > 1.5)
True

And the predicted class is untouched, by construction:

>>> Q = scaler.transform(P)
>>> bool(np.all(Q.argmax(axis=1) == P.argmax(axis=1)))
True

See also

miscalibration_profile : Tells you whether this method suits your data.

fit(P, y)

Fit the temperature by minimising negative log-likelihood.

Parameters:

• P (ndarray) – Predicted probabilities from a held-out calibration set.

• y (ndarray) – Integer class labels.

Returns:

self.

Return type:

TemperatureScaler

Raises:

ValueError – If max_log_temperature is not positive, or the inputs are malformed.

transform(P)

Apply the fitted temperature.

Parameters:

P (ndarray) – Predicted probabilities.

Returns:

Calibrated probabilities, rows summing to 1. Every row’s

argmax is unchanged.

Return type:

ndarray

Raises:

AttributeError – If called before fit().

fit_transform(P, y)

Fit on (P, y) and return the calibrated probabilities.

Parameters:

• P (ndarray) – Predicted probabilities.

• y (ndarray) – Integer class labels.

Returns:

Calibrated probabilities.

Return type:

ndarray

Usage

Choosing a method

import numpy as np

from calibre import miscalibration_profile

rng = np.random.default_rng(0)
truth = rng.dirichlet(np.ones(5) * 0.7, size=4000)
labels = np.array([rng.choice(5, p=t) for t in truth])

# Each class distorted by a different exponent.
skewed = truth ** np.linspace(0.6, 2.4, 5)
scores = skewed / skewed.sum(axis=1, keepdims=True)

profile = miscalibration_profile(scores, labels)
print(f"spread {profile['spread']:.2f}")
print(profile["reading"])

A spread near 0.13 means the miscalibration is even across classes and TemperatureScaler will likely capture it. A spread of 0.4 and above means it is concentrated in particular classes, and a one-parameter method applied to every class cannot express that fix.

What temperature scaling costs

TemperatureScaler never changes the predicted class, so accuracy is exactly preserved — this is asserted on every row in the test suite. But it does reorder cases within a class, at 49.6% of adjacent pairs in our measurements. No standard metric reveals this. If you rank individuals by their probability of a given class, that reordering is real.