Calibration Methods

Every calibrator follows the scikit-learn transformer API: .fit(scores, labels) and .transform(scores), plus sample_weight where it is meaningful. All are binary; for multiclass see Multiclass Calibration.

Which calibrator should I use?

If you don’t want to think about it: CenteredIsotonicCalibrator. It is non-parametric, has nothing to tune, is monotone, and has no plateaus.

You want

Use

Notes

A drop-in isotonic replacement, no tuning

CenteredIsotonicCalibrator

Collapses isotonic’s flat steps to points and interpolates. O(n).

A smooth curve, and you can afford cross-validation

SplineCalibrator

Monotone spline; picks its own smoothing by CV on log-loss.

A smooth curve with smoothing you control

RegularizedIsotonicCalibrator

Same model, you set alpha instead of tuning it. Fast.

Exactly scikit-learn’s isotonic behaviour

IsotonicCalibrator

Thin wrapper, plus optional plateau diagnostics.

Guaranteed strictly increasing output

RelaxedPAVACalibrator

min_slope forces a minimum step between adjacent scores.

To allow small ranking violations if they fit better

NearlyIsotonicCalibrator

lam trades monotonicity against fit.

Accuracy near specific decision thresholds

CDIIsotonicCalibrator

Research-grade; needs your operating thresholds.

Base Classes

class calibre.BaseCalibrator(enable_diagnostics=False)[source]

Bases: BaseEstimator, TransformerMixin

Base class for all calibrators.

All calibrator classes should inherit from this base class to ensure consistent API and functionality. This follows the scikit-learn transformer interface with fit/transform/fit_transform methods.

Parameters:

enable_diagnostics (bool) – Whether to run plateau diagnostics after fitting.

Notes

Subclasses must implement the fit() and transform() methods. The fit_transform() method is provided by default.

Examples

>>> import numpy as np
>>> from calibre import BaseCalibrator
>>>
>>> class SimpleCalibrator(BaseCalibrator):
...     def __init__(self, enable_diagnostics=False):
...         super().__init__(enable_diagnostics=enable_diagnostics)
...     def fit(self, X, y):
...         self.mean_ = np.mean(y)
...         return self
...
...     def transform(self, X):
...         return np.full_like(X, self.mean_)
>>>
>>> X = np.array([0.1, 0.3, 0.5])
>>> y = np.array([0, 1, 1])
>>>
>>> cal = SimpleCalibrator()
>>> _ = cal.fit(X, y)
>>> cal.transform(X)
array([0.66666667, 0.66666667, 0.66666667])
fit(X, y, sample_weight=None)[source]

Fit the calibrator.

This method implements the template method pattern: it handles data storage and diagnostics, while delegating the actual fitting logic to the abstract _fit_impl() method that subclasses must implement.

Parameters:

  • X (ndarray) – The values to be calibrated (e.g., predicted probabilities).

  • y (ndarray) – The target values (e.g., true labels).

  • sample_weight (ndarray | None) – Non-negative per-observation weights. Calibrators that cannot honour weights raise rather than ignore them.

Returns:

Returns self for method chaining.

Return type:

BaseCalibrator

transform(X)[source]

Apply calibration to new data.

Parameters:

X (ndarray) – The values to be calibrated.

Returns:

Calibrated values.

Raises:

NotImplementedError – This method must be implemented by subclasses.

Return type:

ndarray

fit_transform(X, y, sample_weight=None, **fit_params)[source]

Fit the calibrator and then transform the data.

This is a convenience method that combines fit() and transform() in a single call. The default implementation simply calls fit() followed by transform().

Parameters:

  • X (ndarray) – The values to be calibrated.

  • y (ndarray) – The target values.

  • sample_weight (ndarray | None) – Non-negative per-observation weights.

  • **fit_params (object) – Ignored. Accepted for scikit-learn pipeline compatibility.

Returns:

Calibrated values.

Return type:

ndarray

Examples

>>> import numpy as np
>>> from calibre import IsotonicCalibrator
>>> X = np.array([0.1, 0.3, 0.5, 0.7, 0.9])
>>> y = np.array([0, 0, 1, 1, 1])
>>> cal = IsotonicCalibrator()
>>> X_calibrated = cal.fit_transform(X, y)
has_diagnostics()[source]

Check if diagnostic information is available.

Returns:

True if diagnostics have been computed and are available.

Return type:

has_diag

Examples

>>> from calibre import IsotonicCalibrator
>>> import numpy as np
>>>
>>> X = np.array([0.1, 0.3, 0.5])
>>> y = np.array([0, 1, 1])
>>>
>>> cal = IsotonicCalibrator(enable_diagnostics=True)
>>> _ = cal.fit(X, y)
>>> cal.has_diagnostics()
True
get_diagnostics()[source]

Get diagnostic results.

Returns:

Diagnostic results from plateau analysis, or None if

diagnostics were not computed or are not available.

Return type:

dict | None

Examples

>>> from calibre import IsotonicCalibrator
>>> import numpy as np
>>>
>>> X = np.array([0.1, 0.3, 0.5])
>>> y = np.array([0, 1, 1])
>>>
>>> cal = IsotonicCalibrator(enable_diagnostics=True)
>>> _ = cal.fit(X, y)
>>> cal.get_diagnostics()["n_plateaus"]
1
diagnostic_summary()[source]

Get a human-readable summary of diagnostic analysis.

Returns:

Human-readable plateau summary.

Return type:

summary

Examples

>>> from calibre import IsotonicCalibrator
>>> import numpy as np
>>>
>>> X = np.array([0.1, 0.3, 0.5, 0.7, 0.9])
>>> y = np.array([0, 0, 1, 1, 1])
>>>
>>> cal = IsotonicCalibrator(enable_diagnostics=True)
>>> _ = cal.fit(X, y)
>>> print(cal.diagnostic_summary())
Detected 2 plateau(s):

Warnings:
  ... Plateau 1 at [0.100, 0.300] has only 2 samples - may be unreliable
  ... Plateau 2 at [0.500, 0.900] has only 3 samples - may be unreliable
set_fit_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

BaseCalibrator

class calibre.MonotonicMixin[source]

Mixin for calibrators that maintain monotonicity.

This mixin provides utility methods for calibrators that aim to preserve or enforce monotonic relationships between inputs and outputs.

check_monotonicity(y)[source]

Check if an array is monotonically increasing.

Parameters:
Return type:

bool

enforce_monotonicity(y)[source]

Enforce monotonicity on an array.

Parameters:
Return type:

ndarray

Notes

This is a utility mixin that doesn’t require any specific attributes. It’s designed to be mixed in with BaseCalibrator subclasses that need monotonicity guarantees.

static check_monotonicity(y, strict=False)[source]

Check if an array is monotonically increasing.

Parameters:

  • y (ndarray) – Values to check for monotonicity.

  • strict (bool) – If True, check for strictly increasing (no equal consecutive values). If False, check for non-decreasing (allows equal consecutive values).

Returns:

True if the array is monotonic according to the specified criteria.

Return type:

bool

Examples

>>> import numpy as np
>>> from calibre.base import MonotonicMixin
>>>
>>> y1 = np.array([0.1, 0.2, 0.3, 0.4])
>>> MonotonicMixin.check_monotonicity(y1)
True
>>>
>>> y2 = np.array([0.1, 0.3, 0.2, 0.4])
>>> MonotonicMixin.check_monotonicity(y2)
False
>>>
>>> y3 = np.array([0.1, 0.2, 0.2, 0.3])
>>> MonotonicMixin.check_monotonicity(y3, strict=False)
True
>>> MonotonicMixin.check_monotonicity(y3, strict=True)
False
static enforce_monotonicity(y, inplace=False)[source]

Enforce monotonicity on an array.

This method ensures the array is non-decreasing by replacing any value that is less than the previous value with the previous value.

Parameters:

  • y (ndarray) – Values to make monotonic.

  • inplace (bool) – If True, modify the array in place. Otherwise, return a copy.

Returns:

Monotonically increasing version of the input array.

Return type:

ndarray

Examples

>>> import numpy as np
>>> from calibre.base import MonotonicMixin
>>>
>>> y = np.array([0.1, 0.3, 0.2, 0.5, 0.4])
>>> y_mono = MonotonicMixin.enforce_monotonicity(y)
>>> print(y_mono)
[0.1 0.3 0.3 0.5 0.5]
>>>
>>> # Original array unchanged
>>> print(y)
[0.1 0.3 0.2 0.5 0.4]

Other Calibrators

Isotonic Calibrator

class calibre.IsotonicCalibrator(y_min=None, y_max=None, increasing=True, out_of_bounds='clip', enable_diagnostics=False)[source]

Bases: BaseCalibrator

Isotonic regression calibrator.

This calibrator wraps sklearn’s IsotonicRegression for probability calibration.

Parameters:

  • y_min (float | None) – Lower bound for the calibrated values.

  • y_max (float | None) – Upper bound for the calibrated values.

  • increasing (bool) – Whether the calibration function should be increasing.

  • out_of_bounds (str) – How to handle out-of-bounds values in transform. Options: ‘nan’, ‘clip’, ‘raise’.

  • enable_diagnostics (bool) – Whether to enable plateau diagnostics analysis.

Examples

>>> import numpy as np
>>> from calibre import IsotonicCalibrator
>>>
>>> X = np.array([0.1, 0.2, 0.3, 0.4, 0.5])
>>> y = np.array([0, 0, 1, 1, 1])
>>>
>>> # Basic usage
>>> cal = IsotonicCalibrator()
>>> _ = cal.fit(X, y)
>>> X_calibrated = cal.transform(X)
>>>
>>> # With diagnostics
>>> cal = IsotonicCalibrator(enable_diagnostics=True)
>>> _ = cal.fit(X, y)
>>> cal.has_diagnostics()
True
>>> cal.get_diagnostics()["n_plateaus"]
2

Notes

Isotonic regression finds the best monotonic fit to the data, which is particularly useful for calibration because well-calibrated predictions should maintain the rank order of predictions while improving probability estimates.

See also

NearlyIsotonicCalibrator : Relaxed monotonicity constraint SmoothedIsotonicCalibrator : Isotonic with smoothing

transform(X)[source]

Apply isotonic calibration to new data.

Parameters:

X (ndarray) – The values to be calibrated.

Returns:

Calibrated values.

Raises:

ValueError – If called before fit().

Return type:

ndarray

set_fit_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

IsotonicCalibrator

Spline Calibrator

class calibre.SplineCalibrator(n_knots=10, degree=3, knots='quantile', alpha=None, link='logit', cv=5, max_cv_samples=20000, random_state=0, clip_output=True, enable_diagnostics=False)[source]

Bases: BaseCalibrator

Monotone spline calibration with cross-validated smoothing.

Fits

\[g\big(f(x)\big) = \theta + \sum_k \delta_k I_k(x), \qquad \delta_k \ge 0\]

where the \(I_k\) are I-splines (each non-decreasing) and \(g\) is the link. Because every basis function is non-decreasing and every coefficient is non-negative, \(f\) is non-decreasing by construction; the link is increasing, so the calibrated probability is too.

Parameters:

  • n_knots (int) – Number of knots. The basis has n_knots + degree - 1 functions. Used only when alpha is given; otherwise cross-validation selects it.

  • degree (int) – B-spline degree. 3 gives the usual cubic behaviour.

  • knots (str) – "quantile" (default) places knots at score quantiles; "uniform" spaces them evenly. Quantile is normally right for calibration, where scores pile up wherever the base model is confident and uniform knots spend resolution on empty regions.

  • alpha (float | None) – Roughness penalty on the coefficient increments. None (default) selects it, along with n_knots, by cross-validation. A number fixes it and skips cross-validation.

  • link (str) – "logit" (default) fits a penalised Bernoulli likelihood: log-loss is the proper score for binary labels, and predictions land in (0, 1) with no clipping. "identity" fits penalised least squares on the probability scale – a single bounded linear solve.

  • cv (int) – Number of cross-validation folds. Stratified when y is binary.

  • max_cv_samples (int | None) – Cap on the number of observations used for hyperparameter selection. The final model is always refit on the full sample; this only bounds the cost of the search, which would otherwise fit the grid once per fold over every row (at n=100k that is ~50s against 0.3s for a single fit). Selecting two scalars from a large random subsample costs essentially nothing statistically. Set to None to search on all of the data.

  • random_state (int | None) – Seed for the cross-validation split. Defaults to 0 so that fit is reproducible: cross-validation here only selects a hyperparameter, and a fit that silently returns a different curve on each identical call is a trap. Pass None to draw the split from the global RNG instead.

  • clip_output (bool) – Clip calibrated values into [0, 1]. A no-op for link="logit".

  • enable_diagnostics (bool) – Whether to enable plateau diagnostics analysis.

basis_

The fitted basis. Its knots come from the same fit that produced coef_.

intercept_

Fitted intercept, on the link scale.

coef_

Fitted non-negative increment coefficients.

alpha_

The penalty actually used – selected by cross-validation, or echoed back from alpha.

n_knots_

The knot count actually used.

n_features_in_

Always 1. Present for scikit-learn compatibility.

Notes

Non-negative coefficients on a plain B-spline basis do not give monotonicity. B-spline basis functions are bumps, so a non-negative combination of them is a non-negative function and nothing more – a single non-negative coefficient already traces a curve that rises and then falls. Monotonicity requires non-negativity on the coefficient differences, which is exactly what the I-spline (cumulative) basis encodes; see calibre._core.MonotoneSplineBasis. This is the construction behind the SCOP-splines of Pya & Wood (2015) in R’s scam and the penalised B-splines of Eilers & Marx (1996).

Cross-validation selects a hyperparameter and then refits on all the data. It is not a search for whichever fold’s model scored best on its own validation split: that selects on noise and ships a model trained on only (cv-1)/cv of the sample. Folds are scored by log-loss – a proper score – rather than by \(R^2\).

Examples

>>> import numpy as np
>>> from calibre import SplineCalibrator
>>>
>>> rng = np.random.default_rng(0)
>>> x = rng.random(500)
>>> y = (rng.random(500) < x).astype(float)
>>>
>>> cal = SplineCalibrator(alpha=0.1).fit(x, y)
>>> fitted = cal.transform(np.linspace(0, 1, 200))
>>> bool(np.all(np.diff(fitted) >= -1e-10))     # monotone by construction
True
>>> bool(fitted.min() >= 0.0 and fitted.max() <= 1.0)
True

See also

CenteredIsotonicCalibrator : Non-parametric, needs no tuning, also plateau-free. RegularizedIsotonicCalibrator : Same basis, penalty specified rather than tuned.

transform(X)[source]

Map scores through the fitted calibration curve.

Parameters:

X (ndarray) – Scores to calibrate.

Returns:

Calibrated probabilities.

Return type:

ndarray of shape (n_samples,)

Raises:

AttributeError – If called before fit().

calibration_curve(n_points=200)[source]

Sample the fitted map onto a grid, for plotting or inspection.

Parameters:

n_points (int) – Number of grid points across the fitted score range.

Returns:

The sampled curve.

Return type:

PiecewiseLinear

Raises:

AttributeError – If called before fit().

set_fit_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

SplineCalibrator

Regularized Isotonic Calibrator

class calibre.RegularizedIsotonicCalibrator(alpha='auto', n_knots=10, degree=3, knots='quantile', link='logit', cv=5, scoring='log_loss', random_state=0, clip_output=True, enable_diagnostics=False)[source]

Bases: BaseCalibrator

Monotone calibration with an explicit roughness penalty.

Solves

\[\min_{\theta,\ \delta \ge 0}\ \mathcal{L}\big(\theta + M\delta;\ y, w\big) + \alpha \lVert \Delta\delta \rVert^2\]

where M is an I-spline design, so delta >= 0 makes the fit monotone by construction, and \(\Delta\delta\) is the second difference of the underlying B-spline coefficients.

Parameters:

  • alpha (float | str) – Roughness penalty. 0 gives an unpenalised monotone spline; larger values drive the fit toward the best monotone straight line.

  • n_knots (int) – Number of knots in the basis.

  • degree (int) – B-spline degree.

  • knots (str) – "quantile" or "uniform" knot placement.

  • link (str) – "logit" or "identity". See calibre.SplineCalibrator.

  • cv (int) – Number of cross-validation folds used when a hyperparameter is left at "auto". Ignored when every hyperparameter is pinned.

  • scoring (str) – Proper scoring rule the "auto" search minimises. Deliberately not a calibration error: ECE and its relatives are minimised by a constant forecast, so selecting on one would reward throwing resolution away.

  • random_state (int | None) – Seed for the cross-validation split, so an "auto" selection is reproducible.

  • clip_output (bool) – Clip calibrated values into [0, 1].

  • enable_diagnostics (bool) – Whether to enable plateau diagnostics analysis.

basis_

The fitted basis.

intercept_

Fitted intercept, on the link scale.

coef_

Fitted non-negative increment coefficients.

n_features_in_

Always 1. Present for scikit-learn compatibility.

Notes

The penalty is on curvature, not on magnitude. A ridge penalty \(\alpha\sum_i \beta_i^2\) buys no smoothness at all: unconstrained its solution is \(\beta = y/(1+\alpha)\), a uniform deflation of every probability that breaks mean calibration by construction and drives all predictions to zero as \(\alpha\) grows. A second-difference penalty leaves any straight line unpenalised, so the identity map and the empirical base rate both survive it.

Why a fixed basis rather than one parameter per score. Putting a parameter at every unique score makes this a smoothing-spline problem whose penalty operator scales like \(h^{-2} \sim n^{2}\), so the normal equations scale like \(n^{4}\). That is ill-conditioned in a way no solver choice repairs – a constrained QP stops converging above a few thousand distinct scores, ADMM diverges, and a matrix-free least-squares solve fails to converge while the fitted mean collapses away from the base rate. A modest fixed basis with a coefficient penalty – the P-spline construction of Eilers & Marx (1996), as used by the SCOP-splines of Pya & Wood (2015) – has none of those regimes: it fits 100,000 points in milliseconds with monotonicity guaranteed structurally.

Note

alpha=0 no longer reduces to isotonic regression. It gives an unpenalised monotone regression spline, which is smooth rather than piecewise constant. For the exact isotonic fit use calibre.IsotonicCalibrator; to remove isotonic’s plateaus without leaving the non-parametric family, use calibre.CenteredIsotonicCalibrator.

Examples

>>> import numpy as np
>>> from calibre import RegularizedIsotonicCalibrator
>>>
>>> rng = np.random.default_rng(0)
>>> x = rng.random(500)
>>> y = (rng.random(500) < x).astype(float)
>>>
>>> cal = RegularizedIsotonicCalibrator(alpha=1.0).fit(x, y)
>>> fitted = cal.transform(np.linspace(0, 1, 200))
>>> bool(np.all(np.diff(fitted) >= -1e-10))
True

See also

SplineCalibrator : Same estimator with the penalty chosen by cross-validation. CenteredIsotonicCalibrator : Non-parametric and plateau-free. IsotonicCalibrator : The exact isotonic fit.

ALPHA_GRID = (0.0, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0)

Candidate roughness penalties searched when alpha="auto". Matches the grid SplineCalibrator has always used for the same parameter.

transform(X)[source]

Map scores through the fitted calibration curve.

Parameters:

X (ndarray) – Scores to calibrate.

Returns:

Calibrated probabilities.

Return type:

ndarray of shape (n_samples,)

Raises:

AttributeError – If called before fit().

set_fit_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

RegularizedIsotonicCalibrator

Note

This is a monotone spline with a second-difference (curvature) penalty. It is not ridge regression, and alpha=0 is not isotonic regression.

Relaxed PAVA Calibrator

class calibre.RelaxedPAVACalibrator(epsilon='auto', min_slope='auto', cv=5, scoring='log_loss', random_state=0, clip_output=True, enable_diagnostics=False)[source]

Bases: BaseCalibrator

Isotonic regression with a lower bound on each adjacent increment.

Solves

\[\min_{z} \sum_i w_i (y_i - z_i)^2 \quad\text{s.t.}\quad z_{i+1} - z_i \ge L_i\]

in O(n) via the cumulative-shift reduction (see calibre._core.shift_to_pava()): substituting \(u_i = z_i - \sum_{j<i} L_j\) turns the constraint into \(u_{i+1} \ge u_i\), so one weighted PAVA on the shifted targets solves it exactly.

One signed bound spans three estimators:

epsilon = 0

standard isotonic regression

epsilon > 0

epsilon-monotone: decreases up to epsilon allowed

min_slope > 0

strictly increasing, so no plateau can form at all
Parameters:

  • epsilon (float | str) – Largest decrease permitted between adjacent unique scores, in the units of y. So epsilon=0.02 means “tolerate a drop of up to 2 percentage points”.

  • min_slope (float | str) – Minimum required increase between adjacent unique scores. Mutually exclusive with a non-zero epsilon; this is the direction that eliminates plateaus. "auto" (the default) uses 0.01 / n_unique, but only on the untouched default path – that is, when epsilon was also left at "auto" and the search settled on 0. Naming epsilon yourself, including epsilon=0, leaves the slope at 0 and the estimator exactly as documented in the table above.

  • cv (int) – Number of cross-validation folds used when a hyperparameter is left at "auto". Ignored when every hyperparameter is pinned.

  • scoring (str) – Proper scoring rule the "auto" search minimises. Deliberately not a calibration error: ECE and its relatives are minimised by a constant forecast, so selecting on one would reward throwing resolution away.

  • random_state (int | None) – Seed for the cross-validation split, so an "auto" selection is reproducible.

  • clip_output (bool) – Clip calibrated values into [0, 1].

  • enable_diagnostics (bool) – Whether to enable plateau diagnostics analysis.

calibration_curve_

The fitted calibration map.

n_features_in_

Always 1. Present for scikit-learn compatibility.

Notes

epsilon is an absolute tolerance on the target scale, deliberately. An earlier version of this class derived its threshold as a percentile of |diff(y)| over the score-sorted targets, which cannot work for this package’s primary use case: with binary labels those differences are all 0 or 1, so any percentile collapses to either 0 – the relaxation never binds and the estimator is silently just PAVA – or 1, where it never constrains anything. There is no intermediate setting to choose.

Relaxing monotonicity is not free: a decrease in the calibration map reverses the ranking of every score pair it spans, which costs discrimination. To preserve granularity, min_slope is usually the better direction, since it removes plateaus while keeping the map strictly increasing.

That is why the default is a slope rather than nothing. PAVA’s plateaus are an artefact of pooling adjacent violators, not a finding about the data, and at min_slope=0 this estimator keeps only 1-4% of the input’s distinct values. A slope small enough to be invisible in the score recovers almost all of them: measured on logit-inflated designs at n from 300 to 3000, the default retains 80-95% of distinct values for a Brier cost in the fifth decimal. It is 80-95% rather than all of them because clip_output flattens the two ends of a fit that saturates 0 and 1; the plateaus that survive the default are at the boundaries, not in the interior. It scales as 1 / n_unique because a fixed slope safe at n=1000 would need an output range of 10 at n=1e6, and clipping would flatten it back into the plateaus it exists to prevent.

Examples

>>> import numpy as np
>>> from calibre import RelaxedPAVACalibrator
>>>
>>> x = np.array([0.1, 0.2, 0.3, 0.4, 0.5])
>>> y = np.array([0, 0, 1, 0, 1])
>>>
>>> RelaxedPAVACalibrator(epsilon=0.0).fit_transform(x, y)
array([0. , 0. , 0.5, 0.5, 1. ])

Left alone, the default breaks that tie apart rather than reporting two scores as indistinguishable:

>>> default = RelaxedPAVACalibrator().fit_transform(x, y)
>>> bool(np.all(np.diff(default) > 0))
True

A minimum slope leaves no plateau anywhere:

>>> fitted = RelaxedPAVACalibrator(min_slope=0.05).fit_transform(x, y)
>>> bool(np.all(np.diff(fitted) > 0))
True

The bound itself is exact only without clipping. Clipping into [0, 1] can shorten the increments that straddle a boundary, so the guarantee degrades from “>= min_slope” to “> 0” there:

>>> exact = RelaxedPAVACalibrator(
...     min_slope=0.05, clip_output=False
... ).fit_transform(x, y)
>>> bool(np.all(np.diff(exact) >= 0.05 - 1e-12))
True
>>> float(exact.min())                      # below 0, hence the clipping
-0.025

See also

IsotonicCalibrator : The epsilon = 0 special case. CenteredIsotonicCalibrator : Removes plateaus without relaxing monotonicity. NearlyIsotonicCalibrator : Penalises violations instead of bounding them.

EPSILON_GRID = (0.0, 0.001, 0.005, 0.01, 0.02, 0.05, 0.1)

Candidate tolerances searched when epsilon="auto". 0.0 is included so selection can return strict isotonic regression when that fits best.

transform(X)[source]

Map scores through the fitted calibration curve.

Parameters:

X (ndarray) – Scores to calibrate.

Returns:

Calibrated values.

Return type:

ndarray of shape (n_samples,)

Raises:

AttributeError – If called before fit().

set_fit_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

RelaxedPAVACalibrator

Bounds each adjacent increment: epsilon permits small decreases, while min_slope forbids plateaus outright. Solved by shift-to-PAVA in O(n).

Nearly Isotonic Calibrator

class calibre.NearlyIsotonicCalibrator(lam='auto', method='path', cv=5, scoring='log_loss', random_state=0, clip_output=True, enable_diagnostics=False)[source]

Bases: BaseCalibrator

Nearly-isotonic regression for flexible monotonic calibration.

This calibrator implements nearly-isotonic regression, which relaxes the strict monotonicity constraint of standard isotonic regression by penalizing rather than prohibiting violations. This allows for a more flexible fit while still maintaining a generally monotonic trend.

Parameters:

  • lam (float | str) – Regularization parameter controlling the strength of monotonicity constraint. Higher values enforce stricter monotonicity.

  • method (str) –

    Solver for the optimization problem. Both are exact and agree to solver tolerance; path is the faster and needs no CVXPY.

    • 'path': the exact solution path (O(n log n)).

    • 'cvx': convex optimization via CVXPY.

  • cv (int) – Number of cross-validation folds used when a hyperparameter is left at "auto". Ignored when every hyperparameter is pinned.

  • scoring (str) – Proper scoring rule the "auto" search minimises. Deliberately not a calibration error: ECE and its relatives are minimised by a constant forecast, so selecting on one would reward throwing resolution away.

  • random_state (int | None) – Seed for the cross-validation split, so an "auto" selection is reproducible.

  • clip_output (bool) – Clip calibrated values into [0, 1]. Appropriate for probability calibration; turn it off to recover the unconstrained optimum of the objective above, which is what the estimator is actually defined as.

  • enable_diagnostics (bool) – Whether to enable plateau diagnostics analysis.

Notes

Nearly-isotonic regression solves the following optimization problem:

\[\min_{\beta} \sum_{i=1}^{n} (y_i - \beta_i)^2 + \lambda \sum_{i=1}^{n-1} \max(0, \beta_i - \beta_{i+1})\]

where \(\beta\) is the calibrated output, \(y\) are the true labels, and \(\lambda > 0\) controls the strength of the monotonicity penalty.

This formulation penalizes violations of monotonicity proportionally to their magnitude, allowing small violations when they significantly improve the fit.

Interpreting lam. Read it as a bias-variance knob on pooling rather than as permission for non-monotone structure: lam = 0 returns the data untouched, lam -> inf returns the isotonic fit, and intermediate values give shorter plateaus than isotonic regression – finer granularity – in exchange for bounded violations.

This is not the calibrator to reach for if you want granularity. The granularity above is real but it is not free, and here it is not even cheap. Because the objective fits one value per observation to the labels, a small lam returns something close to the raw 0/1 labels: lots of distinct values, all of them overfitted. Measured out of sample on a logit-inflated design at n=3000, lam=0.001 keeps 1074 distinct values at a held-out Brier of 0.191 against isotonic’s 0.116, and every step up the lam grid buys score back by giving granularity away until, by lam=100, the fit is isotonic regression.

That frontier is dominated. On the same data CenteredIsotonicCalibrator keeps 2647 distinct values at a held-out Brier of 0.1159 – more granularity than any lam reaches, at a better score than isotonic. So there is no default lam worth moving to, and none is claimed: unlike RelaxedPAVACalibrator, whose default now breaks plateaus apart at a cost in the fifth decimal, this estimator’s defaults leave it close to isotonic on purpose. Use it when you want bounded monotonicity violations – the thing it uniquely provides – and use CIR or the spline calibrators when you want resolution.

Scaling differs from the source paper. Tibshirani, Hoefling & Tibshirani (2011, Technometrics 53(1), 54-61) put a factor of 1/2 on the squared-error term:

\[\min_{\beta} \tfrac{1}{2} \sum_i (y_i - \beta_i)^2 + \lambda_{\text{paper}} \sum_i \max(0, \beta_i - \beta_{i+1})\]

The objective above omits it, so lam here is twice the paper’s \(\lambda\):

\[\lambda_{\text{here}} = 2\,\lambda_{\text{paper}}\]

Double any penalty value taken from the paper before passing it in. Both solvers are pinned against the authors’ R implementation (neariso) in tests/test_r_reference.py.

Examples

>>> import numpy as np
>>> from calibre import NearlyIsotonicCalibrator
>>>
>>> X = np.array([0.1, 0.2, 0.3, 0.4, 0.5])
>>> y = np.array([0.12, 0.18, 0.35, 0.25, 0.55])
>>>
>>> cal = NearlyIsotonicCalibrator(lam=0.5)
>>> _ = cal.fit(X, y)
>>> X_calibrated = cal.transform(np.array([0.15, 0.35, 0.55]))

See also

IsotonicCalibrator : Strict monotonicity constraint RegularizedIsotonicCalibrator : L2 regularization with strict monotonicity

LAM_GRID = (0.01, 0.1, 0.5, 1.0, 5.0, 10.0, 50.0, 100.0)

Candidate lambdas searched when lam="auto". Spans “essentially the raw data” to “essentially isotonic”, logarithmically.

transform(X)[source]

Map scores through the fitted calibration curve.

Parameters:

X (ndarray) – The values to be calibrated.

Returns:

Calibrated values.

Return type:

X_calibrated

Raises:

AttributeError – If called before fit().

set_fit_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

NearlyIsotonicCalibrator

Note

Penalises rather than forbids monotonicity violations. Two exact solvers: method="path" (default, pure NumPy) and method="cvx" (CVXPY). Note that lam is twice the source paper’s lambda.

Smoothed Isotonic Calibrator

class calibre.SmoothedIsotonicCalibrator(window_length=None, poly_order=3, adaptive=False, min_window=5, max_window=None, enable_diagnostics=False)[source]

Bases: BaseCalibrator, MonotonicMixin

Isotonic regression with Savitzky-Golay smoothing.

Fits weighted PAVA on the pooled unique scores, smooths the fitted values, restores monotonicity with a running maximum, and interpolates linearly between the resulting knots.

Parameters:

  • window_length (int | None) – Window length for the Savitzky-Golay filter, in distinct scores. Forced odd and capped at the number of distinct scores. If None, uses max(5, n_distinct // 10).

  • poly_order (int) – Polynomial order for the filter. Values below 1 are raised to 1.

  • adaptive (bool) – Size the window per point from local density instead of using one fixed window.

  • min_window (int) – Minimum window length when adaptive=True. Values below 3 are raised to 3.

  • max_window (int | None) – Maximum window length when adaptive=True. If None, uses n_distinct // 5.

  • enable_diagnostics (bool) – Run plateau diagnostics after fitting.

calibration_curve_

The fitted calibration map, on the distinct training scores.

poly_order_

poly_order after validation.

min_window_

min_window after validation.

n_features_in_

Always 1. Present for scikit-learn compatibility.

Notes

Window lengths count distinct scores, not observations. Tied scores are pooled before smoothing, because a filter applied across repeated abscissae smooths over points that carry no separate information, and an interpolant cannot be built on repeated abscissae at all.

This estimator does not preserve granularity well. Restoring monotonicity with a running maximum re-flattens the curve wherever the filter introduced a dip, so plateaus come back: on the package’s test datasets it retains roughly 13-16% of the distinct input values, against 100% for CenteredIsotonicCalibrator. If granularity is why you are here, use that instead.

Examples

>>> import numpy as np
>>> from calibre import SmoothedIsotonicCalibrator
>>>
>>> X = np.array([0.1, 0.2, 0.3, 0.4, 0.5])
>>> y = np.array([0.12, 0.18, 0.35, 0.25, 0.55])
>>>
>>> cal = SmoothedIsotonicCalibrator(window_length=7)
>>> _ = cal.fit(X, y)
>>> p = cal.transform(np.array([0.15, 0.45]))
>>> bool(p[0] <= p[1])
True

See also

IsotonicCalibrator : Isotonic regression without smoothing. CenteredIsotonicCalibrator : Smooth by construction rather than by repair.

transform(X)[source]

Map scores through the fitted calibration curve.

Parameters:

X (ndarray) – Scores to calibrate.

Returns:

Calibrated probabilities.

Return type:

ndarray of shape (n_samples,)

Raises:

AttributeError – If called before fit().

set_fit_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

SmoothedIsotonicCalibrator

Note

Savitzky-Golay smoothing of an isotonic fit. Retained for compatibility; prefer SplineCalibrator or RegularizedIsotonicCalibrator for a smooth curve.

Research

Cost- and Data-Informed Isotonic Calibrator

class calibre.CDIIsotonicCalibrator(thresholds=None, threshold_weights=None, bandwidth=0.05, alpha=0.05, gamma=0.15, window=25, normalize_scores=True, clip_output=True)[source]

Bases: BaseEstimator, TransformerMixin

Cost- and Data-Informed Isotonic calibrator (CDI-ISO).

Parameters:

  • thresholds (Iterable[float] | None) – Operating thresholds in [0,1] that matter economically. If None, uniform attention across the score range is assumed.

  • threshold_weights (Iterable[float] | None) – Nonnegative weights matching thresholds. If None, equal weights.

  • bandwidth (float) – Half-width h of the triangular kernel around each threshold (in score units, after optional min-max normalization). Defaults to 0.05.

  • alpha (float) – Significance level for the two-proportion normal approximation used to gate minimum-slope enforcement (default 0.05 -> z≈1.96).

  • gamma (float) – Global multiplier in [0,1] for the minimum-slope budget phi_i (default 0.15).

  • window (int) – Number of adjacent unique-score points used on each side to form the left/right evidence blocks (default 25). Automatically clipped at edges.

  • normalize_scores (bool) – If True (default), min-max normalize training scores to [0,1] for the economics kernel; the same affine scaling is applied at transform time.

  • clip_output (bool) – If True (default), clip calibrated outputs to [0,1].

Notes

  • Builds local bounds L_i = phi_i - epsilon_i on sorted unique training scores.

  • Solves a single weighted PAVA on shifted labels (O(n)) and shifts back.

  • Predictions are stepwise-constant in the training score order.

thresholds: Iterable[float] | None = None
threshold_weights: Iterable[float] | None = None
bandwidth: float = 0.05
alpha: float = 0.05
gamma: float = 0.15
window: int = 25
normalize_scores: bool = True
clip_output: bool = True
fit(scores, y, sample_weight=None)[source]

Fit CDI-ISO on (scores, y).

Parameters:

  • scores (ndarray) – Raw model scores; will be sorted internally. If normalize_scores=True, an affine min-max transform to [0,1] is learned and applied in transform.

  • y (ndarray) – Binary labels {0,1}.

  • sample_weight (ndarray | None) – Nonnegative per-sample weights.

Returns:

Returns self for method chaining.

Raises:

ValueError – If scores and y have different lengths, y contains invalid values, or sample_weight has invalid values.

Return type:

CDIIsotonicCalibrator

transform(scores)[source]

Map new scores to calibrated probabilities (stepwise-constant).

Parameters:

scores (ndarray) – Input scores to calibrate.

Returns:

Calibrated probabilities in [0,1] (if clip_output=True).

Raises:

RuntimeError – If called before fit().

Return type:

ndarray

adjacency_bounds_()[source]

Return the learned local bounds L_i per adjacency (shape: m-1).

Returns None if not fitted.

Return type:

ndarray | None

cumulative_shift_()[source]

Return the cumulative shift R_i (shape: m) or None if not fitted.

Return type:

ndarray | None

breakpoints_()[source]

Return (unique_scores, calibrated_values) on the training grid.

Return type:

tuple[ndarray, ndarray] | None

set_fit_request(*, sample_weight='$UNCHANGED$', scores='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

scoresstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for scores parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

CDIIsotonicCalibrator

set_transform_request(*, scores='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the transform method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to transform if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to transform.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

scoresstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for scores parameter in transform.

Returns

selfobject

The updated object.

Parameters:
Return type:

CDIIsotonicCalibrator

Note

CDI-ISO is research-grade. It uses economic decision theory and statistical evidence to decide where monotonicity should be enforced strictly, and requires you to specify the operating thresholds where discrimination matters most.

Usage Examples

Basic Example

import numpy as np

from calibre import CenteredIsotonicCalibrator

rng = np.random.default_rng(42)
X = rng.uniform(0, 1, 1000)
y = rng.binomial(1, X).astype(float)

calibrator = CenteredIsotonicCalibrator().fit(X, y)

X_new = rng.uniform(0, 1, 100)
y_calibrated = calibrator.transform(X_new)

Warning

Always fit the calibrator on data the model did not train on. A model’s scores on its own training data are already too good, so a calibrator fitted there learns the wrong correction. Use a held-out split, or cross_val_calibrate() for out-of-fold predictions.

Comparing Methods

import numpy as np

from calibre import (
    CenteredIsotonicCalibrator,
    IsotonicCalibrator,
    RegularizedIsotonicCalibrator,
    RelaxedPAVACalibrator,
    SplineCalibrator,
    unique_value_counts,
)

calibrators = {
    "Isotonic": IsotonicCalibrator(),
    "Centered": CenteredIsotonicCalibrator(),
    "Spline": SplineCalibrator(),
    "Relaxed PAVA": RelaxedPAVACalibrator(min_slope=1e-5),
    "Regularized": RegularizedIsotonicCalibrator(alpha=0.1),
}

for name, cal in calibrators.items():
    out = cal.fit(X, y).transform(X)
    n = unique_value_counts(out)["n_unique_y_pred"]
    print(f"{name:14s} {n:5d} distinct values")

CDI-ISO Usage Example

import numpy as np

from calibre import CDIIsotonicCalibrator

cdi_cal = CDIIsotonicCalibrator(
    thresholds=[0.3, 0.7],           # operating decision thresholds
    threshold_weights=[0.6, 0.4],    # relative importance
    bandwidth=0.1,                   # kernel bandwidth around thresholds
    gamma=0.2,                       # minimum slope strength
    alpha=0.05,                      # significance level
    window=30,                       # evidence window size
)
cdi_cal.fit(X, y)
y_calibrated = cdi_cal.transform(X_new)

bounds = cdi_cal.adjacency_bounds_()
breakpoints = cdi_cal.breakpoints_()

print(f"CDI calibrator learned {len(bounds)} local bounds")
print(f"Calibration function has {len(breakpoints[0])} breakpoints")