Calibration Methods

Every calibrator follows the scikit-learn transformer API: .fit(scores, labels) and .transform(scores), plus sample_weight where it is meaningful. All are binary; for multiclass see Multiclass Calibration.

Which calibrator should I use?

If you don’t want to think about it: CenteredIsotonicCalibrator. It is non-parametric, has nothing to tune, and preserves score ordering between pooled isotonic blocks.

You want

Use

Notes

A drop-in isotonic replacement, no tuning

CenteredIsotonicCalibrator

Collapses isotonic’s flat steps to points and interpolates. O(n).

A smooth curve, and you can afford cross-validation

SplineCalibrator

Monotone spline; picks its own smoothing by link-appropriate prediction loss.

A smooth curve with smoothing you control

SplineCalibrator

Set alpha; also set n_knots to skip cross-validation.

Exactly scikit-learn’s isotonic behavior

IsotonicCalibrator

Thin wrapper, plus optional plateau diagnostics.

A fitted-value change bound you control

RelaxedPAVACalibrator

Negative permits bounded drops; positive forces a step before clipping.

To allow small ranking violations if they fit better

NearlyIsotonicCalibrator

lam trades monotonicity against fit.

Accuracy near specific decision thresholds

CDIIsotonicCalibrator

Research-grade; requires operating thresholds and can permit bounded decreases away from them.

Base Classes

class calibre.BaseCalibrator(*, enable_diagnostics=False)[source]

Bases: BaseEstimator, TransformerMixin

Base class for all calibrators.

All calibrator classes should inherit from this base class to ensure consistent API and functionality. This follows the scikit-learn transformer interface with fit/transform/fit_transform methods.

Parameters:

enable_diagnostics (bool) – Whether to run plateau diagnostics after fitting.

Notes

Subclasses implement _fit_impl() and transform(). The shared fit() and fit_transform() methods validate and retain the fitting data, maintain fitted state, and run diagnostics.

Examples

>>> import numpy as np
>>> from calibre import BaseCalibrator
>>>
>>> class SimpleCalibrator(BaseCalibrator):
...     def __init__(self, enable_diagnostics=False):
...         super().__init__(enable_diagnostics=enable_diagnostics)
...     def _fit_impl(self, X, y, sample_weight=None):
...         self.mean_ = np.mean(y)
...
...     def transform(self, X):
...         return np.full_like(X, self.mean_)
>>>
>>> X = np.array([0.1, 0.3, 0.5])
>>> y = np.array([0, 1, 1])
>>>
>>> cal = SimpleCalibrator()
>>> _ = cal.fit(X, y)
>>> cal.transform(X)
array([0.66666667, 0.66666667, 0.66666667])
fit(X, y, sample_weight=None)[source]

Fit the calibrator.

This method implements the template method pattern: it handles data storage and diagnostics, while delegating the actual fitting logic to the abstract _fit_impl() method that subclasses must implement.

Parameters:

  • X (ndarray) – The values to be calibrated (e.g., predicted probabilities).

  • y (ndarray) – The target values (e.g., true labels).

  • sample_weight (ndarray | None) – Non-negative per-observation weights. Calibrators that cannot honour weights raise rather than ignore them.

Returns:

Returns self for method chaining.

Return type:

BaseCalibrator

Raises:

ValueError – If sample_weight is not one-dimensional, does not have the same shape as y, contains a non-finite or negative value, or has no positive mass.

transform(X)[source]

Apply calibration to new data.

Parameters:

X (ndarray) – The values to be calibrated.

Returns:

Calibrated values.

Raises:

NotImplementedError – This method must be implemented by subclasses.

Return type:

ndarray

fit_transform(X, y, sample_weight=None)[source]

Fit the calibrator and then transform the data.

This is a convenience method that combines fit() and transform() in a single call. The default implementation simply calls fit() followed by transform().

Parameters:

  • X (ndarray) – The values to be calibrated.

  • y (ndarray) – The target values.

  • sample_weight (ndarray | None) – Non-negative per-observation weights.

Returns:

Calibrated values.

Return type:

ndarray

Examples

>>> import numpy as np
>>> from calibre import IsotonicCalibrator
>>> X = np.array([0.1, 0.3, 0.5, 0.7, 0.9])
>>> y = np.array([0, 0, 1, 1, 1])
>>> cal = IsotonicCalibrator()
>>> X_calibrated = cal.fit_transform(X, y)
has_diagnostics()[source]

Check if diagnostic information is available.

Returns:

True if diagnostics have been computed and are available.

Return type:

has_diag

Examples

>>> from calibre import IsotonicCalibrator
>>> import numpy as np
>>>
>>> X = np.array([0.1, 0.3, 0.5])
>>> y = np.array([0, 1, 1])
>>>
>>> cal = IsotonicCalibrator(enable_diagnostics=True)
>>> _ = cal.fit(X, y)
>>> cal.has_diagnostics()
True
get_diagnostics()[source]

Get diagnostic results.

Returns:

Diagnostic results from plateau analysis, or None if

diagnostics were not computed or are not available.

Return type:

dict | None

Examples

>>> from calibre import IsotonicCalibrator
>>> import numpy as np
>>>
>>> X = np.array([0.1, 0.3, 0.5])
>>> y = np.array([0, 1, 1])
>>>
>>> cal = IsotonicCalibrator(enable_diagnostics=True)
>>> _ = cal.fit(X, y)
>>> cal.get_diagnostics()["n_plateaus"]
1
diagnostic_summary()[source]

Get a human-readable summary of diagnostic analysis.

Returns:

Human-readable plateau summary.

Return type:

summary

Examples

>>> from calibre import IsotonicCalibrator
>>> import numpy as np
>>>
>>> X = np.array([0.1, 0.3, 0.5, 0.7, 0.9])
>>> y = np.array([0, 0, 1, 1, 1])
>>>
>>> cal = IsotonicCalibrator(enable_diagnostics=True)
>>> _ = cal.fit(X, y)
>>> print(cal.diagnostic_summary())
Detected 2 plateau(s):

Warnings:
... Plateau 1 at [0.100, 0.300] has only 2 observations - may be unreliable
... Plateau 2 at [0.500, 0.900] has only 3 observations - may be unreliable
set_fit_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

BaseCalibrator

class calibre.MonotonicMixin[source]

Mixin for calibrators that maintain monotonicity.

This mixin provides utility methods for calibrators that aim to preserve or enforce monotonic relationships between inputs and outputs.

check_monotonicity(y)[source]

Check if an array is monotonically increasing.

Parameters:
Return type:

bool

enforce_monotonicity(y)[source]

Enforce monotonicity on an array.

Parameters:
Return type:

ndarray

Notes

This is a utility mixin that doesn’t require any specific attributes. It’s designed to be mixed in with BaseCalibrator subclasses that need monotonicity guarantees.

static check_monotonicity(y, strict=False)[source]

Check if an array is monotonically increasing.

Parameters:

  • y (ndarray) – Values to check for monotonicity.

  • strict (bool) – If True, check for strictly increasing (no equal consecutive values). If False, check for non-decreasing (allows equal consecutive values).

Returns:

True if the array is monotonic according to the specified criteria.

Return type:

bool

Examples

>>> import numpy as np
>>> from calibre.base import MonotonicMixin
>>>
>>> y1 = np.array([0.1, 0.2, 0.3, 0.4])
>>> MonotonicMixin.check_monotonicity(y1)
True
>>>
>>> y2 = np.array([0.1, 0.3, 0.2, 0.4])
>>> MonotonicMixin.check_monotonicity(y2)
False
>>>
>>> y3 = np.array([0.1, 0.2, 0.2, 0.3])
>>> MonotonicMixin.check_monotonicity(y3, strict=False)
True
>>> MonotonicMixin.check_monotonicity(y3, strict=True)
False
static enforce_monotonicity(y, inplace=False)[source]

Enforce monotonicity on an array.

This method ensures the array is non-decreasing by replacing any value that is less than the previous value with the previous value.

Parameters:

  • y (ndarray) – Values to make monotonic.

  • inplace (bool) – If True, modify the array in place. Otherwise, return a copy.

Returns:

Monotonically increasing version of the input array.

Return type:

ndarray

Examples

>>> import numpy as np
>>> from calibre.base import MonotonicMixin
>>>
>>> y = np.array([0.1, 0.3, 0.2, 0.5, 0.4])
>>> y_mono = MonotonicMixin.enforce_monotonicity(y)
>>> print(y_mono)
[0.1 0.3 0.3 0.5 0.5]
>>>
>>> # Original array unchanged
>>> print(y)
[0.1 0.3 0.2 0.5 0.4]

Other Calibrators

Isotonic Calibrator

class calibre.IsotonicCalibrator(*, y_min=None, y_max=None, increasing=True, out_of_bounds='clip', enable_diagnostics=False)[source]

Bases: BaseCalibrator

Isotonic regression calibrator.

This calibrator wraps sklearn’s IsotonicRegression for probability calibration.

Parameters:

  • y_min (float | None) – Lower bound for the calibrated values.

  • y_max (float | None) – Upper bound for the calibrated values.

  • increasing (bool) – Whether the calibration function should be increasing.

  • out_of_bounds (str) – How to handle out-of-bounds values in transform. Options: ‘nan’, ‘clip’, ‘raise’.

  • enable_diagnostics (bool) – Whether to enable plateau diagnostics analysis.

Examples

>>> import numpy as np
>>> from calibre import IsotonicCalibrator
>>>
>>> X = np.array([0.1, 0.2, 0.3, 0.4, 0.5])
>>> y = np.array([0, 0, 1, 1, 1])
>>>
>>> # Basic usage
>>> cal = IsotonicCalibrator()
>>> _ = cal.fit(X, y)
>>> X_calibrated = cal.transform(X)
>>>
>>> # With diagnostics
>>> cal = IsotonicCalibrator(enable_diagnostics=True)
>>> _ = cal.fit(X, y)
>>> cal.has_diagnostics()
True
>>> cal.get_diagnostics()["n_plateaus"]
2

Notes

Isotonic regression finds the best monotonic fit to the data, which is particularly useful for calibration because well-calibrated predictions should maintain the rank order of predictions while improving probability estimates.

See also

NearlyIsotonicCalibrator : Relaxed monotonicity constraint SplineCalibrator : Smooth monotone spline calibration

isotonic_: IsotonicRegression
transform(X)[source]

Apply isotonic calibration to new data.

Parameters:

X (ndarray) – The values to be calibrated.

Returns:

Calibrated values.

Return type:

ndarray

set_fit_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

IsotonicCalibrator

Spline Calibrator

class calibre.SplineCalibrator(*, n_knots='auto', degree=3, knots='quantile', alpha='auto', link='logit', cv=5, scoring='auto', max_cv_samples=20000, random_state=0, clip_output=True, enable_diagnostics=False)[source]

Bases: BaseCalibrator

Monotone spline calibration with cross-validated smoothing.

Fits

\[g\big(f(x)\big) = \theta + \sum_k \delta_k I_k(x), \qquad \delta_k \ge 0\]

where the \(I_k\) are I-splines (each non-decreasing) and \(g\) is the link. Because every basis function is non-decreasing and every coefficient is non-negative, \(f\) is non-decreasing by construction; the link is increasing, so the calibrated probability is too.

Parameters:

  • n_knots (int | str) – Number of knots, or "auto" to select it by cross-validation. The basis has n_knots + degree - 1 functions.

  • degree (int) – B-spline degree. 3 gives the usual cubic behavior.

  • knots (str) – "quantile" (default) places knots at score quantiles; "uniform" spaces them evenly. Quantile is normally right for calibration, where scores pile up wherever the base model is confident and uniform knots spend resolution on empty regions.

  • alpha (float | str) – Roughness penalty on the coefficient increments, or "auto" to select it by cross-validation. A number pins it exactly. The loss is weight-normalized, so alpha does not change meaning with sample size or a constant rescaling of observation weights.

  • link (str) – "logit" (default) fits a penalized Bernoulli likelihood: log-loss is the proper score for binary labels, and predictions land in (0, 1) with no clipping. "identity" fits penalized least squares on the probability scale – a single bounded linear solve.

  • cv (int) – Number of cross-validation folds. Stratified when y is binary.

  • scoring (str) – Proper scoring rule minimized during automatic selection. "auto" uses log loss for the logit link and squared error for the identity link.

  • max_cv_samples (int | None) – Cap on the number of observations used for hyperparameter selection. The final model is always refit on the full sample; this only bounds the cost of the search, which would otherwise fit the grid once per fold over every row (at n=100k that is ~50s against 0.3s for a single fit). Selecting two scalars from a large random subsample costs essentially nothing statistically. Set to None to search on all of the data.

  • random_state (int | None) – Seed for the cross-validation split. Defaults to 0 so that fit is reproducible: cross-validation here only selects a hyperparameter, and a fit that silently returns a different curve on each identical call is a trap. Pass None to draw the split from the global RNG instead.

  • clip_output (bool) – Clip calibrated values into [0, 1]. A no-op for link="logit".

  • enable_diagnostics (bool) – Whether to enable plateau diagnostics analysis.

basis_

The fitted basis. Its knots come from the same fit that produced coef_.

intercept_

Fitted intercept, on the link scale.

coef_

Fitted non-negative increment coefficients.

alpha_

The penalty actually used – selected by cross-validation, or echoed back from alpha.

n_knots_

The knot count actually used.

n_features_in_

Always 1. Present for scikit-learn compatibility.

Notes

Non-negative coefficients on a plain B-spline basis do not give monotonicity. B-spline basis functions are bumps, so a non-negative combination of them is a non-negative function and nothing more – a single non-negative coefficient already traces a curve that rises and then falls. Monotonicity requires non-negativity on the coefficient differences, which is exactly what the I-spline (cumulative) basis encodes; see calibre._core.MonotoneSplineBasis. The basis follows Ramsay (1988), while the coefficient-difference penalty follows Eilers & Marx’s (1996) P-splines. Pya & Wood’s (2015) SCOP-splines in R’s scam are a related reference model, not the same fitting algorithm.

Cross-validation selects a hyperparameter and then refits on all the data. It is not a search for whichever fold’s model scored best on its own validation split: that selects on noise and ships a model trained on only (cv-1)/cv of the sample. Logit-link fits are scored by log-loss; identity-link fits are scored by squared error, the loss they actually minimise.

Examples

>>> import numpy as np
>>> from calibre import SplineCalibrator
>>>
>>> rng = np.random.default_rng(0)
>>> x = rng.random(500)
>>> y = (rng.random(500) < x).astype(float)
>>>
>>> cal = SplineCalibrator(alpha=0.1).fit(x, y)
>>> fitted = cal.transform(np.linspace(0, 1, 200))
>>> bool(np.all(np.diff(fitted) >= -1e-10))     # monotone by construction
True
>>> bool(fitted.min() >= 0.0 and fitted.max() <= 1.0)
True

See also

CenteredIsotonicCalibrator : Non-parametric interpolation, without tuning. IsotonicCalibrator : Piecewise-constant non-parametric calibration.

ALPHA_GRID = (0.0, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0)
N_KNOTS_GRID = (5, 10, 20)
transform(X)[source]

Map scores through the fitted calibration curve.

Parameters:

X (ndarray) – Scores to calibrate.

Returns:

Calibrated probabilities.

Return type:

ndarray of shape (n_samples,)

calibration_curve(n_points=200)[source]

Sample the fitted map onto a grid, for plotting or inspection.

Parameters:

n_points (int) – Number of grid points across the fitted score range.

Returns:

The sampled curve.

Return type:

PiecewiseLinear

Raises:

AttributeError – If called before fit().

set_fit_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

SplineCalibrator

Note

This is a monotone spline with a second-difference (curvature) penalty. It is not ridge regression, and alpha=0 is not isotonic regression.

Relaxed PAVA Calibrator

class calibre.RelaxedPAVACalibrator(min_increment, *, clip_output=True, enable_diagnostics=False)[source]

Bases: BaseCalibrator

Isotonic regression with a bound on each adjacent fitted increment.

On the sorted unique input scores, this estimator solves

\[\min_{z} \sum_i w_i (y_i - z_i)^2 \quad\text{s.t.}\quad z_{i+1} - z_i \ge L,\]

where min_increment is \(L\). Substituting \(u_i = z_i - iL\) reduces the problem to weighted isotonic regression, so a single PAVA solve gives the exact constrained least-squares fit.

The sign of the one constraint parameter determines the behavior:

min_increment == 0

ordinary isotonic regression

min_increment < 0

bounded decreases are permitted

min_increment > 0

adjacent fitted values must increase
Parameters:

  • min_increment (float) – Lower bound on the fitted-value change between adjacent unique scores. For example, -0.02 permits a decrease of at most two percentage points, while 0.02 requires an increase of at least two percentage points. There is no automatic default because the appropriate bound depends on the application and score grid.

  • clip_output (bool) – Clip calibrated values into [0, 1]. Clipping preserves non-positive increment bounds but can flatten a positive bound at the output boundaries.

  • enable_diagnostics (bool) – Whether to enable plateau diagnostics analysis.

min_increment_

Bound selected during fit().

calibration_curve_

Fitted piecewise-linear calibration map.

n_features_in_

Always 1. Present for scikit-learn compatibility.

Notes

The bound is measured per adjacent unique score, not per unit of input score. It therefore depends on the score grid. Set it explicitly only when that interpretation is appropriate.

Negative bounds relax monotonicity and can reverse the ranking of scores. Positive bounds preserve ranking but may push an unclipped fit outside the probability range. Select the bound using held-out data and a proper score.

The cumulative-shift construction is the total-order special case of the margin-separated isotonic projection described by Gunasekar, Koyejo, and Ghosh (2016). That result supports the optimization method, not a claim that any particular bound improves probability calibration.

Examples

>>> import numpy as np
>>> from calibre import RelaxedPAVACalibrator
>>> x = np.array([0.1, 0.2, 0.3, 0.4, 0.5])
>>> y = np.array([0, 0, 1, 0, 1])
>>> RelaxedPAVACalibrator(min_increment=0.0).fit_transform(x, y)
array([0. , 0. , 0.5, 0.5, 1. ])

A negative bound permits small fitted decreases:

>>> relaxed = RelaxedPAVACalibrator(
...     min_increment=-0.05, clip_output=False
... ).fit_transform(x, y)
>>> bool(np.all(np.diff(relaxed) >= -0.05 - 1e-12))
True

A positive bound separates adjacent fitted values before clipping:

>>> strict = RelaxedPAVACalibrator(
...     min_increment=0.05, clip_output=False
... ).fit_transform(x, y)
>>> bool(np.all(np.diff(strict) >= 0.05 - 1e-12))
True

See also

IsotonicCalibrator : The min_increment=0 special case. CenteredIsotonicCalibrator : Monotone interpolation between pooled blocks. NearlyIsotonicCalibrator : Penalises rather than bounds violations.

References

Gunasekar, S., Koyejo, O., & Ghosh, J. (2016). Preference completion from partial rankings. Advances in Neural Information Processing Systems 29.

transform(X)[source]

Map scores through the fitted calibration curve.

Parameters:

X (ndarray) – Scores to calibrate.

Returns:

Calibrated values.

Return type:

ndarray of shape (n_samples,)

set_fit_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

RelaxedPAVACalibrator

min_increment is the lower bound on each adjacent fitted-value change. Zero is ordinary isotonic regression, negative values permit bounded decreases, and positive values forbid plateaus before optional output clipping. The bound is required because its appropriate value depends on the application and score grid. The exact weighted least-squares solution uses a cumulative shift followed by PAVA.

Nearly Isotonic Calibrator

class calibre.NearlyIsotonicCalibrator(*, lam='auto', cv=5, scoring='auto', random_state=0, clip_output=True, enable_diagnostics=False)[source]

Bases: BaseCalibrator

Nearly-isotonic regression for flexible monotonic calibration.

This calibrator implements nearly-isotonic regression, which relaxes the strict monotonicity constraint of standard isotonic regression by penalizing rather than prohibiting violations. This allows for a more flexible fit while still maintaining a generally monotonic trend.

Parameters:

  • lam (float | str) – Regularization parameter controlling the strength of monotonicity constraint. Higher values enforce stricter monotonicity.

  • cv (int) – Number of cross-validation folds used when a hyperparameter is left at "auto". Ignored when every hyperparameter is pinned.

  • scoring (str) – Proper scoring rule the "auto" search minimizes. "auto" (the default) uses log loss for probability targets and squared error otherwise. Deliberately not a calibration error: ECE and its relatives are minimised by a constant forecast, so selecting on one would reward throwing resolution away.

  • random_state (int | None) – Seed for the cross-validation split, so an "auto" selection is reproducible.

  • clip_output (bool) – Clip calibrated values into [0, 1]. Appropriate for probability calibration; turn it off to recover the unconstrained optimum of the objective above, which is what the estimator is actually defined as.

  • enable_diagnostics (bool) – Whether to enable plateau diagnostics analysis.

Notes

Nearly-isotonic regression solves the following optimization problem:

\[\min_{\beta} \tfrac{1}{2}\sum_{i=1}^{n} w_i(y_i - \beta_i)^2 + \lambda \sum_{i=1}^{n-1} \max(0, \beta_i - \beta_{i+1})\]

where \(\beta\) is the calibrated output, \(y\) are the true labels, and \(\lambda > 0\) controls the strength of the monotonicity penalty.

This formulation penalizes violations of monotonicity proportionally to their magnitude, allowing small violations when they significantly improve the fit.

Interpreting lam. Read it as a bias-variance knob on pooling rather than as permission for non-monotone structure: lam = 0 returns the data untouched, lam -> inf returns the isotonic fit, and intermediate values give shorter plateaus than isotonic regression – finer granularity – in exchange for penalized violations.

This is not the calibrator to reach for if you only want granularity. Because the objective fits one value per observation to the labels, a small lam approaches the raw outcomes and overfits. Increasing lam buys proper-score performance back by pooling more values. The committed benchmark reports this tradeoff against centered isotonic regression using held-out proper scores and distinct-value counts. Use nearly-isotonic regression when penalized monotonicity violations are the feature you need. Use centered isotonic regression or the spline calibrator when you need granularity.

The objective, lambda scale, and modified PAVA path algorithm follow Tibshirani, Hoefling & Tibshirani (2011, Technometrics 53(1), 54-61). Cross-language fixtures in tests/test_r_reference.py compare the implementation with the authors’ R package, neariso.

Automatic selection keeps \(\lambda / \sum_i w_i\) fixed between each training fold and the final full-data fit. Numeric lam values remain absolute penalties on the paper’s scale.

Examples

>>> import numpy as np
>>> from calibre import NearlyIsotonicCalibrator
>>>
>>> X = np.array([0.1, 0.2, 0.3, 0.4, 0.5])
>>> y = np.array([0.12, 0.18, 0.35, 0.25, 0.55])
>>>
>>> cal = NearlyIsotonicCalibrator(lam=0.5)
>>> _ = cal.fit(X, y)
>>> X_calibrated = cal.transform(np.array([0.15, 0.35, 0.55]))

See also

IsotonicCalibrator : Strict monotonicity constraint SplineCalibrator : Penalized monotone spline calibration.

N_AUTO_LAMBDAS = 9

Number of data-scaled penalties searched when lam="auto". The upper endpoint is the smallest penalty that reaches the isotonic solution.

transform(X)[source]

Map scores through the fitted calibration curve.

Parameters:

X (ndarray) – The values to be calibrated.

Returns:

Calibrated values.

Return type:

X_calibrated

set_fit_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

NearlyIsotonicCalibrator

Note

Penalizes rather than forbids monotonicity violations.

Research

Cost- and Data-Informed Isotonic Calibrator

class calibre.CDIIsotonicCalibrator(thresholds, *, threshold_weights=None, bandwidth=0.05, alpha=0.05, gamma=0.15, clip_output=True, enable_diagnostics=False)[source]

Bases: BaseCalibrator

Cost- and data-informed isotonic calibration (CDI-ISO).

On the sorted unique score groups, this estimator solves

\[\min_z \sum_i w_i(y_i-z_i)^2 \quad\text{s.t.}\quad z_{i+1}-z_i \ge L_i,\]

where \(L_i=\phi_i-\epsilon_i\). Near user-supplied operating thresholds, \(\phi_i\) can require a positive increment when the adjacent empirical rates differ by more than their pooled normal-approximation uncertainty. Away from those thresholds, \(\epsilon_i\) permits a bounded decrease. A cumulative shift reduces the problem exactly to weighted PAVA.

The repository manuscript documents this experimental construction. There is no independent reference implementation or evidence for universal defaults. Choose bandwidth, alpha, and gamma using validation data that was not used to fit the final calibration map, and compare proper scores as well as decision utility at the intended threshold.

Parameters:

  • thresholds (Sequence[float]) – Non-empty operating thresholds in [0, 1]. Thresholds and input scores use the same probability-score scale.

  • threshold_weights (Sequence[float] | None) – Finite non-negative relative importance weights, one per threshold. At least one must be positive. Equal weights are used when omitted.

  • bandwidth (float) – Positive half-width of the triangular threshold kernel, in probability-score units.

  • alpha (float) – Two-sided significance level in (0, 1) for the pooled normal approximation used in the adjacent-rate lower confidence bound.

  • gamma (float) – Minimum-increment multiplier in [0, 1].

  • clip_output (bool) – Clip fitted values into [0, 1]. Clipping preserves ordering but can flatten a positive local increment at a boundary.

  • enable_diagnostics (bool) – Whether to enable plateau diagnostics.

adjacency_bounds_

Learned lower bound for each adjacent fitted increment.

block_rate_

Weighted event rate at each unique training score.

block_weight_

Objective weight at each unique training score.

calibration_curve_

Fitted right-continuous step function.

cumulative_shift_

Cumulative shift used by the PAVA reduction.

economics_weight_

Threshold-kernel weight for each adjacency.

effective_sample_size_

Kish effective sample size used in each block’s uncertainty calculation. This makes the inference invariant to a common rescaling of sample_weight.

thresholds_

Validated operating thresholds.

threshold_weights_

Validated threshold weights normalized to sum to one.

Notes

CDI-ISO estimates adjacent event rates at repeated score values. With continuous scores, most effective block sizes are one and the normal approximation is weak; discretize scores using a prespecified scheme or use another calibrator rather than interpreting those bounds as strong evidence.

The returned map is the right-continuous step function specified in the paper. It is not scikit-learn’s piecewise-linear isotonic interpolant.

Examples

>>> import numpy as np
>>> from calibre import CDIIsotonicCalibrator
>>> scores = np.repeat([0.2, 0.5, 0.8], 20)
>>> outcomes = np.r_[np.zeros(16), np.ones(4),
...                  np.zeros(10), np.ones(10),
...                  np.zeros(4), np.ones(16)]
>>> calibrator = CDIIsotonicCalibrator(thresholds=[0.5])
>>> calibrated = calibrator.fit_transform(scores, outcomes)
>>> bool(np.all((calibrated >= 0.0) & (calibrated <= 1.0)))
True

References

Sood, G. Calibre: Probability Calibration and Evaluation in Python. Repository manuscript, appendix on experimental CDI-ISO. https://github.com/finite-sample/calibre/blob/main/ms/calibre.tex

Kish, L. (1965). Survey Sampling. John Wiley & Sons.

Vickers, A. J., & Elkin, E. B. (2006). Decision curve analysis: A novel method for evaluating prediction models. Medical Decision Making, 26(6), 565–574.

transform(X)[source]

Map probability scores through the fitted CDI-ISO step function.

Parameters:

X (ndarray) – Probability scores in [0, 1].

Returns:

Calibrated values.

Return type:

ndarray of shape (n_samples,)

Raises:

ValueError – If a score is outside [0, 1].

set_fit_request(*, sample_weight='$UNCHANGED$')

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to fit.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters

sample_weightstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED

Metadata routing for sample_weight parameter in fit.

Returns

selfobject

The updated object.

Parameters:
Return type:

CDIIsotonicCalibrator

Note

CDI-ISO is research-grade. It uses economic decision theory and statistical evidence to require positive increments near operating thresholds and permit bounded decreases away from them. It is not globally monotone unless all learned adjacency bounds are non-negative.

Usage Examples

Basic Example

import numpy as np

from calibre import CenteredIsotonicCalibrator

rng = np.random.default_rng(42)
X = rng.uniform(0, 1, 1000)
y = rng.binomial(1, X).astype(float)

calibrator = CenteredIsotonicCalibrator().fit(X, y)

X_new = rng.uniform(0, 1, 100)
y_calibrated = calibrator.transform(X_new)

Warning

Always fit the calibrator on data the model did not train on. A model’s scores on its own training data are already too good, so a calibrator fitted there learns the wrong correction. Use a held-out split, or cross_val_calibrate() for out-of-fold predictions.

Comparing Methods

import numpy as np

from calibre import (
    CenteredIsotonicCalibrator,
    IsotonicCalibrator,
    SplineCalibrator,
    RelaxedPAVACalibrator,
    unique_value_counts,
)

calibrators = {
    "Isotonic": IsotonicCalibrator(),
    "Centered": CenteredIsotonicCalibrator(),
    "Spline": SplineCalibrator(),
    "Relaxed PAVA": RelaxedPAVACalibrator(min_increment=-0.01),
    "Spline (fixed alpha)": SplineCalibrator(alpha=0.1),
}

for name, cal in calibrators.items():
    out = cal.fit(X, y).transform(X)
    n = unique_value_counts(out)["n_unique_predictions"]
    print(f"{name:14s} {n:5d} distinct values")

CDI-ISO Usage Example

import numpy as np

from calibre import CDIIsotonicCalibrator

cdi_cal = CDIIsotonicCalibrator(
    thresholds=[0.3, 0.7],
    threshold_weights=[0.6, 0.4],
    bandwidth=0.1,
    gamma=0.2,
    alpha=0.05,
)
cdi_cal.fit(X, y)
y_calibrated = cdi_cal.transform(X_new)

bounds = cdi_cal.adjacency_bounds_
breakpoints = cdi_cal.calibration_curve_.x

print(f"CDI calibrator learned {len(bounds)} local bounds")
print(f"Calibration function has {len(breakpoints)} breakpoints")

The thresholds and scores must use the same probability-score scale. CDI-ISO’s uncertainty calculation operates on repeated score groups; when scores are continuous, most group sizes are one and the normal approximation is weak. Tune its design parameters and compare proper scores and decision utility on held-out validation data.