Cross-Validation

Shared cross-validation machinery. Every calibrator with an "auto" hyperparameter resolves it through select_by_cv(), so the selection rule is the same everywhere and is implemented once.

Selection is always on a proper scoring rule — log loss or Brier. Calibration error is deliberately rejected as a selection criterion: it is not proper, and a calibrator tuned to minimise ECE can win by discarding resolution. There is a test asserting the rejection.

Out-of-Fold Calibration

calibre.cross_val_calibrate(calibrator, X, y, cv=5, random_state=0)[source]

Return out-of-fold calibrated probabilities.

Each value is produced by a calibrator that never saw that observation, so the result can be scored without the optimism of measuring a fit on its own training data. This is the honest input to calibre.evaluation.score_decomposition().

Parameters:

  • calibrator (Any) – An unfitted calibrator. Cloned per fold, so the object passed in is left untouched.

  • X (ndarray) – Uncalibrated scores.

  • y (ndarray) – Targets.

  • cv (int) – Number of folds.

  • random_state (int | None) – Seed for the folds.

Returns:

Out-of-fold calibrated probabilities,

in the input’s order.

Return type:

ndarray of shape (n_samples,)

Raises:

RuntimeError – If the folds did not cover every observation, which would leave some rows with no out-of-fold prediction at all.

Notes

A calibrator that selects its own parameters runs that selection inside each training fold, which makes this a nested cross-validation and keeps the reported performance honest for a tuned model.

Examples

>>> import numpy as np
>>> from calibre import IsotonicCalibrator
>>> from calibre.selection import cross_val_calibrate
>>> rng = np.random.default_rng(0)
>>> x = rng.uniform(0, 1, 200)
>>> y = rng.binomial(1, x).astype(float)
>>> oof = cross_val_calibrate(IsotonicCalibrator(), x, y, cv=4)
>>> oof.shape
(200,)

Model Selection

calibre.select_by_cv(factory, param_grid, X, y, sample_weight=None, cv=5, scoring='log_loss', max_cv_samples=20000, random_state=0)[source]

Choose parameters by cross-validation on a proper scoring rule.

The winning configuration is not fitted here. The caller refits it on all the data, which matters: keeping a fold’s model would ship an estimator that had seen only (cv-1)/cv of the sample.

Parameters:

  • factory (Callable[..., Any]) – Called with a candidate’s keyword arguments, returning an unfitted calibrator.

  • param_grid (dict[str, Sequence[Any]]) – Mapping from parameter name to candidate values.

  • X (np.ndarray) – Uncalibrated scores.

  • y (np.ndarray) – Targets.

  • sample_weight (np.ndarray | None) – Non-negative per-observation weights.

  • cv (int) – Number of folds.

  • scoring (str) – "log_loss" (default) or "brier".

  • max_cv_samples (int | None) – Subsample above this size before searching. Selection only has to rank candidates, so its cost is bounded; None disables.

  • random_state (int | None) – Seed for folds and subsampling.

Returns:

The winning parameters, ready to splat into factory.

Return type:

dict

Raises:

ValueError – If the grid is empty, scoring is unknown, or every candidate failed.

Examples

>>> import numpy as np
>>> from calibre import NearlyIsotonicCalibrator
>>> from calibre.selection import select_by_cv
>>> rng = np.random.default_rng(0)
>>> x = rng.uniform(0, 1, 300)
>>> y = rng.binomial(1, x).astype(float)
>>> best = select_by_cv(
...     lambda **kw: NearlyIsotonicCalibrator(**kw),
...     {"lam": [0.1, 1.0, 10.0]},
...     x,
...     y,
...     cv=3,
... )
>>> sorted(best)
['lam']
calibre.make_folds(X, y, cv=5, random_state=0)[source]

Build cross-validation folds, stratifying binary targets.

Parameters:

  • X (ndarray) – Uncalibrated scores.

  • y (ndarray) – Targets: binary labels, or values in [0, 1].

  • cv (int) – Requested number of folds. Reduced when the data cannot support it.

  • random_state (int | None) – Seed for the shuffle.

Returns:

(train_index, validation_index) pairs.

Return type:

list of (ndarray, ndarray)

Raises:

ValueError – If cv is below 2.

Notes

With binary targets the fold count is capped by the rarer class, so a rare positive appears in every training split rather than leaving a fold with no positives at all.

Examples

>>> import numpy as np
>>> from calibre.selection import make_folds
>>> x = np.linspace(0, 1, 20)
>>> y = (x > 0.5).astype(float)
>>> len(make_folds(x, y, cv=4))
4
calibre.selection.resolve_auto(value, name, grid, factory, X, y, cv=5, scoring='log_loss', random_state=0, minimum=0.0, sample_weight=None)[source]

Resolve one parameter that may be a number or "auto".

The shared implementation behind every calibrator’s "auto" default, so the selection rules live in one place rather than being restated per estimator.

Parameters:

  • value (float | str) – The constructor argument: a number, or "auto".

  • name (str) – Parameter name, used in the grid and in error messages.

  • grid (Sequence[Any]) – Candidate values searched when value is "auto".

  • factory (Callable[..., Any]) – Called with {name: candidate} to build an unfitted calibrator.

  • X (np.ndarray) – Uncalibrated scores.

  • y (np.ndarray) – Targets.

  • cv (int) – Number of folds.

  • scoring (str) – Selection criterion, a proper scoring rule.

  • random_state (int | None) – Seed for folds.

  • minimum (float) – Smallest permitted numeric value.

  • sample_weight (np.ndarray | None) – Non-negative per-observation weights used during selection.

Returns:

The resolved value. Callers store it on a trailing-underscore

attribute; writing it back onto the constructor argument would break get_params round-tripping and therefore clone.

Return type:

float

Raises:

ValueError – If value is a string other than "auto", or a number below minimum.

Examples

>>> import numpy as np
>>> from calibre.selection import resolve_auto
>>> resolve_auto(
...     0.5, "alpha", [0.1, 1.0], lambda **kw: None, np.array([]), np.array([])
... )
0.5

Usage

import numpy as np

from calibre import CenteredIsotonicCalibrator, cross_val_calibrate

rng = np.random.default_rng(0)
scores = rng.uniform(0, 1, 1500)
labels = rng.binomial(1, scores).astype(float)

# Every returned probability comes from a model that never saw that row.
out_of_fold = cross_val_calibrate(
    CenteredIsotonicCalibrator(), scores, labels, cv=5
)
print(out_of_fold.shape)