Cross-Validation

Shared cross-validation machinery. Every calibrator with an "auto" hyperparameter resolves it through select_by_cv(), so the selection rule is the same everywhere and is implemented once.

The default shuffled folds assume independent, exchangeable observations. They are not valid partitions for grouped, repeated-measures, spatial, or temporal data.

Selection is always on a proper scoring rule — log loss or Brier. Calibrators default to scoring="auto": log loss for probability targets and Brier (squared error) for unbounded identity-scale targets. Naming log loss for an unbounded target raises. Calibration error is deliberately rejected as a selection criterion: it is not proper, and a calibrator tuned to minimise ECE can win by discarding resolution. There is a test asserting the rejection.

Out-of-Fold Calibration

calibre.cross_val_calibrate(calibrator, X, y, *, sample_weight=None, cv=5, random_state=0)[source]

Return out-of-fold calibrated probabilities.

Each value is produced by a calibrator that never saw that observation, so the result can be scored without the optimism of measuring a fit on its own training data. This is the honest input to calibre.evaluation.score_decomposition().

Parameters:

  • calibrator (Any) – An unfitted calibrator. Cloned per fold, so the object passed in is left untouched.

  • X (ndarray) – Uncalibrated scores.

  • y (ndarray) – Targets.

  • sample_weight (ndarray | None) – Non-negative per-observation weights used to fit each training fold. Validation rows retain their original positions, including rows with zero weight. Zero-weight rows do not affect fold stratification or fitting.

  • cv (int) – Number of folds.

  • random_state (int | None) – Seed for the folds.

Returns:

Out-of-fold calibrated probabilities,

in the input’s order.

Return type:

ndarray of shape (n_samples,)

Raises:

  • ValueError – If fewer than two observations have positive weight.

  • RuntimeError – If the folds did not cover every observation, which would leave some rows with no out-of-fold prediction at all.

Notes

A calibrator that selects its own parameters runs that selection inside each training fold, which makes this a nested cross-validation and keeps the reported performance honest for a tuned model.

Examples

>>> import numpy as np
>>> from calibre import IsotonicCalibrator
>>> from calibre.selection import cross_val_calibrate
>>> rng = np.random.default_rng(0)
>>> x = rng.uniform(0, 1, 200)
>>> y = rng.binomial(1, x).astype(float)
>>> oof = cross_val_calibrate(IsotonicCalibrator(), x, y, cv=4)
>>> oof.shape
(200,)

Model Selection

calibre.select_by_cv(calibrator_factory, param_grid, X, y, *, sample_weight=None, cv=5, scoring='log_loss', max_cv_samples=20000, random_state=0)[source]

Choose parameters by cross-validation on a proper scoring rule.

The winning configuration is not fitted here. The caller refits it on all the data, which matters: keeping a fold’s model would ship an estimator that had seen only (cv-1)/cv of the sample.

Parameters:

  • calibrator_factory (Callable[..., Any]) – Called with a candidate’s keyword arguments, returning an unfitted calibrator.

  • param_grid (dict[str, Sequence[Any]]) – Mapping from parameter name to candidate values.

  • X (np.ndarray) – Uncalibrated scores.

  • y (np.ndarray) – Targets.

  • sample_weight (np.ndarray | None) – Non-negative per-observation weights.

  • cv (int) – Number of folds.

  • scoring (str) – "log_loss" (default), "brier", or "auto". Automatic scoring uses log loss for targets in [0, 1] and squared error otherwise.

  • max_cv_samples (int | None) – Subsample above this size before searching. Selection only has to rank candidates, so its cost is bounded; None disables.

  • random_state (int | None) – Seed for folds and subsampling.

Returns:

The winning parameters, ready to pass to calibrator_factory.

Return type:

dict

Raises:

ValueError – If the grid is empty, scoring is unknown, log loss is requested for targets outside [0, 1], or every candidate failed.

Examples

>>> import numpy as np
>>> from calibre import NearlyIsotonicCalibrator
>>> from calibre.selection import select_by_cv
>>> rng = np.random.default_rng(0)
>>> x = rng.uniform(0, 1, 300)
>>> y = rng.binomial(1, x).astype(float)
>>> best = select_by_cv(
...     lambda **kw: NearlyIsotonicCalibrator(**kw),
...     {"lam": [0.1, 1.0, 10.0]},
...     x,
...     y,
...     cv=3,
... )
>>> sorted(best)
['lam']
calibre.make_folds(X, y, *, cv=5, random_state=0)[source]

Build cross-validation folds, stratifying binary targets.

Parameters:

  • X (ndarray) – Uncalibrated scores.

  • y (ndarray) – Finite binary or continuous calibration targets.

  • cv (int) – Requested number of folds. Reduced when the data cannot support it.

  • random_state (int | None) – Seed for the shuffle.

Returns:

(train_index, validation_index) pairs.

Return type:

list of (ndarray, ndarray)

Raises:

ValueError – If fewer than two observations are supplied, cv is not an integer, or cv is below 2.

Notes

With binary targets the fold count is capped by the rarer class, so a rare class appears in every training and validation split. For continuous targets, the fold count is capped by the sample size.

The shuffled folds assume independent, exchangeable observations. They are not appropriate for grouped, repeated-measures, spatial, or temporal data.

Splitting delegates to sklearn.model_selection.StratifiedKFold for binary targets and sklearn.model_selection.KFold otherwise.

Examples

>>> import numpy as np
>>> from calibre.selection import make_folds
>>> x = np.linspace(0, 1, 20)
>>> y = (x > 0.5).astype(float)
>>> len(make_folds(x, y, cv=4))
4
calibre.selection.resolve_auto(parameter_value, parameter_name, parameter_grid, calibrator_factory, X, y, *, cv=5, scoring='log_loss', random_state=0, minimum_value=0.0, sample_weight=None)[source]

Resolve one parameter that may be a number or "auto".

The shared implementation behind every calibrator’s "auto" default, so the selection rules live in one place rather than being restated per estimator.

Parameters:

  • parameter_value (float | str) – The constructor argument: a number, or "auto".

  • parameter_name (str) – Parameter name, used in the grid and in error messages.

  • parameter_grid (Sequence[Any]) – Candidate values searched when the value is "auto".

  • calibrator_factory (Callable[..., Any]) – Called with the candidate to build an unfitted calibrator.

  • X (np.ndarray) – Uncalibrated scores.

  • y (np.ndarray) – Targets.

  • cv (int) – Number of folds.

  • scoring (str) – Selection criterion, a proper scoring rule.

  • random_state (int | None) – Seed for folds.

  • minimum_value (float) – Smallest permitted numeric value.

  • sample_weight (np.ndarray | None) – Non-negative per-observation weights used during selection.

Returns:

The resolved value. Callers store it on a trailing-underscore

attribute; writing it back onto the constructor argument would break get_params round-tripping and therefore clone.

Return type:

float

Raises:

ValueError – If parameter_value is a string other than "auto", or a number below minimum_value.

Examples

>>> import numpy as np
>>> from calibre.selection import resolve_auto
>>> resolve_auto(
...     0.5, "alpha", [0.1, 1.0], lambda **kw: None, np.array([]), np.array([])
... )
0.5

Usage

import numpy as np

from calibre import CenteredIsotonicCalibrator, cross_val_calibrate

rng = np.random.default_rng(0)
scores = rng.uniform(0, 1, 1500)
labels = rng.binomial(1, scores).astype(float)

# Every returned probability comes from a model that never saw that row.
out_of_fold = cross_val_calibrate(
    CenteredIsotonicCalibrator(), scores, labels, cv=5
)
print(out_of_fold.shape)