Plateau Diagnostics

Isotonic regression produces a step function, and each step is a plateau: a range of input scores that all come out equal. Inside a plateau, cases are indistinguishable — which matters as soon as you rank, threshold, or bucket the output.

These functions find the plateaus and report how much data each one rests on. The analysis is purely structural: it describes the shape of a fitted curve and does not test whether a plateau is statistically justified.

Diagnostics

calibre.run_plateau_diagnostics(input_scores, calibrated_predictions)

Detect and analyze plateaus (flat regions) in calibration curves.

This function identifies flat regions where the calibrator outputs the same value for multiple inputs, and flags potentially problematic plateaus based on simple, interpretable criteria like sample count.

The diagnosis is purely structural: it counts how many samples support each flat region. It does not take the true labels, because it makes no claim about whether a plateau is justified by the outcomes – only about whether enough data sits underneath it to say anything at all.

Parameters:

• input_scores (ndarray) – Original predicted probabilities.

• calibrated_predictions (ndarray) – Calibrated probabilities.

Returns:

Dictionary containing:

• 'n_plateaus': Number of plateaus detected.

• 'plateaus': List of plateau information dicts, each containing:

• 'plateau_id': Unique identifier (0-indexed).

• 'input_score_range': Minimum and maximum input score.

• 'calibrated_value': Constant output value of the plateau.

• 'n_observations': Number of observations in the plateau.

• 'support': 'adequate', 'sparse' or 'very_sparse'.

• 'warnings': List of warning messages about problematic plateaus.

Return type:

diagnostics

Raises:

ValueError – If the two arrays have different lengths.

Examples

>>> input_scores = np.array([0.1, 0.2, 0.3, 0.7, 0.8, 0.9])
>>> calibrated = np.array([0.2, 0.2, 0.2, 0.8, 0.8, 0.8])
>>> diagnostics = run_plateau_diagnostics(input_scores, calibrated)
>>> print(diagnostics['n_plateaus'])
2
>>> for warning in diagnostics['warnings']:
...     print(warning)
Plateau 1 at [0.100, 0.300] has only 3 observations - may be unreliable
Plateau 2 at [0.700, 0.900] has only 3 observations - may be unreliable

calibre.detect_plateaus(calibrated_predictions, *, min_width=2)

Detect plateaus (consecutive identical values) in calibrated predictions.

Parameters:

• calibrated_predictions (ndarray) – Calibrated probabilities ordered by input score.

• min_width (int) – Minimum number of consecutive identical values to count as a plateau.

Returns:

List of (start_index, end_index, value) tuples for each

detected plateau. Indices are inclusive.

Return type:

plateaus

Examples

>>> y_cal = np.array([0.2, 0.2, 0.2, 0.5, 0.8, 0.8])
>>> plateaus = detect_plateaus(y_cal)
>>> [(lo, hi, float(v)) for lo, hi, v in plateaus]
[(0, 2, 0.2), (4, 5, 0.8)]

calibre.diagnostics.analyze_plateau_simple(input_scores, start_idx, end_idx, value, plateau_id)

Analyze a single plateau with simple, interpretable metrics.

Parameters:

• input_scores (ndarray) – Sorted input predictions.

• start_idx (int) – Start index of plateau (inclusive).

• end_idx (int) – End index of plateau (inclusive).

• value (float) – The constant value of the plateau.

• plateau_id (int) – Unique identifier for this plateau.

Returns:

Dictionary with plateau information:

• plateau_id

• input_score_range: (minimum, maximum) input value

• calibrated_value: output value

• n_observations: number of observations

• support: ‘adequate’, ‘sparse’, or ‘very_sparse’

Return type:

plateau_info

Usage

import numpy as np

from calibre import IsotonicCalibrator, run_plateau_diagnostics

rng = np.random.default_rng(0)
scores = np.sort(rng.random(400))
labels = (rng.random(400) < scores).astype(float)

calibrator = IsotonicCalibrator().fit(scores, labels)
report = run_plateau_diagnostics(scores, calibrator.transform(scores))

print(f"{report['n_plateaus']} plateaus")
for plateau in report["plateaus"][:3]:
    low, high = plateau["input_score_range"]
    print(
        f"  [{low:.3f}, {high:.3f}] -> {plateau['calibrated_value']:.3f} "
        f"({plateau['n_observations']} observations, {plateau['support']})"
    )

Plateaus flagged very_sparse rest on few observations. report["warnings"] collects those as readable messages.

Built-in diagnostics

Every calibrator can run this automatically at fit time:

from calibre import IsotonicCalibrator

cal = IsotonicCalibrator(enable_diagnostics=True)
cal.fit(scores, labels)

if cal.has_diagnostics():
    print(cal.diagnostic_summary())

Scope

Only two things are implemented here: plateau detection with a sample-count density label, and the diversity learning curve. Earlier changelogs advertised bootstrap tie stability, conditional AUC among tied pairs, minimum detectable difference, and a supported/limited-data/inconclusive classifier. None of those were ever written.

For a statistical rather than structural account of a fitted curve — where the flat regions are, and how much of the score they cost you — use the CORP decomposition in CORP Evaluation.