Plateau Diagnostics

Isotonic regression produces a step function, and each step is a plateau: a range of input scores that all come out equal. Inside a plateau, cases are indistinguishable — which matters as soon as you rank, threshold, or bucket the output.

These functions find the plateaus and report how much data each one rests on. The analysis is purely structural: it describes the shape of a fitted curve and does not test whether a plateau is statistically justified.

Diagnostics

calibre.run_plateau_diagnostics(X, y_calibrated)[source]

Detect and analyze plateaus (flat regions) in calibration curves.

This function identifies flat regions where the calibrator outputs the same value for multiple inputs, and flags potentially problematic plateaus based on simple, interpretable criteria like sample count.

The diagnosis is purely structural: it counts how many samples support each flat region. It does not take the true labels, because it makes no claim about whether a plateau is justified by the outcomes – only about whether enough data sits underneath it to say anything at all.

Parameters:

  • X (ndarray) – Original predicted probabilities.

  • y_calibrated (ndarray) – Calibrated probabilities.

Returns:

Dictionary containing:

  • 'n_plateaus': Number of plateaus detected.

  • 'plateaus': List of plateau information dicts, each containing:

  • 'plateau_id': Unique identifier (0-indexed).

  • 'x_range': Tuple of (min, max) input values in the plateau.

  • 'value': The constant output value of the plateau.

  • 'width': Number of samples in the plateau.

  • 'n_samples': Number of samples (same as width).

  • 'sample_density': 'adequate', 'sparse' or 'very_sparse'.

  • 'warnings': List of warning messages about problematic plateaus.

Return type:

diagnostics

Examples

>>> X = np.array([0.1, 0.2, 0.3, 0.7, 0.8, 0.9])
>>> y_cal = np.array([0.2, 0.2, 0.2, 0.8, 0.8, 0.8])
>>> diagnostics = run_plateau_diagnostics(X, y_cal)
>>> print(diagnostics['n_plateaus'])
2
>>> for warning in diagnostics['warnings']:
...     print(warning)
Plateau 1 at [0.100, 0.300] has only 3 samples - may be unreliable
Plateau 2 at [0.700, 0.900] has only 3 samples - may be unreliable
calibre.detect_plateaus(y_calibrated, min_width=2)[source]

Detect plateaus (consecutive identical values) in calibrated predictions.

Parameters:

  • y_calibrated (ndarray) – Sorted calibrated probabilities.

  • min_width (int) – Minimum number of consecutive identical values to count as a plateau.

Returns:

List of (start_index, end_index, value) tuples for each

detected plateau. Indices are inclusive.

Return type:

plateaus

Examples

>>> y_cal = np.array([0.2, 0.2, 0.2, 0.5, 0.8, 0.8])
>>> plateaus = detect_plateaus(y_cal)
>>> [(lo, hi, float(v)) for lo, hi, v in plateaus]
[(0, 2, 0.2), (4, 5, 0.8)]
calibre.diagnostics.analyze_plateau_simple(X, start_idx, end_idx, value, plateau_id)[source]

Analyze a single plateau with simple, interpretable metrics.

Parameters:

  • X (ndarray) – Sorted input predictions.

  • start_idx (int) – Start index of plateau (inclusive).

  • end_idx (int) – End index of plateau (inclusive).

  • value (float) – The constant value of the plateau.

  • plateau_id (int) – Unique identifier for this plateau.

Returns:

Dictionary with plateau information:

  • plateau_id

  • x_range: (min, max) of input values

  • value: output value

  • width: number of samples

  • n_samples: same as width

  • sample_density: ‘adequate’, ‘sparse’, or ‘very_sparse’

Return type:

plateau_info

calibre.diagnostics.diversity_learning_curve(X, y, calibrator=None, sample_sizes=None, n_trials=10, random_state=None)[source]

Measure how calibration diversity changes with training sample size.

This diagnostic tool helps determine whether you have sufficient training data for stable calibration. If diversity continues increasing with sample size, more data would likely improve calibration granularity.

Parameters:

  • X (ndarray) – Input features (predicted probabilities).

  • y (ndarray) – True binary labels.

  • calibrator (Any) – Calibrator to test. If None, uses IsotonicCalibrator.

  • sample_sizes (list[int] | None) – Sample sizes to test. If None, uses default range covering 10% to 100% of available data.

  • n_trials (int) – Number of random trials per sample size for averaging.

  • random_state (int | None) – Random state for reproducibility.

Returns:

Sample sizes tested. diversities: Mean fraction of unique calibrated values at each size.

Return type:

sizes

Raises:

ValueError – If X and y have different lengths.

Notes

This function is computationally expensive as it fits the calibrator multiple times (n_trials x len(sample_sizes) fits). Use for diagnostic analysis, not routine evaluation.

The diversity metric measures granularity: higher diversity means more unique calibrated values, indicating better discrimination. If diversity plateaus, you have sufficient data. If it keeps increasing, more data would help.

Examples

>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> X = rng.uniform(0, 1, 200)
>>> y = (X > 0.5).astype(int)
>>>
>>> sizes, divs = diversity_learning_curve(
...     X, y, sample_sizes=[50, 100, 200], n_trials=2, random_state=0
... )
>>> sizes
[50, 100, 200]
>>> len(divs) == 3 and all(0.0 <= d <= 1.0 for d in divs)
True

Rising diversity suggests more data would buy more granularity; a flat tail suggests the calibrator has the resolution the data can support.

See also

unique_value_counts : Count unique values in calibrated predictions run_plateau_diagnostics : Detect and analyze plateaus

Usage

import numpy as np

from calibre import IsotonicCalibrator, run_plateau_diagnostics

rng = np.random.default_rng(0)
scores = np.sort(rng.random(400))
labels = (rng.random(400) < scores).astype(float)

calibrator = IsotonicCalibrator().fit(scores, labels)
report = run_plateau_diagnostics(scores, calibrator.transform(scores))

print(f"{report['n_plateaus']} plateaus")
for plateau in report["plateaus"][:3]:
    low, high = plateau["x_range"]
    print(
        f"  [{low:.3f}, {high:.3f}] -> {plateau['value']:.3f} "
        f"({plateau['n_samples']} samples, {plateau['sample_density']})"
    )

Plateaus flagged very_sparse rest on few observations. report["warnings"] collects those as readable messages.

Built-in diagnostics

Every calibrator can run this automatically at fit time:

from calibre import IsotonicCalibrator

cal = IsotonicCalibrator(enable_diagnostics=True)
cal.fit(scores, labels)

if cal.has_diagnostics():
    print(cal.diagnostic_summary())

Scope

Only two things are implemented here: plateau detection with a sample-count density label, and the diversity learning curve. Earlier changelogs advertised bootstrap tie stability, conditional AUC among tied pairs, minimum detectable difference, and a supported/limited-data/inconclusive classifier. None of those were ever written.

For a statistical rather than structural account of a fitted curve — where the flat regions are, and how much of the score they cost you — use the CORP decomposition in CORP Evaluation.