Plateau Diagnostics Demo

This notebook demonstrates the use of isotonic regression plateau diagnostics to distinguish between noise-based flattening (good) and limited-data flattening (bad).

Overview

When isotonic regression creates flat regions (plateaus) in calibration curves, it could be for two reasons:

1. Noise-based flattening (good): Adjacent scores truly have similar risks, and pooling reduces variance without losing meaningful resolution.

2. Limited-data flattening (bad): Adjacent scores have different risks, but the calibration sample is too small to detect the difference.

This package provides comprehensive diagnostics to help distinguish between these cases.

1. Generate Synthetic Data

Let’s create two scenarios:

• Scenario A: Data with genuine flat regions (noise-based flattening)

• Scenario B: Data with smooth trends but small sample size (limited-data flattening)

import numpy as np
from sklearn.isotonic import IsotonicRegression
from sklearn.model_selection import train_test_split

from calibre import (
    IsotonicCalibrator,
    NearlyIsotonicCalibrator,
    RegularizedIsotonicCalibrator,
)
from calibre.diagnostics import run_plateau_diagnostics
from calibre.metrics import (
    calibration_diversity_index,
    plateau_quality_score,
    progressive_sampling_diversity,
    tie_preservation_score,
)

np.random.seed(42)

def create_genuine_plateau_data(n=200, noise_level=0.05):
    """Create data with genuine flat regions."""
    X = np.sort(np.random.uniform(0, 1, n))

    # Create true probabilities with intentional flat regions
    y_true = np.zeros(n)
    y_true[: n // 4] = 0.1  # Flat low region
    y_true[n // 4 : n // 2] = np.linspace(0.1, 0.4, n // 4)  # Rising
    y_true[n // 2 : 3 * n // 4] = 0.4  # Flat middle region
    y_true[3 * n // 4 :] = np.linspace(0.4, 0.8, n // 4)  # Rising

    # Add small amount of noise
    y_true += np.random.normal(0, noise_level, n)
    y_true = np.clip(y_true, 0, 1)

    # Generate binary outcomes
    y_binary = np.random.binomial(1, y_true)

    return X, y_binary, y_true


def create_smooth_small_data(n=50):
    """Create smooth data with small sample size."""
    X = np.sort(np.random.uniform(0, 1, n))

    # Smooth sigmoid-like curve
    y_true = 1 / (1 + np.exp(-8 * (X - 0.5)))

    # Generate binary outcomes
    y_binary = np.random.binomial(1, y_true)

    return X, y_binary, y_true


# Generate both scenarios
X_genuine, y_genuine, y_true_genuine = create_genuine_plateau_data()
X_small, y_small, y_true_small = create_smooth_small_data()

print(f"Genuine plateau data: {len(X_genuine)} samples")
print(f"Small sample data: {len(X_small)} samples")

Genuine plateau data: 200 samples
Small sample data: 50 samples

2. Basic Isotonic Regression with Diagnostics

Let’s start with the simple wrapper that automatically runs diagnostics:

# Scenario A: Genuine plateaus (updated for v0.4.1)
print("=== Scenario A: Genuine Plateau Data ===")
cal_genuine = IsotonicCalibrator(enable_diagnostics=True)
cal_genuine.fit(X_genuine, y_genuine)

print("\nDiagnostic Summary:")
if cal_genuine.has_diagnostics():
    print(cal_genuine.diagnostic_summary())
else:
    print("No diagnostics available")

# Get calibrated predictions
y_cal_genuine = cal_genuine.transform(X_genuine)

=== Scenario A: Genuine Plateau Data ===

Diagnostic Summary:
Detected 7 plateau(s):

Warnings:
  ⚠ Plateau 4 at [0.817, 0.832] has only 4 samples - may be unreliable
  ⚠ Plateau 7 at [0.972, 0.987] has only 3 samples - may be unreliable

# Scenario B: Small sample data (updated for v0.4.1)
print("=== Scenario B: Small Sample Data ===")
cal_small = IsotonicCalibrator(enable_diagnostics=True)
cal_small.fit(X_small, y_small)

print("\nDiagnostic Summary:")
if cal_small.has_diagnostics():
    print(cal_small.diagnostic_summary())
else:
    print("No diagnostics available")

y_cal_small = cal_small.transform(X_small)

=== Scenario B: Small Sample Data ===

Diagnostic Summary:
Detected 5 plateau(s):

Warnings:
  ⚠ Plateau 1 at [0.052, 0.102] has only 3 samples - may be unreliable
  ⚠ Plateau 4 at [0.533, 0.581] has 5 samples - consider collecting more data in this range

3. Advanced Diagnostic Analysis

For more detailed analysis, call run_plateau_diagnostics directly:

# Split data for more thorough analysis (train/test)
X_train, X_test, y_train, y_test = train_test_split(
    X_genuine, y_genuine, test_size=0.3, random_state=42
)

# First fit calibrator and get predictions
cal = IsotonicCalibrator()
cal.fit(X_train, y_train)
y_cal_train = cal.transform(X_train)

# Run plateau diagnostics on the calibrated results. The diagnosis is
# structural -- it reads the calibrated curve, not the labels -- so the
# true outcomes are not an argument.
results = run_plateau_diagnostics(X_train, y_cal_train)

print("Detailed diagnostic results:")
print(f"Detected {results['n_plateaus']} plateau regions")

if results["n_plateaus"] > 0:
    print("\nPlateau details:")
    for i, plateau in enumerate(results["plateaus"]):
        print(f"  Plateau {i + 1}:")
        if "x_range" in plateau:
            print(
                f"    X range: [{plateau['x_range'][0]:.3f}, "
                f"{plateau['x_range'][1]:.3f}]"
            )
        if "value" in plateau:
            print(f"    Value: {plateau['value']:.3f}")
        if "n_samples" in plateau:
            print(f"    Samples: {plateau['n_samples']}")
        if "sample_density" in plateau:
            print(f"    Density: {plateau['sample_density']}")

if results["warnings"]:
    print("\nWarnings:")
    for warning in results["warnings"]:
        print(f"  ⚠️  {warning}")

Detailed diagnostic results:
Detected 6 plateau regions

Plateau details:
  Plateau 1:
    X range: [0.007, 0.051]
    Value: 0.111
    Samples: 9
    Density: sparse
  Plateau 2:
    X range: [0.058, 0.242]
    Value: 0.273
    Samples: 33
    Density: adequate
  Plateau 3:
    X range: [0.242, 0.832]
    Value: 0.365
    Samples: 74
    Density: adequate
  Plateau 4:
    X range: [0.835, 0.897]
    Value: 0.571
    Samples: 7
    Density: sparse
  Plateau 5:
    X range: [0.897, 0.970]
    Value: 0.857
    Samples: 14
    Density: adequate
  Plateau 6:
    X range: [0.986, 0.987]
    Value: 1.000
    Samples: 2
    Density: very_sparse

Warnings:
  ⚠️  Plateau 1 at [0.007, 0.051] has 9 samples - consider collecting more data in this range
  ⚠️  Plateau 4 at [0.835, 0.897] has 7 samples - consider collecting more data in this range
  ⚠️  Plateau 6 at [0.986, 0.987] has only 2 samples - may be unreliable

4. Diagnostic Metrics

Let’s explore the specific diagnostic metrics:

# Compare original vs calibrated predictions
iso_basic = IsotonicRegression()
iso_basic.fit(X_genuine, y_genuine)
y_cal_basic = iso_basic.transform(X_genuine)

# Tie preservation score
tie_score = tie_preservation_score(X_genuine, y_cal_basic)
print(f"Tie preservation score: {tie_score:.3f}")

# Plateau quality score
quality_score = plateau_quality_score(X_genuine, y_genuine, y_cal_basic)
print(f"Plateau quality score: {quality_score:.3f}")

# Calibration diversity
diversity_orig = calibration_diversity_index(X_genuine)
diversity_cal = calibration_diversity_index(y_cal_basic)
diversity_relative = calibration_diversity_index(y_cal_basic, diversity_orig)

print(f"Original diversity: {diversity_orig:.3f}")
print(f"Calibrated diversity: {diversity_cal:.3f}")
print(f"Relative diversity: {diversity_relative:.3f}")

Tie preservation score: 0.642
Plateau quality score: 0.423
Original diversity: 1.000
Calibrated diversity: 0.040
Relative diversity: 0.040

5. Progressive Sampling Analysis

This helps distinguish limited-data flattening by showing how diversity changes with sample size:

# Progressive sampling analysis
sample_sizes, diversities = progressive_sampling_diversity(
    X_genuine, y_genuine, sample_sizes=[50, 100, 150, 200], n_trials=10, random_state=42
)

print("Progressive sampling results:")
for size, div in zip(sample_sizes, diversities):
    print(f"  Sample size {size}: diversity = {div:.3f}")

# Interpret trend
slope = (diversities[-1] - diversities[0]) / (sample_sizes[-1] - sample_sizes[0])
if slope > 0.001:
    print(
        "\nInterpretation: Increasing diversity suggests potential "
        "limited-data flattening"
    )
elif slope < -0.001:
    print("\nInterpretation: Decreasing diversity (unusual pattern)")
else:
    print("\nInterpretation: Stable diversity suggests genuine flatness")

Progressive sampling results:
  Sample size 50: diversity = 0.098
  Sample size 100: diversity = 0.066
  Sample size 150: diversity = 0.053
  Sample size 200: diversity = 0.040

Interpretation: Stable diversity suggests genuine flatness

6. Comparison with Alternative Methods

Let’s compare strict isotonic regression with softer alternatives:

# Compare different calibration methods (updated for v0.4.1)
from calibre.metrics import mean_calibration_error

# Fit different calibrators
iso_strict = IsotonicCalibrator()
iso_nearly = NearlyIsotonicCalibrator(lam=1.0)
iso_reg = RegularizedIsotonicCalibrator(alpha=0.1)

calibrators = {
    "Strict Isotonic": iso_strict.fit(X_genuine, y_genuine),
    "Nearly Isotonic": iso_nearly.fit(X_genuine, y_genuine),
    "Regularized": iso_reg.fit(X_genuine, y_genuine),
}

# Compare diversity and calibration error
print("Method comparison:")
for name, cal in calibrators.items():
    try:
        y_pred = cal.transform(X_genuine)
        diversity = calibration_diversity_index(y_pred)
        error = mean_calibration_error(y_genuine, y_pred)
        n_unique = len(np.unique(y_pred))

        print(f"  {name}:")
        print(f"    Diversity: {diversity:.3f}")
        print(f"    Calibration error: {error:.3f}")
        print(f"    Unique values: {n_unique}/{len(y_pred)}")
    except Exception as e:
        print(f"  {name}: Error - {e}")

Method comparison:
  Strict Isotonic:
    Diversity: 0.040
    Calibration error: 0.000
    Unique values: 8/200
  Nearly Isotonic:
    Diversity: 0.015
    Calibration error: 0.000
    Unique values: 3/200
  Regularized:
    Diversity: 0.795
    Calibration error: 0.000
    Unique values: 159/200

7. Visualization (if matplotlib available)

Let’s create some visualizations to better understand the diagnostics:

8. Practical Decision Framework

Based on the diagnostic results, here’s how to make practical decisions:

def recommend_calibration_method(diagnostic_results, diversity_trend_slope=None):
    """Provide calibration method recommendations based on diagnostics."""

    if diagnostic_results["n_plateaus"] == 0:
        return "Standard isotonic regression (no plateaus detected)"

    # Count plateau types based on sample density
    concerning = 0
    total = diagnostic_results["n_plateaus"]

    for plateau in diagnostic_results["plateaus"]:
        if "sample_density" in plateau:
            if plateau["sample_density"] in ["sparse", "very_sparse"]:
                concerning += 1

    recommendations = []

    if concerning == 0:
        recommendations.append(
            "✅ Standard isotonic regression (all plateaus appear genuine)"
        )
    elif concerning / total > 0.5:
        recommendations.append("⚠️  Consider softer calibration methods:")
        recommendations.append(
            "   - Nearly isotonic regression (allows small violations)"
        )
        recommendations.append("   - Regularized isotonic regression")
        recommendations.append("   - Spline calibration")
    else:
        recommendations.append("🤔 Mixed evidence - consider:")
        recommendations.append("   - Cross-validation between strict and soft methods")
        recommendations.append("   - Collecting more calibration data if possible")

    # Additional recommendations based on diversity trend
    if diversity_trend_slope is not None:
        if diversity_trend_slope > 0.001:
            recommendations.append(
                "📈 Increasing diversity with sample size suggests "
                "limited-data flattening"
            )
            recommendations.append(
                "   -> Strongly recommend collecting more data or using softer methods"
            )
        elif diversity_trend_slope < -0.001:
            recommendations.append(
                "📉 Unusual decreasing diversity pattern - investigate data quality"
            )

    return "\n".join(recommendations)


# Get recommendations for our data (updated for v0.4.1)
slope = (diversities[-1] - diversities[0]) / (sample_sizes[-1] - sample_sizes[0])
recommendations = recommend_calibration_method(results, slope)

print("=== CALIBRATION METHOD RECOMMENDATIONS ===")
print(recommendations)

=== CALIBRATION METHOD RECOMMENDATIONS ===
🤔 Mixed evidence - consider:
   - Cross-validation between strict and soft methods
   - Collecting more calibration data if possible

9. Summary and Best Practices

What the diagnostics actually tell you

run_plateau_diagnostics is a structural check. It reads the calibrated curve and reports where it went flat and how many samples sit underneath each flat region:

1. Plateau count and extent: how much of the score range collapsed to a single value.

2. Sample density per plateau (adequate / sparse / very_sparse): a plateau resting on four samples is not evidence that four scores share a risk — it is evidence that you cannot tell.

3. Progressive sampling diversity (progressive_sampling_diversity): if distinct output values keep rising with sample size, the flatness is a sample-size artifact rather than a property of the data.

Only the third of these uses the outcomes. The first two describe the fitted curve, which is why the diagnostics are a starting point for investigation, not a hypothesis test.

Best Practices

1. Run diagnostics whenever you use isotonic regression on scores you care about ranking.

2. Treat sparse plateaus as a data question, not a modeling one — the fix is usually more data in that score range.

3. Compare against a granularity-preserving method such as CenteredIsotonicCalibrator or SplineCalibrator, and pick on held-out log-loss or Brier score rather than on the diagnostic alone.

4. Document the choice, including the held-out numbers that justified it.

print("🎉 Plateau diagnostics demo completed!")
print("\nKey takeaways:")
print("1. Not all plateaus are created equal")
print("2. Diagnostics help distinguish genuine vs. artifactual flattening")
print("3. Multiple complementary tests provide robust evidence")
print("4. Consider both statistical and domain-specific evidence")
print("5. When in doubt, prefer softer calibration methods")

🎉 Plateau diagnostics demo completed!

Key takeaways:
1. Not all plateaus are created equal
2. Diagnostics help distinguish genuine vs. artifactual flattening
3. Multiple complementary tests provide robust evidence
4. Consider both statistical and domain-specific evidence
5. When in doubt, prefer softer calibration methods