Examples and Tutorials

This section provides comprehensive examples and tutorials for using Calibre.

• Basic Usage Examples
  • Complete Workflow Example
  • Handling Different Data Types
  • Visualization Examples
  • Cross-Validation for Calibration
  • Common Pitfalls and Solutions
• Advanced Usage Examples
  • Multi-Class Calibration
  • Custom Calibration Pipelines
  • Handling Concept Drift
  • Calibration for Specific Domains
  • Performance Optimization
• Performance Benchmarks
  • The design
  • Results
  • What resolution actually looks like
  • Paired differences against the baseline
  • Where calibre loses
  • What keeps it honest

Interactive Jupyter Notebooks

We provide focused, executable Jupyter notebooks for hands-on learning:

Interactive Examples

• Getting Started with Calibre
• Validation and Evaluation
• Plateau Diagnostics Demo
• Performance Comparison
• Evaluating a Calibrator Honestly
• Multiclass: Which Method Do You Actually Need?

Notebook Overview

📚 Getting Started (Getting Started with Calibre)

• Basic calibration workflow with realistic ML predictions

• Choosing the right calibrator for your data

• Visual validation with reliability diagrams

• Quick start guide for new users

🔍 Validation and Evaluation (Validation and Evaluation)

• Comprehensive calibration quality assessment

• Mathematical property validation (bounds, monotonicity, granularity)

• Performance across different miscalibration patterns

• Edge case testing and robustness analysis

🩺 Diagnostics and Troubleshooting (Plateau Diagnostics Demo)

• Plateau diagnostic tools for isotonic regression

• Distinguishing genuine vs. limited-data flattening

• Progressive sampling: how granularity changes with sample size

• Decision framework for method selection

⚡ Performance Comparison (Performance Comparison)

• Systematic comparison across all calibration methods

• Performance on overconfident, underconfident, and distorted predictions

• Computational efficiency and method ranking

• Guidelines for choosing the optimal method

📐 Evaluating Calibration (Evaluating a Calibrator Honestly)

• CORP reliability diagrams: no bin count to choose

• Consistency and confidence bands for uncertainty quantification

• The MCB / DSC / UNC decomposition — separating fixable miscalibration from lost discrimination

• Why in-sample calibration error is identically zero, and what to use instead

🎯 Multiclass (Multiclass: Which Method Do You Actually Need?)

• Two regimes with different winners, measured against known true probabilities

• miscalibration_profile: telling which regime your data is in

• Why temperature scaling can never change accuracy, and when that is the problem

• The within-class reordering that no standard metric reveals

Running the Notebooks

To run these notebooks locally:

git clone https://github.com/finite-sample/calibre.git
cd calibre
uv sync --all-extras --dev
jupyter notebook docs/notebooks/

Or install required dependencies:

pip install matplotlib  # only needed to plot the examples yourself

Additional Documentation Examples

Basic Usage Examples

The Basic Usage Examples section covers:

• Simple calibration workflows

• Choosing the right calibration method

• Evaluating calibration quality

• Common use cases and patterns

Advanced Usage Examples

The Advanced Usage Examples section includes:

• Multi-class calibration strategies

• Handling imbalanced datasets

• Cross-validation for calibration

• Custom calibration pipelines

Performance Benchmarks

The Performance Benchmarks section provides:

• Comparative analysis of different methods

• Performance on various dataset types

• Computational efficiency comparisons