Examples and Tutorials

This section provides comprehensive examples and tutorials for using Calibre.

• Basic Usage Examples
  • Complete Workflow Example
  • Handling Different Data Types
  • Visualization Examples
  • Cross-Validation for Calibration
  • Common Pitfalls and Solutions
• Advanced Usage Examples
  • Multi-Class Calibration
  • Custom Calibration Pipelines
  • Handling Concept Drift
  • Calibration for Specific Domains
  • Performance Optimization
• Performance Benchmarks
  • Interactive Performance Notebook
  • Method Comparison Summary

Interactive Jupyter Notebooks

We provide focused, executable Jupyter notebooks for hands-on learning:

Interactive Examples

• Getting Started with Calibre
• Validation and Evaluation
• Plateau Diagnostics Demo
• Performance Comparison

Notebook Overview

📚 Getting Started (Getting Started with Calibre)

• Basic calibration workflow with realistic ML predictions

• Choosing the right calibrator for your data

• Visual validation with reliability diagrams

• Quick start guide for new users

🔍 Validation and Evaluation (Validation and Evaluation)

• Comprehensive calibration quality assessment

• Mathematical property validation (bounds, monotonicity, granularity)

• Performance across different miscalibration patterns

• Edge case testing and robustness analysis

🩺 Diagnostics and Troubleshooting (Plateau Diagnostics Demo)

• Plateau diagnostic tools for isotonic regression

• Distinguishing genuine vs. limited-data flattening

• Bootstrap stability analysis and progressive sampling

• Decision framework for method selection

⚡ Performance Comparison (Performance Comparison)

• Systematic comparison across all calibration methods

• Performance on overconfident, underconfident, and distorted predictions

• Computational efficiency and method ranking

• Guidelines for choosing the optimal method

Running the Notebooks

To run these notebooks locally:

git clone https://github.com/finite-sample/calibre.git
cd calibre
pip install -e ".[dev]"
jupyter notebook docs/source/notebooks/

Or install required dependencies:

pip install calibre[examples]  # Installs matplotlib, seaborn, pandas

Additional Documentation Examples

Basic Usage Examples

The Basic Usage Examples section covers:

• Simple calibration workflows

• Choosing the right calibration method

• Evaluating calibration quality

• Common use cases and patterns

Advanced Usage Examples

The Advanced Usage Examples section includes:

• Multi-class calibration strategies

• Handling imbalanced datasets

• Cross-validation for calibration

• Custom calibration pipelines

Performance Benchmarks

The Performance Benchmarks section provides:

• Comparative analysis of different methods

• Performance on various dataset types

• Computational efficiency comparisons