Calibration Metrics

Calibration metrics for pairwise win rate predictions.

When your model says “A beats B with probability 0.65”, does A actually win 65% of the time? These tools check.

winference.calibration.expected_calibration_error(predicted, observed, n_bins=10)[source]

Expected Calibration Error (ECE).

Parameters:

  • predicted (NDArray[float64]) – Array of predicted win probabilities, shape (n,).

  • observed (NDArray[float64]) – Array of binary outcomes (0 or 1), shape (n,).

  • n_bins (int) – Number of bins for grouping predictions.

Returns:

Weighted average |predicted - observed| across bins.

Return type:

float

winference.calibration.brier_score(predicted, observed)[source]

Brier score: mean squared error of probability predictions.

Lower is better. Perfect calibration → 0. Random guessing → 0.25.

Parameters:

  • predicted (NDArray[float64])

  • observed (NDArray[float64])

Return type:

float

winference.calibration.log_loss(predicted, observed)[source]

Binary cross-entropy loss. Lower is better.

Parameters:

  • predicted (NDArray[float64])

  • observed (NDArray[float64])

Return type:

float

winference.calibration.reliability_diagram(predicted, observed, n_bins=10, ax=None, label='', color=None)[source]

Reliability diagram data and optional plot.

Parameters:

  • predicted (NDArray[float64]) – Array of predicted probabilities.

  • observed (NDArray[float64]) – Array of binary outcomes.

  • n_bins (int) – Number of bins.

  • ax (Any) – Matplotlib Axes. If provided, plot on it.

  • label (str) – Label for the legend.

  • color (str | None) – Color for the plot line.

Returns:

Dict with ‘bin_midpoints’, ‘bin_accuracy’, ‘bin_counts’, ‘ece’.

Return type:

dict[str, Any]

winference.calibration.compare_calibration(methods, observed, n_bins=10)[source]

Compare calibration across multiple prediction methods.

Parameters:

  • methods (dict[str, NDArray[float64]]) – Dict mapping method name to predicted probabilities.

  • observed (NDArray[float64]) – Binary outcomes.

  • n_bins (int) – Number of bins for ECE calculation.

Returns:

{ece, brier, logloss}}.

Return type:

Dict of {name