Plotting

matplotlib is an optional dependency:

pip install 'calibre[plots]'

Importing calibre does not import matplotlib, and neither does importing calibre.plots. Each function imports it when first called, and raises an ImportError naming the install command if it is missing.

Conventions

Plots draw; they do not compute. Every function takes an already-computed object – a ReliabilityDiagram, a score_decomposition() result, a bands mapping. Uncertainty bands are a parameter and never an implicit flag, because consistency_bands() is a thousand PAV refits and must not fire inside an innocuous-looking .plot() call. Nothing here ever calls .fit(): fitting a calibrator on the data you are about to display is the mistake that quietly ruins calibration, so plot_calibrator_comparison() refuses an unfitted calibrator rather than fitting it for you.

Two functions are deliberate exceptions, because sweeping the computation is the plot: plot_ece_bin_sensitivity() and plot_resolution_frontier().

Axes in, axes out. Single-panel functions take ax=None and return the Axes they drew on – the very object you passed, when you passed one. Multi-panel functions take axes=None and return a Figure.

No global state. These functions never call plt.show(), never mutate rcParams, and never reach for the current figure. Use style_context() if you want publication settings applied temporarily.

Reliability diagrams

calibre.plots.plot_reliability_diagram(diagram, *, ax=None, bands=None, density='hist', density_bins=30, style='line', diagonal=True, color=None, label=None)

Draw a CORP reliability diagram.

The curve is the PAV-recalibrated conditional event probability at each distinct forecast value. Where it sits above the diagonal the forecasts were too low; below, too high. Unlike a binned reliability diagram there is no bin count to choose, so the picture cannot be changed by choosing a different one.

Parameters:

• diagram (ReliabilityDiagram) – A fitted diagram from corp_reliability().

• ax (Axes | None) – Axes to draw on. A new figure is created when omitted.

• bands (Mapping[str, np.ndarray] | Sequence[Mapping[str, np.ndarray]] | None) – Uncertainty bands from consistency_bands() or confidence_bands(), or a sequence of them to nest several levels. Bands are never computed here: they cost a thousand PAV refits, which must not happen as a side effect of drawing.

• density (str) – How to show where the observations are: "hist" (a marginal histogram below the axes), "rug" (ticks inside the axes) or "none".

• density_bins (int) – Number of bins when density="hist".

• style (str) – "line" to interpolate between the estimated points, matching as_function(), or "step" to show the PAV blocks as the step function they are.

• diagonal (bool) – Whether to draw the line of perfect calibration.

• color (str | None) – Colour of the curve. Defaults to calibre’s semantic blue.

• label (str | None) – Legend label. Omit for no legend entry.

Returns:

The axes drawn on – the very object passed as ax, when one was.

Return type:

Axes

Raises:

ValueError – If density or style is not one of the documented choices, or a band mapping is malformed.

Notes

density="hist" appends a panel below ax, which shrinks ax itself. Inside a subplot_mosaic or similar fixed grid, prefer "rug" or "none" so the layout is left alone.

Examples

>>> import matplotlib
>>> matplotlib.use("Agg")
>>> import numpy as np
>>> from calibre import corp_reliability
>>> from calibre.plots.reliability import plot_reliability_diagram
>>> rng = np.random.default_rng(0)
>>> x = rng.uniform(0, 1, 500)
>>> y = rng.binomial(1, x).astype(float)
>>> ax = plot_reliability_diagram(corp_reliability(x, y), density="none")
>>> ax.get_ylabel()
'observed event frequency'

ReliabilityDiagram.plot(**kwargs)

Draw this diagram.

Convenience wrapper around calibre.plots.plot_reliability_diagram(), which documents every keyword. Needs matplotlib: pip install 'calibre[plots]'.

Parameters:

**kwargs (Any) – Passed straight through to plot_reliability_diagram() – ax, bands, density, style, diagonal, color and label.

Returns:

The axes drawn on.

Return type:

matplotlib.axes.Axes

Examples

>>> import matplotlib
>>> matplotlib.use("Agg")
>>> import numpy as np
>>> from calibre import corp_reliability
>>> rng = np.random.default_rng(0)
>>> x = rng.uniform(0, 1, 200)
>>> y = rng.binomial(1, x).astype(float)
>>> ax = corp_reliability(x, y).plot(density="none")
>>> ax.get_xlabel()
'forecast probability'

Score decomposition

The MCB/DSC/UNC split is the thing no other Python package ships, so it gets two renderings: three comparable panels for reading the components off directly, and a plane for placing several forecasters against each other.

calibre.plots.plot_score_decomposition(decompositions, *, axes=None, score_label='Brier score', figsize=None)

Draw the MCB/DSC/UNC split as three comparable panels.

mean_score = UNC + MCB - DSC. The three terms answer three different questions, and the plot gives each its own panel, sharing the forecaster axis:

• MCB, miscalibration: what recalibration would recover. Less is better.

• DSC, discrimination: what the forecasts buy over always predicting the base rate. More is better.

• the achieved score itself.

UNC is reported in the title rather than drawn. It depends on the outcomes, not on the forecaster, so every panel would show the same number.

Notes

Separate panels rather than one stacked bar, because the terms differ in magnitude by one to two orders: for any competent model DSC is around 0.10 while MCB is around 0.001. On a shared linear axis the bar for the quantity this decomposition exists to expose is thinner than its own outline, and a stacked rendering hides it entirely behind the discrimination bar. Each panel therefore carries its own scale, and every panel starts at zero so bar lengths remain honest.

Parameters:

• decompositions (Mapping[str, object]) – One score_decomposition() result, or a mapping from forecaster name to result to draw one row each.

• axes (Sequence[Axes] | None) – Three existing axes to draw into. A new figure is created when omitted.

• score_label (str) – Name of the score, used for the third panel’s label.

• figsize (tuple[float, float] | None) – Size of the new figure. Scales with the number of forecasters by default.

Returns:

The figure holding the three panels.

Return type:

Figure

Raises:

ValueError – If a decomposition is missing a component, or axes is not length 3.

Examples

>>> import matplotlib
>>> matplotlib.use("Agg")
>>> import numpy as np
>>> from calibre import score_decomposition
>>> from calibre.plots import plot_score_decomposition
>>> rng = np.random.default_rng(0)
>>> x = rng.uniform(0, 1, 500)
>>> y = rng.binomial(1, x).astype(float)
>>> fig = plot_score_decomposition(score_decomposition(x, y))
>>> len(fig.axes)
3

calibre.plots.plot_mcb_dsc_plane(decompositions, *, ax=None, contours=True, n_contours=6, annotate=True)

Place forecasters on the discrimination-miscalibration plane.

Each forecaster is a point at (DSC, MCB). Because UNC is a property of the data rather than of the forecaster, every method on one dataset shares it, and lines of constant score are straight lines of slope 1. Down and to the right is better: more discrimination, less miscalibration.

This is the display to reach for when comparing several methods; the panels in plot_score_decomposition() are the ones for reading a single forecaster component by component.

Parameters:

• decompositions (Mapping[str, Mapping[str, float]]) – Mapping from forecaster name to a score_decomposition() result.

• ax (Axes | None) – Axes to draw on. A new figure is created when omitted.

• contours (bool) – Whether to draw lines of equal score.

• n_contours (int) – How many such lines.

• annotate (bool) – Whether to label each point with its forecaster name.

Returns:

The axes drawn on.

Return type:

Axes

Raises:

ValueError – If decompositions is empty or a decomposition is missing a component.

Examples

>>> import matplotlib
>>> matplotlib.use("Agg")
>>> import numpy as np
>>> from calibre import score_decomposition
>>> from calibre.plots import plot_mcb_dsc_plane
>>> rng = np.random.default_rng(0)
>>> x = rng.uniform(0, 1, 500)
>>> y = rng.binomial(1, x).astype(float)
>>> ax = plot_mcb_dsc_plane({
...     "honest": score_decomposition(x, y),
...     "squashed": score_decomposition(0.25 + 0.5 * x, y),
... })
>>> ax.get_xlabel()
'DSC (discrimination) -- more is better'

Resolution

What calibration cost you in granularity. A step function and a strictly increasing curve can sit on top of each other in a reliability diagram and score identically, which is exactly why isotonic regression’s resolution loss goes unnoticed.

calibre.plots.plot_resolution_loss(outputs, x=None, *, ax=None, annotate_counts=True, precision=6, sort=True)

Draw one “collapse barcode” per method.

Each method gets a horizontal strip. Inside it, one thin vertical tick marks every place along the input range where the calibrated output changes – so the number of ticks is exactly the number of distinct output values, and their spacing is where the resolution went.

Isotonic regression’s strip is sparse enough to count by eye; a resolution-preserving calibrator’s is solid ink. That contrast is the claim this package is built on, drawn rather than asserted, and it needs no legend.

Parameters:

• outputs (Mapping[str, np.ndarray]) – Mapping from method name to that method’s calibrated outputs. Every array must be the same length, being the same observations calibrated different ways.

• x (np.ndarray | None) – The input scores the outputs came from, used for the horizontal axis. When omitted, rank position is used instead.

• ax (Axes | None) – Axes to draw on. A new figure is created when omitted.

• annotate_counts (bool) – Whether to print the distinct-value count at the right of each strip.

• precision (int) – Decimal places at which two outputs count as equal.

• sort (bool) – Whether to order strips by distinct count, most granular at the top.

Returns:

The axes drawn on.

Return type:

Axes

Raises:

ValueError – If outputs is empty, the arrays disagree in length, or x does not match them.

Examples

>>> import matplotlib
>>> matplotlib.use("Agg")
>>> import numpy as np
>>> from calibre import CenteredIsotonicCalibrator, IsotonicCalibrator
>>> from calibre.plots import plot_resolution_loss
>>> rng = np.random.default_rng(0)
>>> scores = rng.uniform(0, 1, 800)
>>> labels = rng.binomial(1, scores).astype(float)
>>> ax = plot_resolution_loss({
...     "isotonic": IsotonicCalibrator().fit(scores, labels).transform(scores),
...     "centered": (
...         CenteredIsotonicCalibrator().fit(scores, labels).transform(scores)
...     ),
... }, scores)
>>> ax.get_xlabel()
'input score'

calibre.plots.plot_resolution_frontier(results, *, ax=None, errorbars=None, score_label='held-out Brier score', highlight=())

Plot held-out score against the number of distinct values retained.

The barcode invites the objection that the extra values might be noise. This answers it: methods that keep far more distinct values sit at the same height, meaning they cost nothing in score. Down is better, right is better, and the interesting finding is usually that the frontier is flat.

Parameters:

• results (Mapping[str, tuple[int, float]]) – Mapping from method name to (n_distinct, score).

• ax (Axes | None) – Axes to draw on. A new figure is created when omitted.

• errorbars (Mapping[str, tuple[float, float]] | None) – Optional mapping from method name to (low, high) absolute score bounds, for instance a bootstrap interval.

• score_label (str) – Label for the y-axis.

• highlight (Sequence[str]) – Names to draw in the accent colour.

Returns:

The axes drawn on.

Return type:

Axes

Raises:

ValueError – If results is empty or a distinct count is not positive.

Examples

>>> import matplotlib
>>> matplotlib.use("Agg")
>>> from calibre.plots import plot_resolution_frontier
>>> ax = plot_resolution_frontier({
...     "isotonic": (56, 0.1515),
...     "centered": (1874, 0.1511),
... }, highlight=["centered"])
>>> ax.get_xscale()
'log'

Comparing calibrators

calibre.plots.plot_calibrator_comparison(calibrators, x, *, ax=None, reference=None, n_grid=500, diagonal=True, annotate_distinct=True)

Overlay the calibration maps that several fitted calibrators learned.

Each calibrator is evaluated on a fine grid spanning the range of x, so what is drawn is the function itself rather than a scatter of its outputs. With annotate_distinct on, each legend entry also carries how many distinct values that calibrator produced on x – so the comparison and the resolution cost land in one figure.

Parameters:

• calibrators (Mapping[str, Any]) – Mapping from name to an already fitted calibrator exposing .transform.

• x (np.ndarray) – Input scores, used both for the grid’s range and for the distinct-value counts.

• ax (Axes | None) – Axes to draw on. A new figure is created when omitted.

• reference (ReliabilityDiagram | None) – Optional CORP diagram of the raw scores, drawn behind in grey as the empirical target the calibrators are trying to match.

• n_grid (int) – Number of grid points.

• diagonal (bool) – Whether to draw the identity line, which is what “no recalibration” would look like.

• annotate_distinct (bool) – Whether to add distinct-value counts to the legend labels.

Returns:

The axes drawn on.

Return type:

Axes

Raises:

ValueError – If calibrators is empty, x is empty, or any calibrator is unfitted.

Examples

>>> import matplotlib
>>> matplotlib.use("Agg")
>>> import numpy as np
>>> from calibre import CenteredIsotonicCalibrator, IsotonicCalibrator
>>> from calibre.plots import plot_calibrator_comparison
>>> rng = np.random.default_rng(0)
>>> scores = rng.uniform(0, 1, 600)
>>> labels = rng.binomial(1, scores).astype(float)
>>> fitted = {
...     "isotonic": IsotonicCalibrator().fit(scores, labels),
...     "centered": CenteredIsotonicCalibrator().fit(scores, labels),
... }
>>> ax = plot_calibrator_comparison(fitted, scores)
>>> ax.get_ylabel()
'calibrated probability'

Calibration error

calibre.plots.plot_ece_bin_sensitivity(y_true, y_pred, *, ax=None, n_bins=None, norm=2, estimators=('plugin', 'debiased', 'sweep'), reference=None, log_x=False)

Plot calibration error against the number of bins.

Binned calibration error is biased upward, because part of every bin’s gap is sampling noise in the label mean rather than miscalibration. The bias grows with the bin count – precisely when a finer picture of the curve is wanted. Plotting the estimators against the bin count shows this directly: the plugin curve climbs, the debiased curve does not, and any single ECE number quoted without its bin count is a point on a rising line.

All three series are computed at the same norm and on the same equal-mass, tie-safe bins, so the only thing separating them is the bias correction. Reaching for expected_calibration_error() instead would mix \(\ell_1\) with \(\ell_2\) and uniform-width bins with equal-mass ones, and the resulting picture would show three different quantities disagreeing rather than one estimator being biased.

This is one of only two plots in calibre.plots that computes anything, because sweeping the computation is the plot.

Parameters:

• y_true (np.ndarray) – Ground truth values (0 or 1).

• y_pred (np.ndarray) – Predicted probabilities.

• ax (Axes | None) – Axes to draw on. A new figure is created when omitted.

• n_bins (Sequence[int] | None) – Bin counts to evaluate. Defaults to range(2, 51).

• norm (int) – The \(\ell_p\) norm shared by every series.

• estimators (Sequence[str]) – Which of "plugin", "debiased" and "sweep" to draw. The sweep chooses its own bin count, so it appears as a horizontal line annotated with the count it settled on.

• reference (float | None) – A known true calibration error, drawn as a horizontal rule. On data that is calibrated by construction this is 0, and everything above it is bias.

• log_x (bool) – Whether to put the bin count on a log scale.

Returns:

The axes drawn on.

Return type:

Axes

Raises:

ValueError – If estimators names something unknown, n_bins is empty or holds a value below 1, or the arrays disagree in length.

Examples

>>> import matplotlib
>>> matplotlib.use("Agg")
>>> import numpy as np
>>> from calibre.plots import plot_ece_bin_sensitivity
>>> rng = np.random.default_rng(0)
>>> p = rng.uniform(0, 1, 2000)
>>> y = rng.binomial(1, p).astype(float)
>>> ax = plot_ece_bin_sensitivity(y, p, n_bins=range(2, 21), reference=0.0)
>>> ax.get_xlabel()
'number of bins'

Multiclass

calibre.plots.plot_miscalibration_profile(profile, *, ax=None, class_names=None, highlight_worst=3, show_reading=True, reading_width=72)

Show where multiclass miscalibration lives, and what to do about it.

Per-class MCB as bars, with the worst classes picked out and the spread in the title. When show_reading is on, the profile’s plain-language recommendation is printed beneath the axes.

That caption is the point. calibre is the only Python calibration package that tells you which multiclass method your data needs, and picking wrong costs about a factor of six; making the reader fetch the string separately would waste the diagnostic.

Parameters:

• profile (Mapping[str, Any]) – A miscalibration_profile() result, with mcb, spread, worst_classes and reading.

• ax (Axes | None) – Axes to draw on. A new figure is created when omitted.

• class_names (Sequence[str] | None) – Names for the classes. Defaults to their indices.

• highlight_worst (int) – How many of the worst classes to draw in the accent colour.

• show_reading (bool) – Whether to print profile["reading"] below the axes.

• reading_width (int) – Column width to wrap the reading at.

Returns:

The axes drawn on.

Return type:

Axes

Raises:

ValueError – If profile is missing a key, or class_names has the wrong length.

Examples

>>> import matplotlib
>>> matplotlib.use("Agg")
>>> import numpy as np
>>> from calibre import miscalibration_profile
>>> from calibre.plots import plot_miscalibration_profile
>>> rng = np.random.default_rng(0)
>>> truth = rng.dirichlet(np.ones(4), size=1500)
>>> y = np.array([rng.choice(4, p=t) for t in truth])
>>> ax = plot_miscalibration_profile(miscalibration_profile(truth, y))
>>> ax.get_ylabel()
'MCB (miscalibration)'

calibre.plots.plot_classwise_reliability(diagrams, *, axes=None, class_names=None, n_cols=3, density='none', figsize=None)

Draw one CORP reliability diagram per class, as small multiples.

Parameters:

• diagrams (Sequence[ReliabilityDiagram]) – The list returned by classwise_reliability().

• axes (Sequence[Axes] | None) – Existing axes to draw into, one per diagram. A new figure is created when omitted.

• class_names (Sequence[str] | None) – Titles for the panels. Defaults to class 0, class 1, …

• n_cols (int) – Panels per row when creating a new figure.

• density (str) – Passed to plot_reliability_diagram(). Defaults to "none" because appending a histogram panel to each cell of a grid distorts the layout.

• figsize (tuple[float, float] | None) – Size of the new figure. Defaults to 3 inches per panel.

Returns:

The figure holding the panels.

Return type:

Figure

Raises:

ValueError – If diagrams is empty, or axes or class_names has the wrong length.

Examples

>>> import matplotlib
>>> matplotlib.use("Agg")
>>> import numpy as np
>>> from calibre import classwise_reliability
>>> from calibre.plots import plot_classwise_reliability
>>> rng = np.random.default_rng(0)
>>> truth = rng.dirichlet(np.ones(3), size=900)
>>> y = np.array([rng.choice(3, p=t) for t in truth])
>>> fig = plot_classwise_reliability(classwise_reliability(truth, y))
>>> len(fig.axes)
3

Styling

calibre.plots.color_cycle(n)

Return n distinguishable colours, cycling if more are asked for.

Parameters:

n (int) – How many colours are needed.

Returns:

Hex colour strings.

Return type:

list of str

Raises:

ValueError – If n is negative.

Examples

>>> color_cycle(3)
['#000000', '#E69F00', '#56B4E9']

Asking for more than the palette holds wraps around rather than failing:

>>> len(color_cycle(12))
12

calibre.plots.style_context(**overrides)

Return a context manager applying publication-friendly rcParams.

Provided because some users do want global settings; making it a context manager means the settings are restored on exit, so calibre never leaves a session’s rcParams altered.

Parameters:

**overrides (Any) – Additional rcParams, overriding the defaults below.

Returns:

A context manager, as returned by

matplotlib.rc_context().

Return type:

contextlib.AbstractContextManager

Examples

>>> import matplotlib
>>> matplotlib.use("Agg")
>>> before = matplotlib.rcParams["savefig.dpi"]
>>> with style_context():
...     pass
>>> matplotlib.rcParams["savefig.dpi"] == before
True

calibre.plots.PALETTE

The Okabe-Ito qualitative palette, which is colourblind-safe. matplotlib’s default tab10 is not: its red and green are indistinguishable under deuteranopia, and a figure that compares calibration methods by colour has to survive that.

calibre.plots.SEMANTIC

Role-to-colour mapping, so that a given quantity keeps the same colour in every figure. MCB is the same red in a decomposition panel, a benchmark scatter and a notebook.

Usage

Reading one calibrator honestly

import matplotlib.pyplot as plt
import numpy as np

from calibre import consistency_bands, corp_reliability

rng = np.random.default_rng(0)
scores = rng.uniform(0, 1, 2000)
labels = rng.binomial(1, np.clip(scores**1.4, 0, 1)).astype(float)

diagram = corp_reliability(scores, labels)
bands = consistency_bands(scores, labels, level=0.9)

ax = diagram.plot(bands=bands)
ax.set_title("where the forecasts went wrong")
plt.show()

Where the model’s score actually went

from calibre import score_decomposition
from calibre.plots import plot_score_decomposition

plot_score_decomposition({
    "uncalibrated": score_decomposition(scores, labels),
    "calibrated": score_decomposition(calibrated, labels),
})

What calibration cost you

from calibre import CenteredIsotonicCalibrator, IsotonicCalibrator
from calibre.plots import plot_resolution_loss

plot_resolution_loss({
    "isotonic": IsotonicCalibrator().fit(scores, labels).transform(scores),
    "centered": (
        CenteredIsotonicCalibrator().fit(scores, labels).transform(scores)
    ),
}, scores)

One tick per distinct output value. Isotonic’s strip is sparse enough to count by eye; the centered fit’s is solid ink.