Decision evaluation

Calibration estimates probabilities. Decision evaluation asks whether acting on those estimates pays off at declared costs or capacity. These functions take predictions from already fitted models; the existing calibrators, select_by_cv(), and calibration_report() remain available independently.

See Choose calibration for a decision for a complete fit, select, and test example.

class calibre.DecisionTask(*, benefit=1.0, cost=0.0, capacity=None, fraction=None)[source]

Specify one payoff objective, without averaging across scenarios.

Parameters:
benefit

Value of selecting a positive outcome; strictly positive.

Type:

float

cost

Cost per selected case, including negative outcomes.

Type:

float

capacity

Integer slots, or None for a probability threshold task.

Type:

int | None

fraction

Fraction of cases to select, rounded down to integer slots. Cannot be combined with capacity. None means no fraction constraint.

Type:

float | None

class calibre.DecisionPolicy(*, prediction=None, rule='threshold', tie_breaker=None)[source]

Specify a prediction column and an outcome-blind action rule.

Parameters:

  • prediction (str | None)

  • rule (Literal['threshold', 'rank', 'all', 'none'])

  • tie_breaker (str | None)

prediction

Key in the supplied prediction mapping; None for baselines.

Type:

str | None

rule

Probability threshold, capacity ranking, act all, or act none.

Type:

Literal[‘threshold’, ‘rank’, ‘all’, ‘none’]

tie_breaker

Optional score column for resolving capacity ranking ties. Remaining ties use the exact expectation of a uniform lottery.

Type:

str | None

calibre.decision_report(y_true, predictions, *, task, policies, reference, n_resamples=2000, interval_level=0.95, random_state=0)[source]

Compare fixed policies on common labeled cases.

Capacity means exactly k slots (fraction rounds down), even when a selected case has negative expected payoff. Rank inputs may be arbitrary finite scores. Threshold inputs must be probabilities; equality means no action. Ties are integrated out, so intervals exclude realized lottery noise. Independent-case bootstrap resampling recomputes capacity allocations within each draw. These percentile intervals are approximate, conditional on fitting and selection, and may be unreliable for small samples or unstable ranking boundaries.

Parameters:

  • y_true (ArrayLike) – Binary observed outcomes for all eligible cases.

  • predictions (Mapping[str, ArrayLike]) – Aligned prediction or score columns from fitted models.

  • task (DecisionTask) – Explicit payoff objective and optional capacity constraint.

  • policies (Mapping[str, DecisionPolicy]) – Nonempty mapping of names to fixed policy definitions.

  • reference (str) – Name of a supplied policy for paired differences.

  • n_resamples (int) – Bootstrap draws; zero disables intervals.

  • interval_level (float) – Confidence level strictly between zero and one.

  • random_state (int) – Nonnegative integer bootstrap seed.

Returns:

Values and paired differences per eligible case.

Return type:

DecisionReport

Raises:

ValueError – If outcomes, predictions, policies, or controls are invalid.

calibre.select_decision_policy(y_true, predictions, *, task, policies, reference, case_ids)[source]

Freeze the first highest-payoff policy on validation cases.

Supply predictions from models fitted without these cases. Selection includes every supplied policy, including baselines. No model is fitted or refitted.

Parameters:

  • y_true (ArrayLike) – Binary validation outcomes.

  • predictions (Mapping[str, ArrayLike]) – Aligned validation predictions from already fitted models.

  • task (DecisionTask) – Decision objective to freeze for final evaluation.

  • policies (Mapping[str, DecisionPolicy]) – Named candidate policies, in tie-preference order.

  • reference (str) – Candidate used for paired comparisons on validation and test.

  • case_ids (Sequence[Hashable]) – Unique nonmissing validation case identifiers.

Returns:

Frozen specification and validation provenance.

Return type:

DecisionSelection

calibre.evaluate_decision_policy(selection, y_true, predictions, *, case_ids, n_resamples=2000, interval_level=0.95, random_state=0)[source]

Evaluate frozen candidates on disjoint test cases without reselection.

The selected policy remains selection.selected regardless of test ranks. ID checks detect declared validation/test overlap, not leakage during fitting or misleading identifiers. All candidate comparisons are descriptive pointwise intervals, not simultaneous coverage or permission to select again on test.

Parameters:

  • selection (DecisionSelection) – Frozen validation selection.

  • y_true (ArrayLike) – Binary test outcomes.

  • predictions (Mapping[str, ArrayLike]) – Test columns from the same fitted models used in selection.

  • case_ids (Sequence[Hashable]) – Unique nonmissing test identifiers, disjoint from validation.

  • n_resamples (int) – Independent-case bootstrap draws; zero disables intervals.

  • interval_level (float) – Confidence level strictly between zero and one.

  • random_state (int) – Nonnegative integer bootstrap seed.

Returns:

Test payoffs and paired intervals with the frozen reference.

Return type:

DecisionReport

Raises:

ValueError – If validation and test identifiers overlap.

class calibre.DecisionReport(task, policies, n_observations, values, action_rates, reference, differences, intervals, interval_level, n_resamples, random_state)[source]

Store payoffs per eligible case and paired comparisons.

Parameters:
task

Objective used for every policy.

Type:

DecisionTask

policies

Frozen policy definitions.

Type:

Mapping[str, DecisionPolicy]

n_observations

Number of evaluation cases.

Type:

int

values

Mean observed payoff, averaged over any boundary lottery.

Type:

Mapping[str, float]

action_rates

Expected fraction selected.

Type:

Mapping[str, float]

reference

Policy used as the comparison baseline.

Type:

str

differences

Policy payoff minus reference payoff.

Type:

Mapping[str, float]

intervals

Paired percentile bootstrap intervals for differences.

Type:

Mapping[str, tuple[float, float]]

interval_level

Requested confidence level, or None without intervals.

Type:

float | None

n_resamples

Number of independent-case bootstrap draws.

Type:

int

random_state

Bootstrap seed.

Type:

int

class calibre.DecisionSelection(selected, validation, case_ids)[source]

Freeze the empirical validation winner and its evaluation specification.

Parameters:
selected

Name of the highest validation payoff policy; first on ties.

Type:

str

validation

Validation report, including all candidate policy definitions.

Type:

DecisionReport

case_ids

Unique validation identifiers for checking test separation.

Type:

frozenset[Hashable]