Group Testing

Heterogeneous group testing and per-group calibration.

Tests whether model strengths are constant across prompt categories (homogeneous BT) or differ by category (heterogeneous BT). If heterogeneity is warranted, fits per-category BT models and provides composable win rate predictions for any target distribution over categories.

The formal test is a likelihood-ratio test:

H0: theta_{i,k} = theta_i for all i, k (homogeneous) H1: theta_{i,k} free (heterogeneous)

Lambda = -2(l0 - l1) ~ chi2 with (K-1)(N-1) degrees of freedom.

class winference.groups.GroupTest(models, groups)

Bases: object

Likelihood-ratio test for heterogeneity across prompt groups.

Examples

>>> from winference import GroupTest
>>> comparisons = [("A", "B", True)] * 20 + [("A", "B", False)] * 20
>>> group_labels = ["math"] * 20 + ["creative"] * 20
>>> gt = GroupTest(models=["A", "B"], groups=["math", "creative"])
>>> _ = gt.fit(comparisons, group_labels)
>>> gt.test_result()["reject_at_05"]  # A wins math, B wins creative
True

Parameters:

• models (list[str])

• groups (list[str])

property group_labels: list[str]

Group label of each fitted comparison, in the order supplied.

fit(comparisons, group_labels, reg=0.0001)

Fit null (pooled) and alternative (per-group) BT models.

Parameters:

• comparisons (list[tuple[str, str, bool]]) – List of (model_a, model_b, a_wins) tuples.

• group_labels (list[str]) – Category label for each comparison, same length as comparisons.

• reg (float) – Regularisation for BT fitting.

Returns:

Self for method chaining.

Raises:

ValueError – If comparisons and group_labels have different lengths.

Return type:

Self

test_result()

Likelihood-ratio test for group heterogeneity.

Returns:

statistic (LRT statistic Lambda), df (degrees of freedom), p_value (p-value under chi2 null), reject_at_05 (bool).

Return type:

Dict with keys

Raises:

RuntimeError – If fit() has not been called.

per_group_strengths()

Return {group: {model: theta}} for each fitted group.

Return type:

dict[str, dict[str, float]]

class winference.groups.GroupCalibrator(group_test)

Bases: object

Composable win rate calibration using per-group BT models.

After fitting per-group BT models, compute win rates for any target distribution over groups:

P(i > j | pi*) = sum_k pi*_k * sigmoid(theta_{i,k} - theta_{j,k})

This is the key advantage: calibration that transfers under distribution shift.

Parameters:

group_test (GroupTest)

win_probability(model_a, model_b, target_distribution=None)

Composite win probability under a target group distribution.

Parameters:

• model_a (str) – First model name.

• model_b (str) – Second model name.

• target_distribution (dict[str, float] | None) – Dict mapping group to weight. Weights are normalised internally. If None, uses the empirical distribution from the training data.

Returns:

Composite win probability P(model_a beats model_b).

Return type:

float

win_probability_matrix(target_distribution=None)

NxN composite win probability matrix.

Parameters:

target_distribution (dict[str, float] | None)

Return type:

NDArray[float64]

sensitivity_analysis(model_a, model_b, n_draws=1000, concentration=1.0)

How much does P(a > b) vary as the target distribution changes?

Draws random target distributions from Dirichlet(concentration) and reports the range and std of the composite win probability.

Parameters:

• model_a (str)

• model_b (str)

• n_draws (int)

• concentration (float)

Return type:

dict[str, float]