stable_cart.bootstrap_instability

stable_cart.bootstrap_instability(model_factory, X_train, y_train, X_eval, task='continuous', n_bootstrap=200, random_state=None, prediction_method='predict', groups=None)[source]

Measure how much a model’s predictions move when the training data is perturbed.

This is the quantity “prediction stability” usually refers to: refit the model on bootstrap resamples of the training data and measure the spread of its predictions for the same evaluation point. Lower is better.

A model that ignores its training data scores a perfect zero here, so always read instability next to an appropriate performance measure on separate validation or test data.

Parameters

model_factory

Zero-argument callable returning a fresh, unfitted estimator.

X_train

Training features to resample.

y_train

Training targets to resample.

X_eval

Fixed evaluation points. These must not change between resamples; comparing predictions across different points measures nothing.

task

‘continuous’ for regression, ‘categorical’ for classification.

n_bootstrap

Number of bootstrap resamples. The default matches pminternal; the returned Monte Carlo standard errors say whether it is enough here.

random_state

Seed for the bootstrap samples. Estimator randomness remains under model_factory.

prediction_method

Prediction representation to compare; see bootstrap_predictions().

groups

Cluster label per training row, for correlated data; see bootstrap_predictions().

Returns

dict[str, float | int]

instability_mean, instability_p90 and instability_max over the evaluation points. For ‘continuous’ the per-point statistic is the variance of predictions; for categorical labels it is the fraction of resamples disagreeing with that point’s modal prediction. mape is Riley and Collins’s mean absolute prediction error against the model fitted on the full training data. pairwise_mean compares two independently refitted models. Monte Carlo standard errors accompany both aggregate comparison measures. Fit, draw, and one-class rejection counts expose the classification bootstrap’s conditioning.

Raises ValueError (from bootstrap_predictions()) if task is not ‘continuous’ or ‘categorical’, or n_bootstrap is below 2.

Examples

>>> from sklearn.datasets import make_regression
>>> from sklearn.tree import DecisionTreeRegressor
>>> from stable_cart import bootstrap_instability
>>> X, y = make_regression(n_samples=200, n_features=5, random_state=0)
>>> result = bootstrap_instability(
...     lambda: DecisionTreeRegressor(max_depth=6, random_state=0),
...     X[:150], y[:150], X[150:], n_bootstrap=10, random_state=0,
... )
>>> 0.0 <= result["mape_standard_error"] < result["mape"]
True
Parameters:
Return type:

dict[str, float | int]