stable_cart.bootstrap_predictions

stable_cart.bootstrap_predictions(model_factory, X_train, y_train, X_eval, task='continuous', n_bootstrap=200, random_state=None, prediction_method='predict', groups=None)[source]

Refit a model on bootstrap resamples and return every prediction it made.

This is the raw material of the Riley and Collins (2023) instability protocol: the model-building step is repeated on resamples of the training data, and each refitted model predicts for the same individuals. The summaries in bootstrap_instability() and the plots in stable_cart.stability_plots are both computed from what this returns, so a user who wants a different summary does not have to refit anything.

Parameters

model_factory

Zero-argument callable returning a fresh, unfitted estimator.

X_train

Training features to resample.

y_train

Training targets to resample.

X_eval

Fixed evaluation points, identical across resamples.

task

‘continuous’ for regression, ‘categorical’ for classification.

n_bootstrap

Number of bootstrap resamples. The default matches pminternal, the R implementation of the same protocol. Read the returned Monte Carlo standard errors rather than trusting any default: they say directly whether more resamples would move the answer.

random_state

Seed for the bootstrap samples. Randomness inside the estimator remains under model_factory; set estimator seeds there when the audit should isolate sampling variation or be exactly reproducible.

prediction_method

'predict' measures movement in predicted values or class labels. For classification, 'predict_proba' measures movement in the full probability vector, with columns aligned by the estimator’s classes_ attribute. Probability movement is summarized by squared Euclidean distance for pairwise instability and mean absolute difference for MAPE.

groups

Cluster label per training row. When supplied, clusters are resampled with replacement and each drawn cluster is taken whole, so within-cluster correlation survives the resample. Rows that are correlated in the data but resampled independently make the audit look far more stable than it is: with a cluster random effect three times the idiosyncratic noise, the row bootstrap recovered 0.19 of the true refit-to-refit variance while the cluster bootstrap recovered 0.99. Precision still comes from the number of clusters, not the number of rows.

Returns

dict[str, Any]

original — predictions of the model fitted on the full training data, shape (n_eval,) for values or labels and (n_eval, n_classes) for class probabilities; original_labels — class labels or numeric predictions from the original fit, used to score a frontier; bootstrap — predictions of each resampled model, shape (n_bootstrap, n_eval) or (n_bootstrap, n_eval, n_classes); per_point — the instability statistic for each evaluation point (variance across resamples for ‘continuous’, disagreement with the modal prediction for ‘categorical’); pairwise — the expected disagreement between two independently resampled models at each point. For ‘continuous’ this is the mean squared difference; for categorical labels, the probability that two resampled models disagree; for class probabilities, squared Euclidean distance. It is computed from every unordered pair, not an arbitrary subset; mape_per_point — mean absolute prediction error against the original model, per evaluation point; mape_standard_error and pairwise_standard_error — Monte Carlo standard errors for the two aggregate means. The pairwise error uses the delete-one jackknife for the all-pairs U-statistic; n_fit_attempts — the original fit plus every accepted bootstrap fit; n_resample_attempts — all bootstrap draws, including rejected ones; n_rejected_resamples — one-class classification draws that were redrawn because common classifiers are undefined on them; task — echoed back so downstream code need not be told again.

Raises

ValueError

If an argument is invalid or probability columns cannot be aligned.

Examples

>>> from sklearn.datasets import make_regression
>>> from sklearn.tree import DecisionTreeRegressor
>>> from stable_cart import bootstrap_predictions
>>> X, y = make_regression(n_samples=200, n_features=5, random_state=0)
>>> out = bootstrap_predictions(
...     lambda: DecisionTreeRegressor(max_depth=6, random_state=0),
...     X[:150], y[:150], X[150:], n_bootstrap=10, random_state=0,
... )
>>> out["bootstrap"].shape  # one row per resample, one column per eval point
(10, 50)
Parameters:
Return type:

dict[str, Any]