Evidence: expected results and useful trade-offs¶
Correctness means obtaining the expected result for the stated objective. The formalization proves that weighted bin counts and label totals preserve the complete binned isotonic objective. Exact-reference tests check its implementation with forgetting, the identity prior, late labels, single-class data, and reset/serialization. This is a stronger foundation than requiring the package to win an empirical benchmark.
When changing bins or forgetting, the objective itself changes. The useful
question is then which method gives the lowest chosen error within the user’s
resource budget. TradeoffReport.select minimizes Brier, log loss, or binned
calibration error subject to p95 prediction/update latency and serialized-state
limits. The user supplies those limits; no universal equivalence margin is
assumed. A report can also return no feasible configuration.
Protocol¶
The Elec2 dataset is OpenML 151, version 1, MD5
8ca97867d960ae029ae3a9ac2c923d34, with 45,312 chronological records.
Fit a frozen logistic base model on the first 20%.
Use the first half of the next 10% to initialize every candidate. Frozen references retain that fitted map; other methods continue updating.
Score the second half of that validation period in predict-then-observe order. Select one setting per predeclared family by Brier and select operating points by their stated objectives and budgets.
Reconstruct each selected candidate from the entire validation period, then evaluate it on the final 70%. No final-period labels select settings.
The fixed search includes 36 streamcal settings (20/50/100 bins, 0.5/0.9/0.99/1 decay, 1/10/100 prior weight); rolling references use 336/1,344/5,376-observation windows and refit every 1/4/16 batches. Online logistic uses River SGD with learning rates 0.001/0.01/0.1. There are also raw, frozen, and accumulating references. Each scored batch has up to 336 rows. The complete search results are retained, including losing configurations.
Paired Brier intervals use arch circular block bootstrap over 1,344 consecutive
test observations and 2,000 replicates. This preserves pairing and weights
individual observations equally. These are descriptive dependence-aware
intervals from one stream; stationarity and block length affect their validity.
An interval containing zero does not prove equivalence.
Interpreting the result¶
The stronger rolling references largely close the Brier gap from the original accumulating-only comparison. Streamcal and tuned rolling sigmoid are close on this stream, while streamcal retains less state. Direct rolling isotonic can also be faster than streamcal in individual timing runs. Online logistic has smaller serialized state than streamcal and wins some validation selections, but performs worse on the final period. These results show why an honest operating-point selector cannot always choose streamcal. Validation-feasible choices can exceed a tight latency budget on the final period; the generated table records those failures instead of treating the validation measurement as a runtime guarantee.
Controlled experiments report distance to a known conditional probability, alongside proper scores. They include cases where the raw model is already correct and where adaptation adds estimation noise. Sparse-label experiments still score all forecasts offline but reveal only a subset of labels for updates. Delayed experiments use River’s event scheduling and prediction-age weights. Fixed settings across these scenarios expose weaknesses rather than retuning each method on its test data.
The learning curves in the JSON show cumulative test-period scores. They do not establish that fewer labels are needed: the methods saw a common warm-up period, and the underlying distribution changes across the curve. The formal claim is fewer retained historical records for the same binned objective.
Reproduce¶
The committed measurements are a recorded snapshot from candidate 492bd39. The release changes that measured source only to accept a zero-byte selection budget and document that boundary; its test settings also enforce strict configuration, markers, expected failures, and warnings. None of the benchmark configurations uses a zero-byte budget. The JSON fingerprints identify the measured snapshot, not the final release; they have not been relabeled. Check out that candidate to reproduce its implementation, or run the following on the release to collect new measurements:
uv sync --all-groups --all-extras
make evidence
This writes benchmarks/results/quality.json, benchmarks/results/resources.json,
and the tables included below. Results carry package versions, dataset identity,
split points, settings, and a SHA-256 fingerprint of implementation files and
the dependency lock. The resource sweep runs methods serially in fresh Python
processes. Measurements describe this machine and invocation; neither p95
latency nor serialized size is a worst-case execution or process-memory bound.
The evidence workflow archives the JSON. Ordinary tests check numerical identities and behavior; they do not require superiority on Elec2 or a timing win on a shared CI runner.
Chronological Elec2 results¶
Method |
Brier |
Array bytes |
Serialized bytes |
p95 predict ms |
p95 update ms |
|---|---|---|---|---|---|
accumulating-isotonic |
0.2183 |
583,400 |
582,382 |
0.384 |
8.171 |
accumulating-sigmoid |
0.2273 |
580,032 |
580,978 |
1.389 |
16.595 |
accumulating-temperature |
0.2544 |
580,032 |
580,961 |
1.263 |
24.658 |
frozen-isotonic |
0.3234 |
1,544 |
1,434 |
0.222 |
0.044 |
frozen-sigmoid |
0.2849 |
32 |
958 |
1.134 |
0.045 |
frozen-temperature |
0.3300 |
32 |
941 |
0.824 |
0.048 |
online-logistic/lr=0.1 |
0.2008 |
0 |
702 |
2.329 |
3.350 |
raw |
0.3642 |
0 |
54 |
0.013 |
0.074 |
rolling-isotonic/w=336/r=1 |
0.1818 |
5,896 |
6,302 |
0.261 |
0.873 |
rolling-sigmoid/w=336/r=1 |
0.1787 |
5,408 |
6,338 |
1.304 |
5.098 |
rolling-temperature/w=336/r=1 |
0.2450 |
5,408 |
6,321 |
1.041 |
3.983 |
stream/b=20/d=0.5/p=1.0 |
0.1778 |
808 |
1,308 |
0.039 |
1.272 |
stream/b=20/d=0.5/p=10.0 |
0.1850 |
808 |
1,308 |
0.040 |
1.011 |
Times measure calls on batches of up to 336 observations. Array bytes include upstream fitted models. Python scalar state is reflected in serialized size. Serialization is not process memory.
Reference |
Brier difference |
Paired 95% interval |
|---|---|---|
accumulating-isotonic |
-0.0405 |
[-0.0574, -0.0251] |
accumulating-sigmoid |
-0.0494 |
[-0.0644, -0.0338] |
accumulating-temperature |
-0.0765 |
[-0.0874, -0.0651] |
frozen-isotonic |
-0.1456 |
[-0.1916, -0.1004] |
frozen-sigmoid |
-0.1071 |
[-0.1379, -0.0771] |
frozen-temperature |
-0.1522 |
[-0.1876, -0.1176] |
online-logistic/lr=0.1 |
-0.0229 |
[-0.0349, -0.0111] |
raw |
-0.1864 |
[-0.2312, -0.1437] |
rolling-isotonic/w=336/r=1 |
-0.0040 |
[-0.0107, 0.0038] |
rolling-sigmoid/w=336/r=1 |
-0.0008 |
[-0.0065, 0.0055] |
rolling-temperature/w=336/r=1 |
-0.0672 |
[-0.0779, -0.0567] |
Differences are streamcal minus reference. Intervals are marginal, not simultaneous across methods. Crossing zero establishes neither superiority nor equivalence.
Constrained validation choices¶
Every row requires p95 prediction calls <=100 ms. Update budgets also apply to p95 calls. Choices use validation data. The final column checks the budgets again on test.
Objective |
Update budget ms |
State budget bytes |
Choice |
Test objective |
Test budget met |
|---|---|---|---|---|---|
brier |
0.1 |
2,048 |
raw |
0.3642 |
True |
brier |
1.0 |
65,536 |
stream/b=20/d=0.5/p=10.0 |
0.1850 |
False |
brier |
100.0 |
65,536 |
online-logistic/lr=0.1 |
0.2008 |
True |
binned_calibration_error |
0.1 |
2,048 |
raw |
0.3812 |
True |
binned_calibration_error |
1.0 |
65,536 |
stream/b=20/d=0.5/p=10.0 |
0.0689 |
False |
binned_calibration_error |
100.0 |
65,536 |
online-logistic/lr=0.1 |
0.1147 |
True |
Controlled expected-result experiments¶
Mean squared distance to the known conditional probability over three seeds. Lower is better. Settings are fixed across scenarios. Sparse cases reveal one in ten labels; delayed cases reveal them 0-399 events later. No settings are chosen from these displayed results.
Scenario |
Raw |
Streamcal |
Rolling isotonic |
Online logistic |
|---|---|---|---|---|
calibrated |
0.00000 |
0.00258 |
0.00546 |
0.00470 |
stationary |
0.03752 |
0.00315 |
0.00578 |
0.01049 |
abrupt |
0.03718 |
0.01387 |
0.01514 |
0.02573 |
gradual |
0.01328 |
0.00449 |
0.00658 |
0.00968 |
sparse |
0.03752 |
0.01098 |
0.01416 |
0.05875 |
delayed |
0.03718 |
0.02267 |
0.02776 |
0.04120 |
Isolated resource measurements¶
Medians of two fresh processes: 10,000 observations, batches of 100. Rolling and accumulating references refit every 16 batches here.
| Method | Arrays bytes | Serialized bytes | Peak process MiB | p95 predict ms | p95 update ms | |—|—:|—:|—:|—:|—:|—:| | stream | 808 | 1308 | 161.6 | 0.043 | 1.406 | | rolling | 6152 | 6430 | 161.9 | 0.230 | 0.494 | | accumulating | 163528 | 162446 | 162.1 | 0.227 | 1.542 | | online-logistic | 0 | 702 | 161.4 | 0.774 | 1.120 |
Peak RSS includes interpreter, common imports, model and temporaries. The JSON also records baseline RSS, single-event calls, batches of 1,000, stream lengths 1,000 and 10,000, and 100,000-observation runs with batches of 1,000.
Provenance¶
Python 3.13.2. macOS-27.0-arm64-arm-64bit-Mach-O
Quality source SHA-256: 0ca06f68f9fa16f09d293355242985d7c951fa9a269e61ab6360b037ad5855d0.
Resource source SHA-256: 0ca06f68f9fa16f09d293355242985d7c951fa9a269e61ab6360b037ad5855d0.
Limits and decision rules¶
Binary probabilities only. Labels must correspond to the original forecasts.
Per-update decay depends on batch cadence. Prediction-age half-life uses explicit timestamps and is invariant to regrouping available labels at a fixed final time, up to floating point arithmetic.
Bin resolution changes the approximation. Equality to the binned objective does not imply equality to unbinned isotonic regression.
Binned error depends on its bin count and can reward uninformative forecasts. Keep proper scores visible even when choosing that diagnostic as the objective.
AUROC across a changing calibration map need not equal raw AUROC, even though every individual map is monotone.
The calibrator owns no pending-label queue. The offline evaluator and caller’s delayed-label storage are outside the bounded-state claim.
Select on validation, evaluate once on a later period, and recheck resource budgets on the deployment hardware. Validation winners can lose out of sample.