Evidence¶
There are two questions: does streamcal compute what it claims, and is that useful?
Does it compute what it claims? The formalization proves that keeping two numbers per bin fits exactly the same map as keeping the whole history. The test suite checks the implementation against a full-history reference, including forgetting, the prior, late labels, streams with only one outcome class, and reset and serialization. That is the correctness guarantee. It does not depend on winning any benchmark.
Is it useful? That is an empirical question, and this page answers it on one real stream and a set of simulations.
What we found¶
On the electricity stream, streamcal roughly halved forecast error (Brier score down 51% from the uncalibrated model). It was clearly better than 9 of the 11 alternatives. Refitting sigmoid or isotonic calibration on the most recent 336 labels did about as well: the data cannot separate them from streamcal. Streamcal stored about a fifth of their state, and under 1/400th of what the methods that refit on all history stored.
In simulations where the true probability is known, streamcal had the smallest error of the three adaptive methods in every scenario. When the model was already calibrated, every adaptive method made it slightly worse, streamcal by the least (about 5 percentage points of noise).
Choosing by validation is not a guarantee. Under a 1 ms update budget, the streamcal setting chosen on validation data took 1.01 ms on the test period, just over budget. Under a looser budget, online logistic regression won on validation but did worse than streamcal on test.
Results on the electricity stream¶
Brier score is the mean squared gap between forecast and outcome; lower is better. Change is relative to the uncalibrated model. State is the calibrator’s saved (serialized) size. Update time is the 95th-percentile time to learn from one batch of up to 336 labels.
Method |
Brier |
Change vs. uncalibrated |
State |
Update ms |
|---|---|---|---|---|
streamcal (20 bins, decay 0.5, prior 1) |
0.178 |
−51% |
1.3 KB |
1.3 |
Sigmoid, refit on recent window (last 336) |
0.179 |
−51% |
6.2 KB |
5.1 |
Isotonic, refit on recent window (last 336) |
0.182 |
−50% |
6.2 KB |
0.9 |
streamcal (20 bins, decay 0.5, prior 10) |
0.185 |
−49% |
1.3 KB |
1.0 |
Online logistic regression |
0.201 |
−45% |
702 B |
3.4 |
Isotonic, refit on all history |
0.218 |
−40% |
569 KB |
8.2 |
Sigmoid, refit on all history |
0.227 |
−38% |
567 KB |
17 |
Temperature, refit on recent window (last 336) |
0.245 |
−33% |
6.2 KB |
4.0 |
Temperature, refit on all history |
0.254 |
−30% |
567 KB |
25 |
Sigmoid, fit once, never updated |
0.285 |
−22% |
958 B |
<0.1 |
Isotonic, fit once, never updated |
0.323 |
−11% |
1.4 KB |
<0.1 |
Temperature, fit once, never updated |
0.330 |
−9.4% |
941 B |
<0.1 |
Uncalibrated |
0.364 |
— |
54 B |
<0.1 |
Streamcal against each alternative¶
Using streamcal (20 bins, decay 0.5, prior 1) instead of each alternative. A negative change means streamcal’s Brier score is lower (better), as a percentage of the alternative’s score. The range is a 95% interval from a block bootstrap that respects the order of the stream.
Instead of |
Brier change |
95% interval |
Verdict |
|---|---|---|---|
Uncalibrated |
−51% |
−63% to −39% |
streamcal better |
Temperature, fit once, never updated |
−46% |
−57% to −36% |
streamcal better |
Isotonic, fit once, never updated |
−45% |
−59% to −31% |
streamcal better |
Sigmoid, fit once, never updated |
−38% |
−48% to −27% |
streamcal better |
Temperature, refit on all history |
−30% |
−34% to −26% |
streamcal better |
Temperature, refit on recent window (last 336) |
−27% |
−32% to −23% |
streamcal better |
Sigmoid, refit on all history |
−22% |
−28% to −15% |
streamcal better |
Isotonic, refit on all history |
−19% |
−26% to −11% |
streamcal better |
Online logistic regression |
−11% |
−17% to −5.5% |
streamcal better |
Isotonic, refit on recent window (last 336) |
−2.2% |
−5.9% to +2.1% |
no clear difference |
Sigmoid, refit on recent window (last 336) |
−0.5% |
−3.7% to +3.1% |
no clear difference |
Each interval is for one comparison on its own; they are not adjusted for making eleven at once. “No clear difference” means the data cannot separate the two, not that they are equivalent.
Choosing under a budget¶
TradeoffReport.select picks the most accurate method that fits a
latency and memory budget, using only validation data. The last
column re-checks the budget on the test period, where timings can
differ from validation.
Optimizing |
Update budget |
Memory budget |
Chosen |
Test score |
Budget held on test? |
|---|---|---|---|---|---|
Brier |
0.1 ms |
2.0 KB |
Uncalibrated |
0.364 |
yes |
Brier |
1 ms |
64 KB |
streamcal (20 bins, decay 0.5, prior 10) |
0.185 |
no (1.01 ms) |
Brier |
100 ms |
64 KB |
Online logistic regression |
0.201 |
yes |
Calibration error |
0.1 ms |
2.0 KB |
Uncalibrated |
0.381 |
yes |
Calibration error |
1 ms |
64 KB |
streamcal (20 bins, decay 0.5, prior 10) |
0.069 |
no (1.01 ms) |
Calibration error |
100 ms |
64 KB |
Online logistic regression |
0.115 |
yes |
Simulations with a known answer¶
In simulated streams the true probability is known, so each method’s error can be measured directly: the typical distance between its forecast and the true probability, in percentage points (root mean squared, averaged over three seeds). Lower is better. The same settings are used in every scenario.
Scenario |
Uncalibrated |
streamcal |
Rolling isotonic |
Online logistic |
|---|---|---|---|---|
Already calibrated |
0.0 |
5.1 |
7.4 |
6.9 |
Miscalibrated, stable |
19.4 |
5.6 |
7.6 |
10.2 |
Sudden shift |
19.3 |
11.8 |
12.3 |
16.0 |
Gradual drift |
11.5 |
6.7 |
8.1 |
9.8 |
Only 1 in 10 labels seen |
19.4 |
10.5 |
11.9 |
24.2 |
Labels arrive late |
19.3 |
15.1 |
16.7 |
20.3 |
Late labels arrive between 0 and 399 events after their forecast.
Resource use in isolation¶
Each method run alone in a fresh process on 10,000 observations in batches of 100 (median of two runs). Rolling and accumulating methods refit every 16 batches here.
Method |
Array state |
Saved size |
Whole process |
Predict ms |
Update ms |
|---|---|---|---|---|---|
streamcal |
808 B |
1.3 KB |
162 MB |
<0.1 |
1.4 |
Rolling isotonic |
6.0 KB |
6.3 KB |
162 MB |
0.2 |
0.5 |
Isotonic, all history |
160 KB |
159 KB |
162 MB |
0.2 |
1.5 |
Online logistic regression |
0 B |
702 B |
161 MB |
0.8 |
1.1 |
Online logistic regression keeps its state in Python dictionaries, which the array count does not see; its saved size does. The whole-process figure is dominated by Python and its libraries, which is why it barely differs between methods. Times are 95th percentiles per call.
Provenance¶
Measured with streamcal 0.2.0 on Python 3.13.2 (macOS-27.0-arm64-arm-64bit-Mach-O). Source fingerprint 0ca06f68f9fa; the full SHA-256
is in the JSON files.
How the comparison was run¶
Data. The Elec2 electricity-price dataset (OpenML 151, version
1, MD5 8ca97867d960ae029ae3a9ac2c923d34):
45,312 half-hourly records, used in time order.
Base model. A logistic regression is fit on the first 20% and then frozen. Its probabilities are what every method calibrates.
Validation. Over the next 10%, each method warms up on the first half. Its settings are then chosen on the second half by Brier score.
Test. The final 70% is never used to choose anything. Every method predicts each batch of 336 records (one week) before seeing that batch’s outcomes, then learns from them.
Settings searched. Streamcal: 36 combinations of 20/50/100 bins, 0.5/0.9/0.99/1 decay and 1/10/100 prior weight. Rolling methods: windows of 336, 1,344 or 5,376 records, refit every 1, 4 or 16 batches. Online logistic regression (River): learning rates 0.001, 0.01 or 0.1. Frozen, full-history and uncalibrated references need no tuning. The JSON results keep every setting, including the ones that lost.
Intervals. Paired differences are resampled in blocks of 1,344 consecutive records (2,000 resamples), so the intervals account for neighboring records being alike. They describe this one stream and assume its behavior is roughly stable across blocks.
Caveats¶
Binary outcomes only. Always update with the model’s original probabilities, not calibrated ones.
Per-update decay depends on batching. Splitting the same labels into more update calls forgets faster. Half-life mode uses timestamps instead and is unaffected by how labels are grouped.
Binned calibration error depends on the bin count and can reward a forecast that ignores its input. Look at Brier score alongside it.
Ranking can change. Each fitted map preserves order, but the map changes over time, so AUROC over a whole stream can differ from the raw model’s.
Late labels are your storage. Streamcal keeps no queue of pending labels. Holding them until they arrive is the caller’s job and is outside the fixed-memory claim.
Timings are from one machine (an Apple-silicon Mac). Re-check budgets on your own hardware.
Not evidence of needing fewer labels. The learning curves in the JSON are cumulative scores on a changing stream. The formal result is about storing fewer past records, not learning from fewer labels.
Reproduce¶
The numbers above were measured at commit 492bd39. The code has changed since, so measuring again gives new numbers:
uv sync --all-groups --all-extras
make evidence
This rewrites benchmarks/results/quality.json, benchmarks/results/resources.json
and the tables on this page. The JSON records package versions, dataset
identity, split points, settings, and a fingerprint of the source and
dependency lock. The test suite does not require streamcal to win on Elec2 or
to beat any timing on CI.