How accurate our forecasts are

Method: Protocol v7.4. Evaluation fold B: history cut at the panel anchor minus 5 years, 5 years scored out of sample. The anchor is the newest year at least 50% of an indicator's country series reach, so one fast country cannot move the window for the rest. Selection fold A: history cut at the panel anchor minus 10 years, 5 years forecast. Used only to choose each indicator's mix of models and to size its interval. Its last target year is fold B's as-of year, so nothing chosen on A had seen B's outcomes. Interval widths come from fold A and the older windows anchor-15, anchor-20; published intervals add fold B. The headline accuracy is a 5-year out-of-sample test on history up to each indicator's as-of year, not a check on the published 10-year forecasts. Fold B also decides which v8 (driver-based) indicators are published, so the fold-B score of a published v8 indicator is a validation result, not an independent test (ml_v7/engine.py). The prospective forecast ledger is the independent test. Chronos-2 and TimesFM are pretrained by their vendors on public data that may already contain part of the history they are scored on. The local fit respects the as-of cut; the pretraining corpus is outside our control. Inputs are pseudo-real-time: today's revised data, not the vintage that was available in the as-of year. A revision published later can flatter or punish a model.. Reliability basis: skill on the latest 5-year holdout vs a 'no change' forecast; high = errors at least 15% smaller on 3+ series; never high when the data stopped arriving.

1. Tested on past data

2. Published and waiting

Every published forecast is written down with its year and its range before the year arrives, and the register of those forecasts is public. A pending forecast is not a hit, and it is never counted as one.

3. When we can start checking

The 80% likely ranges held the real value 74.1% of the time for yearly forecasts and 73.6% for monthly ones (target 80%): the ranges are a little too narrow, and we say so. The 5-year holdout is what those numbers were measured on, so they are not proof for a forecast ten years out.

Where our forecasts are weak

For this year's inflation, Satoria's monthly update of the IMF forecast missed 34% less than the IMF's April forecast in September tests (21 years, 50 countries; smaller average absolute error, same month, monthly data in today's revised versions). The advantage comes from newer data, not from a smarter model with the same information, and a simple mechanical estimate does about as well in September. See the full comparison.

The detail, if you want it

Data: https://satoria-pipeline-production.up.railway.app/api/site/stats · published story forecasts and their settlement: https://satoria-pipeline-production.up.railway.app/api/story-resolutions

All stories · Countries · The map of the future · Why trust Satoria · Engine · About · Satoria vs the IMF · How accurate we are · Method · Indicators · For AI agents