Forecast results,
with the working shown.

2024–2026 · Okta, PagerDuty and GitLab

Mean scaled absolute error. Lower is better.

2020 disruption · Box, Domo and Yext

Mean scaled absolute error. Lower is better.

What went into each forecast

1. Use financial records published by the forecast date. Select the first guidance issued with the preceding quarter’s results.
2. Compare an eight-quarter revenue trend, guidance midpoint and guidance corrected for earlier errors.
3. Calculate the correction from the median relative error of up to four earlier guidance/outcome pairs, with a 10% limit in each direction.
4. Select the method using up to five earlier forecast checks. Use the trend when the required guidance or validation is missing.
5. Score the selected forecast against the first reported quarterly revenue. Retain every selected case, including data gaps.

Scaled error divides absolute forecast error by the mean absolute quarterly revenue change over the eight preceding quarters. Each scored case has equal weight. Repeated observations from a company share risks.

How to read the results

In the first 24 cases, mean scaled error was 0.513 for trend, 0.529 for guidance and 0.336 for the selected method. Average error fell by 34.5% against trend and 36.5% against guidance. PagerDuty’s guidance was more accurate than the selected method.

The separate stress test scored 23 of 24 cases and reduced average scaled error by 18.6% against trend. Domo and Yext had lower errors with trend. Box’s October 2021 target had only preliminary prior-quarter figures at its original guidance date and remains a recorded gap.

Warning checks were possible in 45 quarters. A warning meant an estimated probability of at least 25% that revenue would fall below the original guidance floor. None of those quarters had such a shortfall. These results leave the ability to catch a shortfall open.

The companies, periods and rules were fixed before scoring each group. The stress group was chosen after the first group contained no shortfalls. Source preparation could inspect historical records. Both tests are retrospective. These records measure aggregate revenue accuracy. An estimate of the effect caused by a specific decision requires a separate comparison.

The calculation used deterministic source extraction and arithmetic, with zero paid AI calls for these two tests. Earlier AI extraction work helped develop the workflow. This test does not measure added forecast value from agent judgment.

Forecasts and decision scenarios

The software forecasts total revenue under the published business plan. Pricing, contract and investment scenarios calculate results under stated assumptions. Their difference from the forecast is not a measured causal effect.

6 November update: new model tests and source checks