Chapter 6: Time Series Models for Trading and Risk
Chapter Introduction
Every chapter so far has treated an observation as a row that could, in principle, be shuffled: a borrower in Chapter 4, a draw from a posterior in Chapter 5. This chapter is about data in which the order of the rows is the information. A price at 10:01 is not an independent draw from the same urn as the price at 10:00; it is the 10:00 price plus a small shock. Once you take that seriously, almost every habit from the earlier chapters has to be re-examined. A regression of one wandering series on another produces an R² of 0.80 from nothing. A random K-fold split lets the future train the past. A standard error computed as if the rows were independent is wrong by a factor you cannot see. Time-series statistics is the discipline that keeps those errors out of your work — and, as you will see, it is also a toolkit that still earns real money on a trading desk, provided you point it at the right jobs.
Which jobs those are is the message of the chapter, and it is worth stating at the outset because it shapes every section. Time-series models remain important in quantitative trading, but their role has shifted. They have moved to volatility modelling — every risk limit, every position size and every Value-at-Risk number is a forecast of tomorrow’s variance, and a fifteen-line GARCH estimator tracks that variance better than anything else of comparable simplicity. They have moved to structural arbitrage — cointegration, Kalman filters and regime switching, where the object you trade is not a direction but a stationary relationship between prices, or a hidden state that flips between calm and stormy. And they have become the foundation of the machine-learning pipeline: stationary features, lags that do not leak, walk-forward validation and a baseline that must be beaten are all time-series ideas, and a gradient-boosting model that skips them is a model that memorises 2021. What time-series models have largely lost is the job most textbooks still lead with — forecasting tomorrow’s mean return from a single series’ own past. That job has gone to machine learning on the cross-section, and Section 6.2 shows you, with honest out-of-sample numbers, exactly why.
The chapter is organised so that the foundations come first and the money comes later. Section 6.1 builds the stationarity reflex: what a unit root is, why prices remember and returns forget, how the KPSS and ADF tests must be read as a pair, how the ACF and PACF reveal a series’ memory, and how an AIC-chosen ARIMA serves as the baseline every fancier model must beat — and fails to beat on daily returns. Section 6.2 scores forecasts honestly (MAE, RMSE, directional accuracy, and why MAPE is unusable on returns), watches an AIC-winning ARIMA lose money out of sample, and then pivots to what a modern ML pipeline inherits from time series, with a gradient-boosting model trained the way a desk would train it. Section 6.3 introduces cointegration through the trap of spurious regression, tests two pairs that have no rope and one that does, and builds a walk-forward pairs book on Alphabet and the S&P 500. Section 6.4 treats the hedge ratio and the market regime as hidden states — a six-line Kalman filter, a time-varying beta, and a two-regime Markov-switching model that cuts a drawdown by three quarters. Sections 5.5 and 5.6 model the variance directly: ARCH and GARCH by maximum likelihood, multi-step forecasts, Value-at-Risk with a Kupiec back-test through March 2020, the GJR leverage term, ARMA+GARCH, and the volatility-targeting overlay that most systematic funds run on every position.
By the end of the chapter you will be able to look at any series and say, with test statistics, whether a model is allowed to touch it; to score a forecast against the baseline that actually matters; to tell a real long-run relationship from a spurious one; to filter a hidden level, slope or regime one observation at a time; and to turn a variance forecast into a risk limit or a position size that you can defend to a risk committee. You will also be able to explain — with the numbers from this chapter — why none of these tools should be used to forecast the sign of tomorrow’s return.
The data are real throughout: one day of Apple one-minute bars, ten years of Apple, S&P 500 and Dow daily closes, Alphabet against the S&P 500 over 2015–2020, NVIDIA against SPY over 2023–2024, the classic airline-passenger series and the FRED-MD macro panel. Every cell runs in your browser; every number in the prose is printed by a cell, so you can change a window or a threshold and see what moves. The companion slide deck for this chapter follows the same six sections with the same data, seeds and numbers, so the two can be read side by side.
Table of Contents
- Foundations: Stationarity, Unit Roots, ACF/PACF, ARIMA as a Baseline
- Why Mean Forecasting Lost to ML: Forecast Accuracy, Walk-Forward Discipline, Time Series as the Base of the ML Pipeline
- Structural Arbitrage I: Cointegration and Pairs Trading
- Structural Arbitrage II: Kalman Filters and Regime Switching
- Volatility I: ARCH, GARCH(1,1), Forecasts and VaR
- Volatility II: Leverage (GJR), ARMA+GARCH, and Volatility Targeting
Foundations: Stationarity, Unit Roots, ACF/PACF, ARIMA as a Baseline
You will look at one trading day of Apple twice — as prices and as returns — and decide which of the two a model is allowed to touch. Then two tests with opposite null hypotheses give you a verdict you can defend, the autocorrelation function tells you how much memory a series has, and an information criterion picks an ARIMA order — which you will hold out and score, and discover that order selection is not validation.
One day of Apple: prices remember, returns forget
A series is (weakly) stationary if its mean, its variance and its autocovariances \(\text{Cov}(y_t, y_{t-k})\) do not depend on \(t\). The definition sounds technical; the consequence is practical. A model fitted to a series estimates one set of parameters and applies them at every date. If the process that generated the first half of the sample is not the process that generated the second half — if the “typical value” at 10:00 is different from the typical value at 15:00 — then the estimate refers to nothing, and any forecast built on it is a guess dressed as a number. Stationarity is the licence that lets a single set of parameters describe a whole history.
The quickest way to see what stationarity looks like is to look at the same day twice. The file holds 390 one-minute bars of Apple for 2 November 2020. The first cell loads the close, computes the minute-by-minute return, and plots both.
There are 389 rows after dropping the first return. The price opens near 109.6, climbs above 110.5 in the first hour, and drifts down to 108.8 by the close. Wherever it is at 10:00 tells you a great deal about where it will be at 10:01 — the level has memory, and its typical value in the morning is not its typical value in the afternoon. The return, by contrast, is a flat cloud around zero: mean -0.000018, standard deviation 0.00108, with no visible drift in either the centre or the spread. The price is the cumulative sum of these returns, and summing a stationary series is exactly what manufactures a wandering level. That one sentence is the whole of unit-root theory; the rest of the section makes it precise.
The knife-edge: \(a = 1\) versus \(a = 0.8\)
Write the simplest dynamic model, a first-order autoregression:
\[X_t = a\,X_{t-1} + \varepsilon_t, \qquad \varepsilon_t \sim \text{iid}(0, \sigma^2).\]
Substitute the equation into itself and \(X_t = \varepsilon_t + a\varepsilon_{t-1} + a^2\varepsilon_{t-2} + \dots\): today’s value is a weighted sum of every past shock, with weights \(a^k\). If \(|a| < 1\) the weights decay geometrically, old shocks are forgotten, and the variance converges to \(\sigma^2(1 + a^2 + a^4 + \dots) = \sigma^2/(1-a^2)\) — a constant, so the series is stationary. If \(a = 1\) every weight is one, no shock is ever forgotten, and \(\text{Var}(X_t) = t\sigma^2\) grows without bound. That is the random walk, and \(a = 1\) is a unit root. The change \(X_t - X_{t-1} = \varepsilon_t\) is pure noise; the level remembers everything.
The cell feeds identical shocks (seed 1, 200 steps) into \(a = 1\) and \(a = 0.8\), and also generates a trend-stationary series for the next subsection.
Same 200 shocks, two fates. The random walk wanders to -15.8 with a standard deviation of 5.27; the \(a = 0.8\) series never leaves the band [-3.1, 3.1] and has standard deviation 1.15 (its theoretical value is \(1/\sqrt{1 - 0.64} = 1.67\); the sample happens to sit below it, but it is bounded). The difference between \(a = 0.99\) and \(a = 1\) is the difference between a model and a lottery — and, as the tests below will show, it is also a difference that is genuinely hard to detect in a few hundred observations.
Two flavours of non-stationarity, two cures
Non-stationarity comes in two textbook flavours, and they need different treatments. A trend-stationary series is \(y_t = g(t) + X_t\): a deterministic function of time plus a stationary component. The cure is to detrend — regress on \(t\) and keep the residual. A difference-stationary series is \(y_t = y_{t-1} + \varepsilon_t\), the random walk (possibly with drift). The cure is to difference — work with \(\Delta y_t = y_t - y_{t-1}\). The instructor’s shorthand from the lecture is worth memorising: trend-stationary versus unit root is \(a < 1\) around a trend versus \(a = 1\), and the verbs are “detrend” and “difference”.
Applying the wrong cure is not harmless. The cell above built \(y_t = 0.5t + 3\varepsilon_t\) with the same seed and applied both. Detrending recovers the shock scale: the residual has standard deviation 2.77, close to the true 3, and the fitted slope 0.499 is essentially the true 0.5. Differencing the same series gives \(\Delta y_t = 0.5 + 3(\varepsilon_t - \varepsilon_{t-1})\), a stationary but over-differenced series with standard deviation 4.06 — close to \(3\sqrt{2} = 4.24\), twice the shock variance, and with a moving-average unit root that any ARMA fit will struggle with. Differencing the random walk, by contrast, gives 0.93: exactly the shocks. Match the cure to the disease, and to know the disease you need a test.
KPSS and ADF: two tests with opposite nulls
Eyeballing a plot is not evidence, and the two standard tests are designed around opposite null hypotheses, which is why they must be read together.
The KPSS test (Kwiatkowski, Phillips, Schmidt and Shin, 1992) takes stationarity as the null. It regresses the series on a constant (regression="c") or on a constant and a linear trend ("ct"), forms the partial sums of the residuals, and asks whether those partial sums grow like a random walk. Under the null they do not, and the statistic is small; under a unit root they do, and the statistic is large. A small p-value therefore means reject stationarity. One quirk: statsmodels reports the KPSS p-value only inside the interval \([0.01, 0.10]\), because the critical values are tabulated; 0.01 means “at most 0.01” and 0.1 means “at least 0.1”.
The Augmented Dickey–Fuller test (ADF) takes the unit root as the null. It estimates
\[\Delta y_t = \alpha + \beta t + \gamma\, y_{t-1} + \sum_{i=1}^{k} \delta_i\, \Delta y_{t-i} + u_t\]
and tests \(\gamma = 0\), which is \(a = 1\) in the AR(1) notation. The lagged differences (“augmented”) soak up short-run autocorrelation so that \(u_t\) is white noise; autolag="AIC" chooses \(k\). The \(t\)-statistic on \(\gamma\) does not have a \(t\) distribution under the null — it has the Dickey–Fuller distribution, which is why the critical values are more negative than \(-1.96\). A small p-value means reject the unit root. Again "c" includes only the constant and "ct" adds the trend.
The cell wraps both in two small helpers and runs them on the simulated series and on Apple’s minute bars, then on two US macro series from FRED-MD (the first row of that file holds transformation codes and must be dropped).
Take the lines in turn. On the trend series, KPSS around a constant gives statistic 2.102 and p 0.01 — stationarity around a constant is rejected, correctly, because a trending series has no constant mean. Around a trend the statistic drops to 0.058 with p 0.1: trend-stationarity is not rejected. The ADF tells the complementary story: with only a constant it fails to reject the unit root (p 0.95), because a trend looks like drift to a test that is not allowed to fit one; with "ct" it rejects decisively (p 0.0000). The random walk is unambiguous — ADF p 0.81 (unit root not rejected) and KPSS p 0.01 (stationarity rejected) — and so is the AR(0.8): ADF p 0.0000, KPSS p 0.1.
The two results combine into a decision table that you should apply to every new series:
| ADF rejects (p < 0.05) | ADF does not reject | |
|---|---|---|
| KPSS does not reject (p ≥ 0.05) | Stationary — model it | Not enough data to tell (low power) |
| KPSS rejects (p < 0.05) | Conflict: trend-stationary or structural break — try "ct" |
Unit root — difference it |
The ADF alone cannot distinguish “stationary” from “too little data to reject the unit root” — its power against near-unit-root alternatives such as \(a = 0.97\) is notoriously low in samples of a few hundred. The KPSS alone cannot distinguish “unit root” from “deterministic trend”. The pair, with both "c" and "ct", covers the cases; either test alone leaves a blind spot.
Apple’s minute price lands in the conflict cell: KPSS p 0.01 rejects stationarity, while ADF p 0.035 rejects the unit root at 5 % but not at 1 %. With 389 minutes of a series that drifts about one dollar, the ADF is picking up mild mean reversion around the day’s level while the KPSS says the level is not constant. The honest verdict is near-unit-root: do not model the level, model the returns — which pass both tests without any ambiguity (KPSS p 0.1, ADF statistic -25.07, p 0.0000). The macro series give the two clean cases. The CPI level has p 0.9986 with a constant and 0.77 even with a trend — a unit root, which is why macro models use inflation (the 12-month change of the CPI) rather than the CPI level. The unemployment rate has p 0.011: it is stationary in levels, because unemployment goes up and comes back down. Remember this in Section 6.2 — a feature with a unit root goes into a machine-learning model only after it has been differenced.
A complementary, model-free diagnostic used on trading desks is the Hurst exponent \(H\), defined by \(\text{std}(y_{t+k} - y_t) \propto k^H\): regress \(\log \text{std}\) of \(k\)-period changes on \(\log k\) and read the slope. A random walk has \(H = 0.5\); \(H > 0.5\) means the series trends (variance grows faster than linearly in the horizon) and \(H < 0.5\) means it mean-reverts. On short daily samples the simple estimator is biased downward (SPY in 2024 gives about 0.38, NVDA 0.49), so read it as a ranking of persistence across assets rather than as an absolute verdict; the lecture notebook used isom5650.timeseries.hurst_exponent on late-2024 NVDA, AAPL and GLD prices and found all three above 0.7 — a strongly trending quarter.
The course package wraps both tests with a printed table. In Colab, isom5650.timeseries.kpss_test(series, null="c") and isom5650.timeseries.adf_test(series, reg="c") return a two-row Series of statistic and p-value; the numbers agree with the helpers above (Apple’s minute close: KPSS 2.081, p 0.010; ADF −3.000, p 0.035).
The conflict cell: KPSS rejects stationarity and ADF rejects the unit root (at 5 %, not at 1 %). It usually means trend-stationarity or a structural break — re-run both with “ct”; if the conflict survives, treat the series as near-unit-root and model its differences rather than its level, as with Apple’s minute price.
Autocorrelation: a series against its own past
The autocorrelation function (ACF) at lag \(k\) is \(\rho_k = \text{corr}(y_t, y_{t-k})\): how much of today is explained by \(k\) periods ago. The partial autocorrelation function (PACF) at lag \(k\) is the correlation between \(y_t\) and \(y_{t-k}\) after removing the influence of lags \(1, \dots, k-1\) — the coefficient on \(y_{t-k}\) in a regression of \(y_t\) on its first \(k\) lags. The pair is the classic Box–Jenkins diagnostic, and its two plots carry a confidence band at \(\pm 2/\sqrt{n}\): under white noise, sample autocorrelations are approximately normal with standard error \(1/\sqrt{n}\), so a bar inside the band is indistinguishable from zero.
For a global test of “is there any autocorrelation up to lag \(m\)?” the Ljung–Box statistic \(Q(m) = n(n+2)\sum_{k=1}^{m} \hat\rho_k^2/(n-k)\) is distributed \(\chi^2_m\) under the null of no autocorrelation; it appears in Section 6.5 where it is used on squared returns.
The classic teaching series is the monthly airline-passenger count, 144 months from 1949 to 1960. Before running the cell, ask yourself which is larger: the ACF at lag 6 or at lag 12? Lag 6 is closer in time, but lag 12 is the same month one year earlier.
The ACF reads 0.95, 0.68, 0.76 at lags 1, 6 and 12: a slow decay, which is the signature of a trend, with a bump at lag 12 that is higher than lag 6 even though it is further away. Passengers in July 1958 look like July 1957, not like January 1958 — a rising ACF at the seasonal lag is the fingerprint of seasonality sitting on top of a trend. The PACF reads 0.95, -0.23, 0.04: a single large spike and then nothing of consequence. Once you know last month’s count, the month before adds little. The pattern is an AR-like level with a seasonal echo, and it tells you what a model will need: one autoregressive lag and something at lag 12.
Recognising AR from MA — and the ACF of returns
The shapes of the two functions identify the model family. An AR(\(p\)) process, \(y_t = \phi_1 y_{t-1} + \dots + \phi_p y_{t-p} + \varepsilon_t\), has an ACF that decays geometrically (every past value matters, through the chain of intermediate values) but a PACF that cuts off after lag \(p\) (once the first \(p\) lags are in the regression, older ones add nothing). A MA(\(q\)) process, \(y_t = \varepsilon_t + \theta_1\varepsilon_{t-1} + \dots + \theta_q\varepsilon_{t-q}\), is the mirror image: its ACF cuts off after lag \(q\) (a shock is forgotten after \(q\) periods) and its PACF decays. White noise has both inside the band.
| Pattern | ACF | PACF | Model |
|---|---|---|---|
| AR(\(p\)) | decays geometrically | cuts off after lag \(p\) | past values |
| MA(\(q\)) | cuts off after lag \(q\) | decays | past errors |
| ARMA(\(p,q\)) | decays | decays | both |
| White noise | all inside \(\pm 2/\sqrt n\) | same | nothing to model |
Now apply the diagnostic to the series a trader cares about: five years of daily Dow Jones log returns, 2015–2019. The cell also computes the ACF of the absolute returns, and the two panels together are the whole chapter in one figure.
With \(n = 1258\) the white-noise band is \(\pm 0.056\), and all ten return autocorrelations sit inside it: the mean of daily returns has no linear memory that five years of data can detect. But the absolute return has ACF 0.28, 0.26, 0.27, 0.25, 0.17 — every lag far outside the band, and decaying only slowly. The size of a return remembers even though its sign does not. That right-hand panel is what Section 6.5 models; for the rest of this section we stay with the mean, and the honest thing to say about it is that there is very little to model.
ARIMA(\(p, d, q\)): the baseline
The ARIMA model combines three ideas. Difference the series \(d\) times to remove a unit root (“integrated of order \(d\)”), then fit an ARMA(\(p, q\)) to the result:
\[\Delta^d y_t = c + \sum_{i=1}^{p}\phi_i\,\Delta^d y_{t-i} + \varepsilon_t + \sum_{j=1}^{q}\theta_j\,\varepsilon_{t-j}.\]
If \(d = 0\), ARIMA is simply ARMA. Log prices have a unit root, so they need \(d = 1\) — which is the same as modelling log returns with \(d = 0\). Setting \(d = 1\) on a return series would over-difference it and plant a unit root in the moving-average part, exactly the mistake of the trend-series example above.
The order is usually chosen by an information criterion. The Akaike information criterion is \(\text{AIC} = -2\ln L + 2k\): it rewards the maximised log-likelihood and charges two points per parameter, so a term earns its place only if it improves the fit by more than one unit of log-likelihood. Differences in AIC below about 2 are noise. The cell runs a nine-model grid, \(p, q \in \{0, 1, 2\}\) with \(d = 0\), on the Dow returns. Before running it, predict the winner.
The winner is (0, 0, 0) — a constant — with AIC -8409.8, and the runners-up sit within 0.4 of it. Five years of Dow returns contain no autoregressive or moving-average term worth two points of AIC. That is not a failure of ARIMA. It is a fact about the market, consistent with the flat ACF above, and it is the reason Section 6.2 exists: if the best linear model of the mean is a constant, the natural question is whether anything else does better.
Where ARIMA does work: forecast the passengers
ARIMA is the right tool where a series has linear structure, and the airline passengers have plenty. The cell trains on the log of the passengers from 1949 to 1959 (132 months), holds out 1960, runs the same nine-model grid with \(d = 1\) (the level has a trend), refits the AIC winner and produces a twelve-month forecast with its confidence band. For a baseline, it uses the seasonal-naive forecast: 1959’s value for the same month.
By AIC the best order is (2, 1, 1) with AIC -228.9. Out of sample it is a disappointment: RMSE 89.4 passengers over 1960 against 50.7 for the seasonal-naive forecast. Look at the figure. The forecast settles between 427 and 440 passengers every month with bands widening from ±84 to ±188, while actual 1960 swings from 391 in November to 622 in July. The bands are honest — the model knows it is uncertain — but the model is blind to the season, because a non-seasonal ARIMA(2,1,1) has no term at lag 12. The lesson is one to carry through the chapter: order selection is not validation. AIC picks the best of the candidates you offered it; it cannot tell you that the right candidate was not on the list.
The fix is to give the model eyes. A SARIMA adds a seasonal difference and seasonal ARMA terms at lag \(s\); the classic “airline model” is order=(0,1,1) with seasonal_order=(0,1,1,12), two moving-average parameters in total.
RMSE 18.6 — almost five times better than the non-seasonal fit (89.4) and nearly three times better than the seasonal-naive forecast (50.7), on the same two-parameter budget. The AIC agrees, -441.3 against -228.9, and the first six months of the forecast track the actual counts to within a few passengers. Where a series has structure, the classical model is unbeatable per parameter. Where it does not — daily returns — no amount of order selection will manufacture structure that is not there, and the next section shows what happens when you try.
The lecture notebook automates the grid with isom5650.timeseries.find_best_arima(series, train_size=40, compare_size=30, max_p=3, max_d=0, max_q=3), which fits every order on rolling windows and reports RMSE, AIC and BIC side by side. On Dow returns its RMSE ranking and its AIC ranking disagree — ARIMA(3,0,3) has the lowest rolling RMSE but among the worst AICs — which is the same lesson as the passengers in a different guise.
No — it means the mean of returns has no linear memory worth two parameters. The ACF of |r| in the same cell was 0.28, 0.26, 0.27 …: the variance remembers. Time-series models moved from the mean to the variance (§6.5–5.6), to spreads (§6.3) and to hidden states (§6.4).
Why Mean Forecasting Lost to ML: Forecast Accuracy, Walk-Forward Discipline, Time Series as the Base of the ML Pipeline
A forecast is only as good as the baseline it beats. In this section you will score the laziest forecast of all and discover that “predict zero” beats it; you will watch the AIC-winning ARIMA of the previous section lose money out of sample; and then you will pivot to the question that actually matters on a modern desk — what a machine-learning pipeline inherits from time series (stationary features, lags without leakage, walk-forward validation, a baseline) and what a gradient-boosting model honestly delivers when it is given one series and trained the way a desk would train it.
Four ways to score a forecast
With actual values \(y_t\) and forecasts \(\hat y_t\) over \(n\) periods, four scores are in everyday use:
\[\text{MAE} = \tfrac1n\sum|y_t - \hat y_t|, \qquad \text{RMSE} = \sqrt{\tfrac1n\sum (y_t-\hat y_t)^2}, \qquad \text{MAPE} = \tfrac{100}{n}\sum\left|\tfrac{y_t-\hat y_t}{y_t}\right|,\]
\[\text{DA} = \tfrac1n\sum \mathbf 1\{\text{sign}(y_t) = \text{sign}(\hat y_t)\}.\]
The mean absolute error treats a 2-dollar miss as twice as bad as a 1-dollar miss; the root mean squared error treats it as four times as bad, so RMSE is the score to use when large errors are disproportionately costly (as they are in risk) and MAE when they are not. Both are in the units of \(y\). The mean absolute percentage error is unit-free, which makes it popular for sales and passenger counts — and unusable for returns or price changes, because it divides by the actual value, which for a return is frequently near zero and sometimes (in Apple’s minute bars) exactly zero. A single term explodes and dominates the average, or the average is infinite. Use MAPE only for quantities that stay well away from zero. Directional accuracy ignores magnitude entirely and asks how often the sign was right; for a trading rule that goes long or short on the sign, it is the score that maps most directly to whether the rule makes money — though, as you will see, not perfectly.
The naive benchmark, and the forecast that beats it by doing nothing
The laziest forecast is “tomorrow’s change equals today’s”: \(\hat y_{t+1} = \Delta P_t\). The cell builds it on Apple’s minute bars, with the target Y defined as the next minute’s price change (shift(-1)) and the forecast as the current change. Before running it, predict two things: will copying the last change beat a coin on directional accuracy, and will the RMSE of forecasting zero every minute be above or below the naive RMSE?
Commit to a prediction: is the naive forecast’s DA above 0.5, and is the zero forecast’s RMSE larger or smaller than the naive one’s?
Over 388 forecasts the naive rule scores MAE 0.1157 dollars, RMSE 0.1694 and directional accuracy 0.466 — worse than a coin. Minute-by-minute price changes are slightly negatively autocorrelated, a mechanical effect of the bid–ask bounce (a trade at the ask is followed, more often than not, by a trade at the bid), so copying the last change points you the wrong way more often than the right one. The zero forecast has RMSE 0.1057, well below the naive 0.1694. The arithmetic is worth understanding: the error of the naive forecast is \(\Delta P_{t+1} - \Delta P_t\), whose variance is the sum of two change variances (minus twice a small negative covariance); the error of the zero forecast is \(\Delta P_{t+1}\) itself, one change variance. Predicting the mean — which is essentially zero — adds no noise of its own. The consequence is a rule that holds for every forecasting exercise in this book: any forecast you build must beat the zero (or mean) forecast, not the naive one. That zero forecast is also the first baseline in every machine-learning pipeline.
Moving averages: the window is the model
If the naive forecast is too jumpy, average it. The simple moving average \(\hat y_{t+1} = \tfrac{1}{40}\sum_{i=0}^{39} \Delta P_{t-i}\) is .rolling(40).mean(). The exponentially weighted moving average \(s_t = \alpha\,x_t + (1-\alpha)\,s_{t-1}\), with \(s_1 = x_1\), weights the value \(k\) periods back by \(\alpha(1-\alpha)^k\), so recent values count more and old ones fade geometrically; its “half-life” is \(\ln 0.5 / \ln(1 - \alpha)\), about 1.9 periods for \(\alpha = 0.3\). As \(\alpha \to 1\) the EWMA becomes the naive forecast; as \(\alpha \to 0\) it becomes the long-run mean. The cell first checks the recursion on three numbers — with \(x = [10, 20, 30]\) and \(\alpha = 0.3\) you should get \(s_2 = 0.3 \cdot 20 + 0.7 \cdot 10 = 13\) and \(s_3 = 0.3 \cdot 30 + 0.7 \cdot 13 = 18.1\) — then scores both forecasts on the minute bars.
The toy series prints [10.0, 13.0, 18.1], as computed by hand. On the 349 rows for which all three forecasts exist, the SMA(40) scores RMSE 0.0993 against 0.0986 for the zero forecast and 0.1428 for naive. Averaging over forty minutes has crushed the bid–ask bounce and brought the forecast to a tie with predicting zero; its only edge is a direction call, DA 0.521, that a zero forecast cannot make at all. The EWMA with \(\alpha = 0.3\) lands in between at RMSE 0.1075 and DA 0.496, because a two-minute memory still inherits most of the bounce. Two morals. First, the window is the model — the choice between 40 minutes and 2 minutes mattered more than the choice between simple and exponential weighting — and you tuned it by eye, which is why the next step is to let a likelihood choose. Second, compare forecasts only on the same rows: the naive RMSE moved from 0.1694 to 0.1428 merely by dropping the forty warm-up minutes, with no change in skill whatsoever.
The ARIMA trading test: fixed parameters, one step ahead
Now let the data choose the memory. On Apple’s daily log returns over 2020–2023 the nine-model AIC grid of Section 6.1 selects ARIMA(2,0,1). The cell fits it on those four years, then feeds 2024 through res.apply(new_data), which produces one-step-ahead forecasts using the fixed 2020–2023 parameters without refitting — exactly what a trader who had estimated the model on 31 December 2023 would have done each day. The rule trades the sign of the forecast: long when the forecast return is positive, short when it is negative.
Directional accuracy 0.480; RMSE 0.01451, which is worse than the 0.01421 you get by forecasting the training-sample mean every day; and an annualised Sharpe ratio of -0.20 for the sign rule against 1.16 for simply holding the stock through 2024. The AIC winner in sample is a loser out of sample. Nothing was done wrong here: the order was chosen by a defensible criterion, the parameters were frozen honestly, the test year was untouched. The result is simply what a single daily return series contains — almost no linear memory in its mean — and it is the result that moved mean forecasting off the classical desk. The lecture notebook found the same with a 30-day rolling refit of ARIMA(2,0,3): a correlation of -0.08 between forecast and realised change.
The pivot: what the ML pipeline inherits from time series
It is tempting to conclude that if ARIMA cannot forecast returns, a bigger model can, and Chapter 4 showed gradient boosting doing useful work on the credit-risk cross-section. But machine learning did not replace time-series analysis; it sits on top of it. Four things that every return-prediction model quietly assumes are time-series ideas, and skipping any of them produces a model that looks superb in the notebook and fails on the first live day:
| Inherited discipline | From | Failure if skipped |
|---|---|---|
| Stationary features — returns, ratios, differences, never price levels | §6.1 | a tree splits on a price level that never recurs; the model memorises 2021 |
| Lag to avoid leakage — every feature at \(t\) uses data available at the close of \(t-1\) | .shift(1) |
in-sample accuracy 0.9, live accuracy 0.5 |
| Walk-forward validation — expanding window, refit per period, test strictly after train | §6.2 | random K-fold lets the future train the past |
| A baseline that must be beaten — the zero forecast, ARIMA(0,0,0) | §6.1–5.2 | a Sharpe ratio with nothing to compare it to |
The first row is the stationarity reflex applied to features: a tree that sees “S&P at 4,500” cannot generalise to “S&P at 5,500”, but a tree that sees “21-day return of −4 %” can. The second row is the most common single error in student projects and, in the author’s experience, in a surprising number of professional ones. The third row is why KFold is banned in Chapter 4’s time-ordered exercises: an expanding-window walk-forward split trains on everything up to date \(T\), tests on the next block, then moves \(T\) forward and refits, so that every test observation is predicted by a model that has never seen it or anything after it. (A rolling window drops the oldest data as it moves; the choice between the two is the same memory trade-off as the SMA window.) The fourth row is the zero forecast of the previous subsection. Most of the value of a machine-learning pipeline is in these four rows; the model at the end is the last and least important step.
Build the features — and spot the leak
The cell builds eleven features from ten years of S&P 500 daily log returns: five lagged returns, rolling 5- and 21-day means and standard deviations, twelve-month-minus-one-month momentum, and the ratio of short to long volatility. Every rolling quantity is computed and then shifted by one day, so that the feature at date \(t\) uses returns up to \(t-1\) only. The target is today’s return. The first walk-forward fold trains on 2016–2017 and tests on 2018.
Eleven features, all stationary (returns, rolling moments of returns, a ratio), all shifted. The 252-day momentum window means the first usable row is in early 2016, so the first fold has about two years of training data and tests on 2018; each later fold expands the training window by a year. Now the model.
Gradient boosting, trained the way a desk would train it
The specification is the one from Chapter 4 — HistGradientBoostingRegressor with shallow trees (max_depth=2), a slow learning rate, 200 iterations and at least 50 observations per leaf — deliberately not tuned on this data. The loop refits every January on all data to date, predicts the coming year, and trades long/flat on the sign of the prediction: in the index when the forecast is positive, in cash otherwise. Seven refits cover 2018–2024.
These are the honest numbers. Over 1,761 test days the directional accuracy is 0.494; the RMSE of the model, 0.01276, is worse than the zero forecast’s 0.01247; the correlation between prediction and outcome is -0.013. The long/flat rule earns a Sharpe of 0.31 while being in the market 59 % of the time, against 0.57 for holding the index throughout. No single year’s directional accuracy clears 0.54. The same walk-forward discipline that exposed ARIMA exposes the gradient-boosting model. With one series and its own past — even eleven well-built features of it — there is no exploitable mean memory at the daily horizon.
This is the right place to say precisely what machine learning did win. It did not win by forecasting one series from its own history with a bigger model; it won by moving to the cross-section — thousands of stocks at once, ranked against each other rather than forecast in isolation — and to richer features than a series’ own lags: fundamentals, order flow, text, the attention-based cross-sectional features of Section 4.5. Gu, Kelly and Xiu (2020) is the reference result: on the US stock cross-section, shallow trees and two- or three-layer networks beat deep ones, and the gain over linear models comes from interactions among many features, not from depth. The pipeline — stationary features, lags, walk-forward, baseline — is what ML inherited from time series. The alpha is elsewhere, and it is the subject of Chapter 7.
The cell runs, and the colleague’s features look better than the chapter’s — a correlation of 0.455 rather than 0.397 between std21 and today’s squared return, and 0.213 rather than -0.019 between mean21 and today’s return. That is the bug. r.rolling(21).std() at date \(t\) is computed over the window ending at \(t\), so it contains \(r_t\) — the very number the model is being asked to predict — and rolling(21).mean() contains \(r_t/21\). The feature includes the target. A tree will find this instantly, post a superb in-sample and even walk-forward score (walk-forward protects against training on the future, not against a feature that already holds the present), and collapse on the first live day when \(r_t\) is not yet known. The fix is the one the chapter’s cell applied: compute the rolling quantity and then .shift(1), so that the feature at \(t\) closes at \(t-1\). As a rule, every feature should be built with .shift(1) after the rolling window, and the audit question to ask of any feature is “what is the date of the last observation it uses?”
What about LSTMs and sequence models?
Recurrent networks are the sequence model students most often want to try, and in Colab they are a few lines of PyTorch on the same eleven features arranged into 21-day windows:
import torch, torch.nn as nn
class LSTMReg(nn.Module):
def __init__(self, n_feat, hidden=32):
super().__init__(); self.lstm = nn.LSTM(n_feat, hidden, batch_first=True); self.head = nn.Linear(hidden, 1)
def forward(self, x): out, _ = self.lstm(x); return self.head(out[:, -1])
model = LSTMReg(n_feat=11); opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for epoch in range(20): # X_seq: (batch, 21 days, 11 features), y: next-day return
opt.zero_grad(); loss = ((model(X_seq) - y).pow(2)).mean(); loss.backward(); opt.step()Three facts should temper expectations. First, the Gu–Kelly–Xiu result above: on daily equity returns the evidence favours shallow models with good features over deep sequence models. Second, zero-shot time-series foundation models (Chronos, TimesFM and their successors) show no consistent edge over a gradient-boosting baseline on daily returns — there is too little predictable structure in returns for pre-training on other series to transfer. Third, sequence models do help where the sequence itself carries structure: intraday order flow, the shape of the volatility surface, and the joint modelling of many series at once. If your LSTM beats the GBM by four percentage points of accuracy in sample, the first thing to check is not the hidden size but leakage — a sequence window that straddles the test boundary is the classic culprit — and the second is to re-run both models walk-forward on identical features and folds.
Aside: several series at once — VAR and Granger causality
Everything so far used one series. With several, the natural extension of the AR model is the vector autoregression, \(\mathbf y_t = \mathbf c + A_1\mathbf y_{t-1} + \dots + A_p\mathbf y_{t-p} + \mathbf u_t\): every variable regressed on \(p\) lags of every variable, \(k\) OLS equations sharing the same \(kp + 1\) regressors. With \(k = 3\) series and \(p = 4\) lags that is already 39 coefficients, which is why lag order is chosen by BIC and why VAR forecasts are used rolling — refit each period, forecast one step — rather than fixed. The lecture notebook made the point on FRED-MD: a VAR fitted once in 2014 and iterated forward had an inflation MSE of 6.13 over 2015–2022 and was still reading 2.5 % when actual inflation was 8.2 % in September 2022; refit every month, its MSE fell to 0.66, and on the last observation it read 8.22 %. A fixed multi-step VAR forecast decays to the model’s unconditional mean and forgets everything it knew; a rolling one-step forecast is a different, and useful, object.
Within a VAR, \(X\) Granger-causes \(Y\) if lags of \(X\) improve the forecast of \(Y\) beyond \(Y\)’s own lags — an F-test that the \(X\) coefficients in \(Y\)’s equation are jointly zero. It is a statement about forecastability, not mechanism: Christmas-card sales Granger-cause Christmas. On macro data it is informative (on the Stock–Watson quarterly data the fed funds rate Granger-causes inflation at p = 0.008; on FRED-MD with twelve lags, p < 0.0001, while inflation does not Granger-cause the rate, p = 0.51). On liquid daily prices it mostly confirms market efficiency, as the cell shows for NVIDIA and SPY over 2023–2024.
NVDA → SPY gives p 0.6835 and SPY → NVDA gives p 0.126: neither direction at 5 %. Whatever lead–lag exists between the most-traded stock and the index is priced away within the day, long before a daily close can record it. The same test that finds a rich structure in monthly macro data finds nothing in daily returns, which is the theme of this section from one more angle.
The naive error is \(\Delta P_{t+1} - \Delta P_t\), whose variance is the sum of two change variances (the bid–ask bounce makes it slightly worse still); the zero forecast’s error is \(\Delta P_{t+1}\) alone. Predicting the mean adds no noise. Every forecast must beat the zero/mean forecast, not the naive one.
Stationary features (a tree memorising a price level that never recurs); .shift(1) after every rolling window (the target leaking into a feature); expanding-window walk-forward validation (the future training the past); a baseline to beat — the zero forecast (a Sharpe with no comparison). The GBM on the S&P 500 passed all four and still scored DA 0.494.
Structural Arbitrage I: Cointegration and Pairs Trading
Two drunks leave the pub and wander off. Each path is a random walk, and nothing in the previous section will forecast where either of them is going. But if the two are tied together by a rope, the distance between them is stationary: it stretches, and it comes back. That rope is cointegration, and a stationary spread is the one thing in this chapter that a time-series model can trade directly, without forecasting a direction. You will see two pairs with no rope, one with a rope, and learn why the rope always looks tighter in sample than it could have looked at the time.
First, the trap: a regression of two strangers
Most economic and financial series are non-stationary in levels — prices, indices, GDP, the CPI. If you regress one such series on another, OLS will happily report a high R² and an enormous \(t\)-statistic whether or not the two have anything to do with each other. This is the spurious regression problem, noticed by Yule in 1926 as “nonsense correlations” and demonstrated by simulation in Granger and Newbold (1974): two independent random walks, regressed on each other, reject the null of no relationship about three quarters of the time at the nominal 5 % level. The mechanism is that the residual of such a regression is itself a random walk, so it does not average out; the usual standard errors, which assume a residual that reverts to zero, are meaningless.
The cell constructs the cleanest possible example. It regresses Apple’s log price over 2015–2024 on the Dow’s log price over 1985–1995, lined up row by row — 2,516 observations each, thirty years apart, with no economic connection whatsoever. Predict the R².
Two unrelated random walks, 2,516 rows each: will the R² be near 0, near 0.3, or near 0.8?
R² 0.804 and a \(t\)-statistic of 101.7 — a “highly significant” fiction, manufactured by two drifting series. The tell is the third line: an ADF p-value of 0.2138 on the residual. The “relationship” wanders; it never reverts to zero, so there is no relationship. From this comes the rule that governs the whole section: a regression between two unit-root series is meaningful only if the residual is stationary. Testing that residual is the cointegration test.
The Engle–Granger test
Two series \(P^A_t\) and \(P^B_t\) are cointegrated if each has a unit root but some linear combination \(u_t = \log P^A_t - \alpha - \beta\log P^B_t\) is stationary. The vector \((1, -\beta)\) is the cointegrating vector; the stationary \(u_t\) is the long-run equilibrium error. Logs are used because prices grow multiplicatively and a log spread is a percentage deviation, which is what a trade in the two stocks earns: differencing the equilibrium relation gives \(r^A_t = \beta\, r^B_t + \Delta u_t\), so a position long one unit of \(A\) and short \(\beta\) units of \(B\) earns \(\Delta u_t\) — the change in the spread — and nothing else.
Engle and Granger (1987) proposed a two-step test that follows the definition:
- Confirm that each series has a unit root (ADF fails to reject on both).
- Regress \(\log P^A_t = \alpha + \beta\log P^B_t + u_t\) by OLS.
- Test the residual \(\hat u_t\) for a unit root — but with harsher critical values than the ordinary ADF, because \(\hat\beta\) was chosen by least squares precisely to make \(\hat u_t\) look as stationary as possible.
The third point is the one students get wrong. If you run a plain ADF on the OLS residual and it reports p = 0.024, while coint() reports p = 0.080, the second number is the right one. OLS minimises the residual variance, which mechanically shrinks the residual’s apparent persistence; Engle and Granger derived larger critical values to compensate, and statsmodels.tsa.stattools.coint(a, b) performs steps 2 and 3 with those critical values, returning (t_stat, p_value, critical_values). Using plain ADF on the residual over-rejects — it finds ropes that are not there. (For more than two series the Johansen procedure, coint_johansen in statsmodels, tests for the number of cointegrating vectors at once.)
The first candidate pair is NVIDIA against SPY over 2023–2024, 501 trading days.
Both levels keep their unit roots (NVDA p 0.23, SPY p 0.79), so step 1 passes. The OLS slope is 4.38 — over these two years NVIDIA’s log price moved more than four times as much as the index’s. The naive ADF on the residual gives p 0.024, which would tempt you; the Engle–Granger test gives p 0.080. Not cointegrated at 5 %, borderline at 10 %: the rope is loose. The figure shows why — the residual drifts one way for months at a time, which is what a pair of drunks with a very long rope looks like.
Formation then trade: the discipline, on a pair with no rope
A pairs trade is a two-window procedure, and the order of the windows is not negotiable. On the formation window you estimate the rope — \(\alpha\) and \(\beta\) — and check that the residual is stationary. On the subsequent trading window you compute the spread with those coefficients and trade its deviations. Estimating the rope on the trading window and then trading it is using the future twice. The cell estimates on 2023 and watches 2024, standardising the spread with a rolling 40-day mean and standard deviation to form a z-score.
The 2023 residual already fails the stationarity check (ADF p 0.32), and 2024 confirms the verdict: the spread’s mean sits at -0.227, about 1.5 formation-window standard deviations below zero, and it never came back. A rolling z-score would nonetheless have fired on 17 “entry” days — because a z-score standardises whatever it is given, anchor or no anchor — and each of those entries would have been a bet on reversion to a level that no longer existed. No rope, no pairs trade. Apple against the S&P 500 over 2015–2024 fails the same way, with a coint p-value of 0.25. Most candidate pairs do.
A rope that holds: Alphabet and the S&P 500, 2015–2020
The chapter’s positive example is Alphabet against the S&P 500 over five years of daily closes, 1,259 days from October 2015 to October 2020. The cell runs the full test, then estimates how fast a deviation in the spread decays: regressing \(\Delta u_t\) on \(u_{t-1}\) gives a coefficient \(\phi\) (negative if the spread reverts), and the half-life of a deviation is \(-\ln 2/\phi\) — the number of days for half of a shock to the spread to disappear.
Both series have unit roots (p 0.85 and 0.80). The slope is 1.271, and the Engle–Granger statistic is -3.800 against a 5 % critical value of -3.341: p = 0.0136. Cointegrated at 5 %. The spread’s standard deviation is 0.045 in log units — roughly 4.5 % — and a deviation halves in 24.3 days: slow, but it comes back, and the figure shows it crossing zero repeatedly rather than drifting. This is the rope.
Why in-sample cointegration overstates tradability
Before trading it, ask a question that the full-sample test cannot answer: could a trader in 2017 or 2018 have known about this rope? The full-sample \(\beta\) was chosen, with hindsight, to make the entire 2015–2020 residual as stationary as possible — including the part you would have been trading. The cell repeats the test on two formation windows that end in October 2017 and October 2018 and reports how the subsequent spread behaved under each formation window’s coefficients.
With formation to October 2017 the p-value is 0.239; to October 2018 it is 0.113. Neither rejects at 5 %. And in the first case the trading-window spread drifts to a mean of 0.071, about 1.8 formation-window standard deviations from zero. The rope you can see in 2020 is not the rope you could have tied in 2017. In-sample cointegration is a statement about a relationship you could not have known; the tradable question is whether a formation-window rope still holds in the trading window, and the honest answer for this pair, in real time, was “not at 5 %”. The practical response is not to abandon the pair but to re-estimate the rope as you go.
A walk-forward hedge ratio and the z-score
The cell re-ties the rope every 63 trading days (a quarter) on the trailing 252 days (a year) and holds the coefficients for the next quarter. The spread on each day therefore uses only a rope that was known on that day; the z-score uses a 40-day rolling mean and standard deviation of that spread. This is the walk-forward discipline of Section 6.2 applied to a hedge ratio.
Sixteen refits give 1,007 trading days from October 2016. The hedge ratio moves from 0.569 to 1.601 over the period — the rope is not a constant, and a single full-sample \(\beta\) of 1.27 would have been badly wrong for long stretches. There are 110 days with \(|z| > 2\); those are the candidate entries.
The rule: enter at \(|z| > 2\), exit at \(|z| < 0.5\)
The trading rule is the standard one. When \(z > 2\) the spread is rich — Alphabet is expensive relative to the index — so short the spread: short GOOG, long \(\beta\) units of the S&P 500. When \(z < -2\), go long the spread. Close the position when \(|z|\) falls below 0.5. The position decided at the close of day \(t-1\) earns the change in the spread on day \(t\), \(\Delta \log P^{G}_t - \beta\,\Delta\log P^{S}_t\), so the P&L line multiplies yesterday’s position by today’s spread change. The cell reports three statistics — cumulative log return, annualised Sharpe ratio and maximum drawdown — for the rule and for simply holding Alphabet, and repeats the rule with a looser entry at 1.5.
With entry at 2.0 the book is in a trade 35 % of days, with 52 entries and exits. It earns a cumulative 30.5 % with a Sharpe ratio of 0.75 and a maximum drawdown of -11.6 %. Holding Alphabet over the same days earned 68.6 % with a Sharpe of 0.63 and a drawdown of -36.8 %. Less money, but better money — and, more importantly, market-neutral money: the book is long one stock and short the index in the hedge ratio, so it made it through March 2020 with a third of the drawdown of the stock. Loosening the entry to 1.5 puts the book in a trade 47 % of days with 80 round trips, and lifts the cumulative return to 53.0 % and the Sharpe to 1.09 at the same drawdown. Better on paper — but with 80 crossings the transaction costs that this back-test ignores are now roughly 50 % larger, and one tuned threshold on one pair is a hypothesis, not a book. A pairs desk runs dozens of candidate pairs, requires the formation-window test to reject and the half-life to be short (under about 30 days), and stops trading any pair whose rolling residual test starts to fail.
The student’s book is a catastrophe — a cumulative -69.2 %, a Sharpe ratio of -1.47 and a drawdown of -72.5 % from what appears to be the same rule that earned 30.5 % with a Sharpe of 0.75. The difference is a single .shift(1). The position pos on day \(t\) is decided from the z-score computed at the close of day \(t\), which uses that day’s prices; multiplying it by the spread change of the same day books a return the trader could not have earned, because the trade could only be placed after the close. Here the fiction runs against the rule: an entry day is, by construction, a day on which the spread has just jumped through the 2-sigma line, so a short-spread position opened “at” that day’s close is charged with that day’s rise in the spread, and every entry starts with a loss that no real trader would have taken. In a momentum rule the same mistake would flatter the book instead. Either way the number is meaningless: this is look-ahead bias, today’s signal applied to today’s return. The fix is pos.shift(1) — yesterday’s position earns today’s change. The same rule appears in Section 6.4 (yesterday’s regime probability decides today’s exposure) and Section 6.6 (yesterday’s forecast volatility sizes today’s position), and it is the first thing to check in any back-test, whether it looks too good or, as here, inexplicably bad.
coint(). The OLS slope was chosen to minimise the residual variance, which mechanically makes the residual look more stationary; Engle–Granger critical values are harsher to compensate. A plain ADF on the residual over-rejects and finds ropes that are not there (NVDA/SPY: not cointegrated at 5 %).
The full-sample β used the trading window to fit itself, so the in-sample rope is always tighter than the one a trader could have tied. Trade only formation-window ropes, re-estimate them as you go (the walk-forward hedge ratio moved from 0.57 to 1.60), and treat one back-tested threshold as a hypothesis, not a book.
Structural Arbitrage II: Kalman Filters and Regime Switching
A price is a hidden “true level” plus noise. A hedge ratio is a hidden slope that drifts. The market is in a hidden calm or stormy regime. You never observe any of these — but each new data point lets you update your best guess of them, and a recursive update that is optimal under Gaussian noise has a name: the Kalman filter. This section builds it in six lines, extends it to a time-varying beta in eight, and then replaces the continuous hidden state by a discrete one — Hamilton’s Markov-switching model — with one MarkovRegression call that cuts a drawdown by three quarters.
Filter Apple’s minute price, and let statsmodels estimate \(Q\) and \(R\)
The six-line filter needs \(Q\) and \(R\), and in practice you estimate them by maximum likelihood: the innovations \(\nu_t\) are independent normals with variance \(P_{t|t-1} + R\), so their log-likelihood is a function of \((Q, R)\) that a numerical optimiser can maximise. sm.tsa.UnobservedComponents(y, level="local level") does exactly that — it reports \(R\) as sigma2.irregular and \(Q\) as sigma2.level — and then runs the filter. The cell fits it to the 390 minute closes of Apple, feeds the estimated \(Q\) and \(R\) into the six-line filter, and compares the two filtered paths.
The maximum-likelihood estimates have a physical reading. \(\sqrt{R} = \$0.048\) — about a nickel of bid–ask bounce around the true level each minute — and \(\sqrt{Q} = \$0.096\): the true level itself moves about a dime per minute, twice as much as the noise. With \(q = Q/R \approx 4\) the steady-state gain is about 0.83, so the filter follows the data closely, as it should when the level genuinely wanders; the filtered line in the figure hugs the price with only a slight smoothing of the one-minute jitter. The six-line filter matches the statsmodels filtered state to within 0.005 dollars — the tiny gap comes from how the first observation is initialised. The lecture notebook applied the same local-level filter to daily returns rather than prices, to extract a slowly moving drift; with \(Q/R = 0.01\) that filter is an EWMA with \(\alpha = 0.095\), and its sign as a trading signal on 2024 Apple returns gave a Sharpe of 0.70 against 1.16 for buy-and-hold. A Kalman filter is a principled smoother, not a crystal ball; where it earns its keep is in tracking a hidden parameter, which is the next subsection.
Regime switching
Calm and stormy markets are not two points on a continuum of \(\sigma\); they behave like two states that the market jumps between and stays in for weeks. Hamilton (1989) formalised this as a Markov-switching model: an unobserved state \(S_t \in \{0, 1\}\) follows a Markov chain with transition probabilities \(p_{ij} = P(S_t = j \mid S_{t-1} = i)\), and the parameters of the return distribution depend on the state:
\[r_t = \mu_{S_t} + \sigma_{S_t}\,\varepsilon_t, \qquad \varepsilon_t \sim N(0, 1).\]
With two regimes and switching means and variances that is six parameters: two means, two variances and two staying probabilities \(p_{00}\) and \(p_{11}\) (the switching probabilities are their complements). The likelihood is computed by the Hamilton filter, which is Bayes’ rule run forward: given the probability of each state yesterday, predict today’s state probabilities through the transition matrix, observe \(r_t\), and re-weight the states by how well each explains it. The filter’s output is \(P(S_t = j \mid r_1, \dots, r_t)\), the filtered probability, which uses data to date \(t\) only. A backward pass gives the smoothed probability \(P(S_t = j \mid r_1, \dots, r_T)\), which uses the whole sample and is therefore the better picture but the wrong thing to trade on. The expected duration of a spell in state \(i\) is \(1/(1 - p_{ii})\): with \(p_{00} = 0.98\) a calm spell lasts 50 days on average, with \(p_{11} = 0.96\) a stormy one lasts 25. Regimes are persistent, and that persistence is what makes them usable as a risk filter rather than noise.
The cell fits the model to ten years of S&P 500 daily log returns in percent, identifies the calm regime as the one with the smaller variance, and reports the smoothed probability of calm by month.
The calm regime has a daily standard deviation of 0.602 %, a mean of +0.107 % and an expected duration of 54.8 days. The stormy regime has a standard deviation of 1.778 % — three times the calm one, nine times the variance — a negative mean of -0.096 %, and an expected duration of 26.6 days. The desk spends 68 % of its days in the calm state. The model has no calendar and was told nothing about events, yet the 34 mostly-stormy months out of 120 are the ones you would name: January 2016, February and the fourth quarter of 2018, March to June 2020, essentially all of 2022, and August 2024. It found the bear markets from the variance alone, in a fit that takes a fraction of a second.
What a regime filter does to a drawdown
The trading use is a switch: hold the index only when yesterday’s filtered probability of calm exceeds one half. Two details make this honest. It uses the filtered probability, which knows only the past, not the smoothed one, which knows the future; and it uses yesterday’s value, so the position for day \(t\) is decided at the close of \(t-1\).
The filter is in the market 67.9 % of days. It earns a cumulative 76.4 % with a Sharpe ratio of 0.81 and a maximum drawdown of -11.3 %; buy-and-hold earns 105.0 % with a Sharpe of 0.59 and a drawdown of -41.4 %. The filter gave up a quarter of the total return to cut the worst drawdown by three quarters. Note what it did not do: it did not forecast direction. It stepped aside when the variance model said the market had switched into its stormy state, in which the mean is negative and the variance is nine times larger, and it stepped back in when the state flipped back. Like every volatility tool in this chapter it forecasts turbulence, not direction — and the next two sections model that turbulence directly, one day at a time, instead of as two discrete states.
\(K = P/(P+R)\) falls. The filter absorbs less of each surprise and trusts the carried-forward model more than the data. Conversely a level that genuinely moves (large \(Q\)) inflates \(P\) and raises \(K\) — Apple’s minute price had \(Q/R \approx 4\) and a steady-state gain of about 0.83.
The smoothed probability \(P(S_t \mid r_1 \dots r_T)\) uses the whole sample, including the future; the filtered one uses data to \(t\) only. Trading on the smoothed series (or on today’s filtered value with today’s return) is look-ahead bias. Honest: yesterday’s filtered \(P(\text{calm}) > 0.5\) — Sharpe 0.81 vs 0.59, drawdown −11.3 % vs −41.4 %.
Volatility I: ARCH, GARCH(1,1), Forecasts and VaR
Returns look like white noise — until you square them. This section turns that observation into a model. You will see volatility clustering in the Dow, prove it with the autocorrelation of squared returns and Engle’s ARCH-LM test, write a GARCH(1,1) maximum-likelihood estimator in fifteen lines, forecast tomorrow’s variance and the path beyond it, convert the forecast into a Value-at-Risk, and back-test that VaR through March 2020 against a model that assumed the variance was constant.
Which is easier to forecast: the sign of tomorrow’s return, or its size?
Section 6.1 answered the first half of this question: the ACF of daily Dow returns sits inside the white-noise band, and the sign of tomorrow’s return is close to a coin flip. The second half is different. The size of a return clusters: a 3 % day is far more likely after another 3 % day than after a 0.3 % day. Predictable size with an unpredictable sign is precisely what conditional heteroskedasticity means — the variance, conditional on the past, changes through time even though the mean does not. It is the single most robust empirical regularity in financial data.
The cell loads the Dow, forms daily log returns in percent over 2015–2020 — every volatility model in this chapter works in percent, for a numerical reason explained below — and compares the standard deviation of a quiet year with that of one month of 2020.
The daily standard deviation was 0.417 % in 2017 and 6.381 % in March 2020 — a 15-fold swing in scale, which is a 234-fold swing in variance. The worst day, −13.84 % on 16 March 2020, sits inside a cluster of other enormous days rather than arriving alone. Quiet begets quiet; storms beget storms. The autocorrelations make the point quantitatively. With 1,479 observations the white-noise band is ±0.052. Squared returns have ACF 0.442, 0.551, 0.325, 0.324, 0.297 at lags one to five — every one far outside the band, barely decaying — and a Ljung–Box \(Q(10)\) of 1929. Raw returns have ACF −0.195, 0.157, −0.015, −0.080, 0.079: small, though \(Q(10) = 321\) is not zero either, a consequence of the violent reversals of March 2020 rather than of any usable mean memory (in Section 6.6 you will see that once each day is weighted by its volatility, this apparent mean structure changes sign). Same series, two very different autocorrelation functions: that is the fingerprint of ARCH.
Engle’s ARCH and Bollerslev’s GARCH
Engle’s (1982) idea was to keep the return innovation white noise but let its variance depend on the past. Write \(\varepsilon_t = \sigma_t z_t\) with \(z_t \sim N(0, 1)\) i.i.d., and let
\[\text{ARCH}(q):\ \sigma_t^2 = \omega + \sum_{i=1}^{q}\alpha_i \varepsilon_{t-i}^2, \qquad\qquad \text{GARCH}(1,1):\ \sigma_t^2 = \omega + \alpha\,\varepsilon_{t-1}^2 + \beta\,\sigma_{t-1}^2 .\]
Under ARCH, \(\mathbb E[\varepsilon_t \mid \text{past}] = \sigma_t\,\mathbb E[z_t] = 0\), so \(\varepsilon_t\) is uncorrelated with its own past — white noise, with an unpredictable sign. But \(\varepsilon_t^2 = \sigma_t^2 z_t^2\) has conditional mean \(\sigma_t^2\), a linear function of past squared shocks, so the squares follow an AR(\(q\)). That is exactly the pattern in the Dow data. The unconditional variance is \(\omega/(1 - \sum\alpha_i)\), finite only if \(\sum \alpha_i < 1\). The lecture notebook simulated an ARCH(3) with \(\omega = 0.2\) and \(\alpha = (0.4, 0.1, 0.1)\): the ACF of \(\varepsilon_t\) was flat, the ACF of \(\varepsilon_t^2\) was 0.62, 0.41, 0.23 and then faded after lag 3, and the sample variance was 0.475 against the theoretical \(0.2/(1 - 0.6) = 0.5\).
Bollerslev’s (1986) GARCH adds yesterday’s variance to the recursion. Substitute the GARCH(1,1) equation into itself and \(\sigma_t^2\) becomes a weighted sum of all past squared shocks with geometrically decaying weights \(\alpha, \alpha\beta, \alpha\beta^2, \dots\): GARCH(1,1) is an ARCH(\(\infty\)) with geometric weights, and the ACF of \(\varepsilon_t^2\) decays geometrically at rate \(\alpha + \beta\) instead of cutting off at a fixed lag. That is why one \(\beta\) replaces a long ARCH lag structure. Three parameters, three jobs: \(\omega\) is the baseline, with unconditional variance \(\omega/(1 - \alpha - \beta)\); \(\alpha\) is the news channel, how much yesterday’s shock moves today’s variance; \(\beta\) is persistence, how slowly the variance decays back toward the baseline. The sum \(\alpha + \beta\) is the persistence of a shock to variance, and the half-life of that shock is \(\ln 0.5/\ln(\alpha + \beta)\). Real equity markets have \(\alpha + \beta \approx 0.95\)–\(0.99\).
The ARCH-LM test: one OLS you already know
A picture is not a test, and Engle’s test is one regression. Under the null of no ARCH, \(\sigma_t^2\) is constant and squared residuals are unpredictable from their own lags. So regress \(\hat\varepsilon_t^2\) on \(\hat\varepsilon_{t-1}^2, \dots, \hat\varepsilon_{t-q}^2\); under the null, \(LM = nR^2\) is distributed \(\chi^2_q\). A large \(R^2\) means yesterday’s squared shocks predict today’s. The cell implements the test from scratch, checks it against statsmodels’ het_arch, and repeats it on Apple’s returns over 2018–2020.
On the Dow the regression that “should” have \(R^2 = 0\) has \(R^2 = 0.362\): \(LM = 533.2\) against a \(\chi^2_5\) whose 1 % critical value is 15.1. het_arch reports the same 533.2 — you have just re-implemented it. The lag-2 coefficient, 0.446, is the largest of the five, a Monday-after-Friday echo. As a control, the same code on 1,479 i.i.d. normals gives \(LM \approx 5\) and \(p = 0.42\) — a test that always rejected would be useless, and this one discriminates. Apple over 756 days clusters like the index: \(LM = 132.6\), \(p < 10^{-4}\), with squared-return ACF 0.319, 0.302, 0.159.
In Colab: the arch package, and why you read its output carefully
In Colab the lecture notebook fits GARCH in one line with Kevin Sheppard’s arch package, which is not available in the browser:
from arch import arch_model
re = dji["Return"] # decimal log returns, 1985 onward
GARCH11 = arch_model(re.iloc[0:252*5], p=1, q=1).fit(disp="off")
print(GARCH11.summary())
# mu 1.0176e-03
# omega 3.5235e-06 std err 4.250e-12
# alpha[1] 0.1000 std err 5.500e-02
# beta[1] 0.8800 std err 3.678e-02Look at those estimates: \(\hat\alpha = 0.1000\) and \(\hat\beta = 0.8800\) to four decimals, and a standard error on \(\omega\) of \(4 \times 10^{-12}\). They are arch’s default starting values. With returns in decimals, \(\omega\) is of order \(10^{-6}\), the likelihood surface is nearly flat in it, and the optimiser stopped where it started; the package prints a warning asking you to rescale the data by 100, which the notebook ignored. The tell was the absurd standard error. Fitted properly on the same 1985–1990 window in percent, the estimates are \(\hat\alpha = 0.152\) and \(\hat\beta = 0.801\), with \(\hat\omega\) 2.4 times the stalled value. Scale returns to percent before you optimise, and always read the convergence flag. It is a lesson about numerical optimisation in general, and the reason every return series in these two sections is multiplied by 100.
The estimator in fifteen lines
With \(r_t = \mu + \varepsilon_t\) and Gaussian \(z_t\), the log-likelihood of one observation is \(-\tfrac12\left[\ln(2\pi\sigma_t^2) + \varepsilon_t^2/\sigma_t^2\right]\), where \(\sigma_t^2\) comes from the recursion started at the sample variance. Sum over \(t\), negate, and minimise over \((\mu, \omega, \alpha, \beta)\) with bounds that keep \(\omega\) positive and \(\alpha, \beta\) in \([0, 1]\). That is the whole estimator: a variance recursion, a likelihood, and scipy.optimize.minimize with L-BFGS-B. Equity indices typically show \(\alpha \approx 0.05\)–\(0.2\) and \(\alpha + \beta \approx 0.95\)–\(0.99\); before running the fit on the Dow, predict whether the persistence lands above or below 0.95.
For the Dow over 2015–2020, will the fitted persistence \(\hat\alpha + \hat\beta\) come out above or below 0.95?
The fit converges in well under a second: \(\hat\mu = 0.082\) % per day, \(\hat\omega = 0.037\), \(\hat\alpha = 0.214\), \(\hat\beta = 0.761\). Persistence is 0.975 — above 0.95, as for most equity indices — so a shock to variance halves in about 27 days. The implied long-run daily standard deviation, \(\sqrt{\omega/(1 - \alpha - \beta)} = 1.217\) %, sits next to the sample standard deviation of about 1.24 %: the model’s baseline matches the data. The Dow’s \(\alpha\) of 0.21 is on the high side for an index, a consequence of 2020 sitting inside the window; Apple over 2020–2024 (Section 6.6) has \(\alpha = 0.086\) and \(\beta = 0.885\), a smaller news channel and a longer memory.
The conditional volatility path, and forecasts
Given the parameters, the recursion produces a time series of conditional standard deviations \(\hat\sigma_t\), one per day. Forecasting is then mechanical. Tomorrow’s variance is exact: \(\hat\sigma^2_{T+1} = \omega + \alpha\varepsilon_T^2 + \beta\sigma_T^2\), because everything on the right is known at \(T\). Beyond that, \(\mathbb E[\varepsilon^2_{T+h}] = \sigma^2_{T+h}\), so each further step is \(\hat\sigma^2_{T+h} = \omega + (\alpha+\beta)\,\hat\sigma^2_{T+h-1}\) — a geometric slide from today’s variance toward the long-run level at rate \(\alpha + \beta\). On 13 November 2020, the last day of the sample, \(\varepsilon_T = 1.283\) and \(\sigma_T^2 = 1.913\), so tomorrow’s variance is \(0.0374 + 0.214 \times 1.283^2 + 0.761 \times 1.913 = 1.845\).
The conditional volatility ranges from 0.43 % per day on 9 November 2017 to 8.88 % on 17 March 2020 — 141 % annualised. Volatility is not a number; it is a time series, and the model tracks it one day at a time. The forecast path runs 1.845 → 1.810 → 1.704 → 1.560 at horizons 1, 5, 20 and 60 days, sliding toward the long-run 1.48. A GARCH forecast is a decay path, not a level: the further out you look, the less today matters. One consequence for practice — the variance over the next 20 days is \(\sum_{h=1}^{20}\hat\sigma^2_{T+h}\), not \(20\,\hat\sigma^2_{T+1}\); the “square-root-of-time” rule assumes i.i.d. returns and overstates the horizon variance whenever today’s volatility is above its long-run level, as it usually is when anyone is asking.
From variance to Value-at-Risk, and the honest back-test
Value-at-Risk at level \(p\) is the loss exceeded with probability \(p\). Given tomorrow’s mean and standard deviation it is one quantile away, \(\text{VaR}_p = -(\hat\mu + \hat\sigma_{T+1}\,q_p)\), reported as a positive loss — but which distribution’s quantile? With the normal \(q_{0.01} = -2.326\), \(\hat\mu = 0.082\) and \(\hat\sigma_{T+1} = \sqrt{1.845} = 1.358\), tomorrow’s 1 % VaR is \(-(0.082 + 1.358 \times (-2.326)) = 3.077\) % of book value. But the standardised residuals \(z_t = \varepsilon_t/\hat\sigma_t\) should be checked: GARCH removes clustering, not necessarily all of the fat tail, and if \(z_t\) still has excess kurtosis a Student-\(t\) quantile is the better choice. The unit-variance \(t\) quantile is \(t_p(\nu)\sqrt{(\nu - 2)/\nu}\), so that the \(t\) and the normal are compared at the same variance.
An in-sample hit rate is not a test of a risk model, because the parameters were fitted to the very days being counted. The honest test fits on 2015–2018 (1,006 days), freezes the parameters, runs the variance recursion forward through 2019–2020 (473 days) and counts the days on which the loss exceeded the VaR. The Kupiec test compares the observed hit rate with the nominal \(p\) by a likelihood-ratio statistic that is \(\chi^2_1\) under the null. For comparison the cell also back-tests an unconditional VaR that uses the training-sample mean and standard deviation and never changes.
The standardised residuals still have excess kurtosis 2.43 and a fitted \(\nu = 12.7\): fat tails survive GARCH, just thinner than the raw returns’ excess kurtosis of about 24. Tomorrow’s 1 % VaR is 3.08 % under the normal and 3.23 % under the \(t\). In sample the normal 1 % VaR is breached on 2.5 % of days and the \(t\) on 2.2 % — both too often — while at 5 % both are close to nominal (5.8 % and 6.1 %). Out of sample, 473 days of 2019–2020 should produce about five breaches of a 1 % VaR. GARCH-normal produces 18 (3.8 %, Kupiec \(p < 0.001\)); the unconditional VaR produces 31 (6.6 %). At 5 % the GARCH VaR is breached 32 times (6.8 %, Kupiec \(p = 0.09\), not rejected) and the unconditional one 47 times (9.9 %). The GARCH-normal model is still too thin in the 1 % tail; the fixed-standard-deviation model is simply wrong. Where the breaches fall is the more instructive statistic: the unconditional line at about −2.6 % is crossed ten times in March 2020 alone, while GARCH’s is crossed once that month — by 17 March its \(\hat\sigma\) was already near 6 % and its VaR line had moved out of the way. A risk model’s job is to move before the tenth breach.
Those are the optimiser’s starting values. In decimals \(\omega \sim 10^{-6}\), the likelihood is nearly flat in it and the optimiser stalls; the tell is an \(\omega\) standard error of \(4 \times 10^{-12}\). Scale returns to percent before fitting and read the convergence flag — in percent the same window gives \(\alpha = 0.152\), \(\beta = 0.801\).
A shock to variance decays at rate 0.975 per day (half-life 27 days), so the \(h\)-step forecast slides geometrically from today’s variance toward \(\omega/(1-\alpha-\beta)\). The 20-day variance is the sum of the 20 forecast variances, not \(20\hat\sigma^2_{T+1}\); the square-root-of-time rule overstates it whenever today is stormier than the long run.
Volatility II: Leverage (GJR), ARMA+GARCH, and Volatility Targeting
One extra parameter lets the variance model react differently to losses and to gains. In this section you will test whether it is needed, stack GARCH on top of an ARMA mean, and then do with \(\hat\sigma_t\) what every systematic fund does: not forecast direction, but scale the position — and see what that does to a drawdown.
Does a −2 % day raise tomorrow’s variance more than a +2 % day?
Symmetric GARCH cannot tell the two apart: \(\varepsilon_{t-1}^2\) is the same number. But for equities the evidence is overwhelming that it should. Glosten, Jagannathan and Runkle (1993) added an indicator for a negative shock:
\[\sigma_t^2 = \omega + \left(\alpha + \gamma\,\mathbf 1[\varepsilon_{t-1}<0]\right)\varepsilon_{t-1}^2 + \beta\sigma_{t-1}^2 .\]
A positive shock feeds \(\alpha\varepsilon^2\) into tomorrow’s variance; a negative one feeds \((\alpha + \gamma)\varepsilon^2\). The expected sign of \(\gamma\) for a stock index is positive, for two reasons that go under the name of the leverage effect: a fall in the equity price mechanically raises a firm’s debt-to-equity ratio and hence the riskiness of its equity, and — more important empirically — falls are accompanied by fear, forced selling and demand for protection, all of which raise volatility more than a rally of the same size. Because half of the shocks are negative, persistence under GJR is \(\alpha + \gamma/2 + \beta\). Nelson’s EGARCH captures the same asymmetry in logs; GJR has the advantage that plain GARCH(1,1) is nested inside it by setting \(\gamma = 0\), so a likelihood-ratio test of the leverage effect is free.
The cell generalises the estimator of Section 6.5 by one parameter. Setting the bound on \(\gamma\) to \((0, 0)\) recovers the symmetric model, so one function fits both, and it prints the two log-likelihoods. The LR statistic \(2(\ell_{\text{GJR}} - \ell_{\text{GARCH}})\) is \(\chi^2_1\) under \(H_0: \gamma = 0\), with a 5 % critical value of 3.84; before running it, compute the statistic from the two log-likelihoods you expect the cell to print, −1815.05 and −1836.45.
The log-likelihoods are −1815.05 (GJR) and −1836.45 (GARCH): what is \(LR = 2(\ell_{\text{GJR}} - \ell_{\text{GARCH}})\), and does it clear 3.84?
\(\hat\gamma = 0.254\) and \(LR = 42.8\), with \(p = 6 \times 10^{-11}\): the leverage effect is not in doubt. A −2 % day feeds 0.335 of its square into tomorrow’s variance; a +2 % day feeds only 0.081 — a 4.2-fold asymmetry. The persistence, \(\alpha + \gamma/2 + \beta = 0.974\), is essentially unchanged from the symmetric fit, so the GJR term redistributes the news channel between good and bad days rather than changing how long shocks last. The S&P 500 over the same window tells the same story, with \(\hat\gamma = 0.272\) and \(LR = 38.9\).
The news-impact curve
The cleanest way to see what \(\gamma\) does is the news-impact curve of Engle and Ng (1993): plot tomorrow’s variance as a function of today’s shock, holding yesterday’s variance at its long-run level. For symmetric GARCH the curve is a parabola centred at zero; for GJR it is two half-parabolas with different curvature.
After a −3 % day tomorrow’s variance is 4.11 (a standard deviation of about 2.0 %); after a +3 % day it is 1.82 (about 1.35 %). The symmetric model splits the difference and is wrong on both sides — too calm after a crash, too nervous after a rally. For a risk desk the asymmetry has a direct implication: a down day tomorrow raises the day-after VaR by roughly four times as much as an up day of the same size, and a limit that ignores this is systematically too small after losses, which is exactly when it matters.
ARMA + GARCH: model the mean, then the variance
Everything so far assumed \(r_t = \mu + \varepsilon_t\) — a constant mean. If the mean has ARMA structure, the innovation \(\varepsilon_t\) must be defined first, as the residual of the mean model, and only then can its variance be modelled. The two-step procedure is therefore: fit an ARIMA to \(r_t\), take its residuals, fit GARCH to the residuals. The reverse order makes no sense, because GARCH has no residual to hand back. The one-shot alternative is joint estimation of the mean and variance parameters in a single likelihood, which in Colab the armagarch package provides:
import armagarch as ag
mdl = ag.empModel(re.to_frame(), ag.ARMA(order={'AR':1,'MA':0}),
ag.garch(order={'p':1,'q':1}), ag.tStudent())
mdl.fit(); mdl.summary() # notebook: AR = -0.066, alpha = 0.1, beta = 0.8 (1985-2020)The cell runs the two steps on the Dow: an AR(1) for the mean, then symmetric GARCH on its residuals, and finally the diagnostic that decides whether the job is done — the ACF of the standardised residuals \(z_t = \hat\varepsilon_t/\hat\sigma_t\) and of their squares.
The AR(1) coefficient is \(\hat\phi = -0.195\) with a standard error of 0.008 — apparently a strongly significant negative autocorrelation in the Dow’s daily mean. Now look at the standardised residuals: the ACF of \(z\) at lag one is +0.151, the opposite sign. What happened is that the AR(1) was estimated by ordinary least squares, in which every day counts equally, and it was pulled by the giant reversals of March 2020: a −13.8 % day followed by a +11 % day contributes several hundred times as much to the OLS objective as a normal pair of days. Once each day is weighted by \(1/\hat\sigma_t\), the mean dynamics look different, and the “significant” \(\phi\) is largely an artefact of heteroskedasticity. The ACF of \(z^2\) — 0.03, −0.01, 0.02 — is clean: the variance model has done its job. The mismatch in the mean is exactly why joint estimation, in which the mean equation is implicitly weighted by \(\sigma_t\), is preferred whenever the mean actually matters. For most equity applications it does not, and the constant-mean GARCH of Section 6.5 is the model of choice.
Timing: do calm days pay better per unit of risk?
The lecture notebook asks “when is a good time to buy?” and a volatility model gives one candidate answer. Take Apple over 2020–2024, 1,258 days; fit GARCH(1,1); sort the days into quartiles of the forecast \(\hat\sigma_t\) — which is known at the close of \(t-1\), so this is a legitimate conditioning variable — and compare the next-day returns across quartiles. Then implement the crudest possible rule: hold the stock when \(\hat\sigma_t\) is below a threshold, sit in cash otherwise. The median forecast volatility is about 1.70 % a day; the cell uses a threshold of 2.0 %.
Apple’s GARCH has \(\alpha = 0.086\) and \(\beta = 0.885\) — a smaller news channel and a longer memory than the Dow’s. Across the quartiles the mean daily return is roughly flat, from 0.104 % in the calm quartile to 0.049 % in the stormy one, while the standard deviation more than doubles, from 1.35 % to 2.91 %. The reward does not scale with the risk. Return per unit of daily risk falls from 0.077 in the calmest quartile to 0.017 in the stormiest: low forecast volatility does not predict higher returns, it predicts lower risk for the same return, which for a risk-adjusted investor is the same thing. The threshold rule shows the consequence. At 2.0 % it is in the market 73 % of days, earns a cumulative 95 % against 123 % for buy-and-hold, with a Sharpe of 0.89 against 0.78 and a maximum drawdown of −25.5 % against −37.7 %. Less money, better risk-adjusted money. The rule sat out 146 days in 2020 — the COVID crash and the rebound, because it never forecasts direction, only turbulence — and only 13 days in 2024. An on/off switch is crude, though; the industry version is continuous.
Volatility targeting: size the position by target over \(\hat\sigma_t\)
Instead of in-or-out, hold a fraction of the index that is inversely proportional to forecast volatility,
\[w_t = \min\!\left(\frac{\sigma^\ast}{\hat\sigma_t},\, w_{\max}\right),\]
more when it is calm and less when it is stormy, so that the realised volatility of the position stays near the target \(\sigma^\ast\). This is volatility targeting, and it is how CTAs, risk-parity funds and most systematic books scale every position they hold: the model in Section 6.5 forecasts tomorrow’s \(\sigma\), and the weight is the target divided by that forecast, with a leverage cap. Because the weight uses \(\hat\sigma_t\), which is known at the close of \(t-1\), there is no look-ahead. The cell applies it to the S&P 500 over 2020–2024 with a target of 1 % a day and a cap of 1.5 times. Over these five years buy-and-hold realised 1.35 % a day with a −41 % drawdown; predict what the overlay does to the realised volatility, the drawdown, the Sharpe ratio and the total return before you run it.
The realised daily standard deviation of the overlay is 0.994 % against the 1 % target, where buy-and-hold realised 1.350 %: the scaling did what it was designed to do. The vol-targeted book earns a cumulative 60.7 % with a Sharpe ratio of 0.77 and a maximum drawdown of -22.6 %; buy-and-hold earns 59.9 %, a Sharpe of 0.56 and a drawdown of -41.4 %. Same money, half the drawdown. The weight ran from 0.13 in March 2020, when \(\hat\sigma\) was near 8 %, up to the 1.5 cap in the quiet months of 2021 and 2023–2024, and the leverage in those calm years paid for the de-leveraging in March 2020 and through 2022. The mechanism is the quartile table from the previous subsection: because return per unit of risk is higher when it is calm, a rule that holds more when it is calm raises the Sharpe ratio, and because the drawdowns live in the stormy months, holding less in them roughly halves the drawdown. Nothing here forecasts direction. The entire edge is the variance forecast from Section 6.5, and this is the job that GARCH still owns on every systematic desk.
Volmageddon, 5 February 2018. Through 2017 the VIX averaged about 11 and the Dow’s daily standard deviation was 0.42 % — the calmest year on record. Products that sold volatility, led by the VelocityShares inverse-VIX note XIV with about US$1.9 billion in assets, had returned more than 180 % in the year, and their risk was sized on that sample. On 5 February 2018 the S&P 500 fell 4.1 % and the VIX rose from 17.3 to 37.3 — up 116 % in one session — and XIV lost 96 % of its value after the close. A \(\hat\sigma_t\) estimated on a calm window is the model’s forecast for tomorrow; a persistence of 0.97 means a shock to variance takes weeks to unwind, and the GJR term says losses hit harder than gains. A position sized on 2017’s \(\sigma\) with no room for \(\alpha\varepsilon_{t-1}^2\) was sized for a world without news.
The Swiss franc floor, 15 January 2015. From September 2011 the Swiss National Bank held EUR/CHF above 1.20, and for three years the exchange rate behaved like a textbook stationary series pinned to its floor: ADF rejected, KPSS did not, and the spread to 1.20 mean-reverted within days. Traders treated the floor as a cointegrating rope and sized carry and short-volatility positions on a variance estimated from those quiet years. On 15 January 2015 the SNB abandoned the floor without warning; EUR/CHF fell from 1.20 to 0.85 within minutes — about 60 standard deviations of the pre-announcement daily change. FXCM needed a US$300 million rescue the next day; Alpari UK entered insolvency; Everest Capital’s US$830 million Global fund was wiped out. A stationarity test tells you about the sample you fed it. A regime held in place by a policy is stationary until the policy changes, and no ADF statistic, GARCH \(\hat\sigma_t\) or smoothed regime probability can see the meeting at which it does. Test the spread, then ask who is holding the rope.
Bad news feeds \(\alpha+\gamma = 0.335\) of its square, good news \(\alpha = 0.081\) — a 4.2× asymmetry; on the news-impact curve −3 % gives variance 4.11 and +3 % gives 1.82. Persistence is \(\alpha + \gamma/2 + \beta = 0.974\), unchanged from symmetric GARCH: \(\gamma\) redistributes the news channel, it does not lengthen memory.
From the variance forecast alone. Return per unit of risk is higher in calm periods (Apple quartiles: 0.077 vs 0.017), so \(w_t = \min(\sigma^\ast/\hat\sigma_t, w_{\max})\) holds more when it is calm and less when it is stormy; realised sd lands at 0.99 % vs 1.35 %, the Sharpe rises from 0.56 to 0.77 and the drawdown falls from −41.4 % to −22.6 %.
Chapter Wrap-up
You can now do the four things a time-series toolkit is for on a modern desk, and — just as important — you can say with numbers what it is not for.
Foundation. Given any series, you run KPSS and ADF as a pair with both "c" and "ct", place the result in the decision table, and either detrend, difference or walk away; you read the ACF and PACF for the memory of the mean and, separately, the memory of the size; and you fit an AIC-chosen ARIMA as the baseline every fancier model must beat, holding it out and scoring it against a naive forecast rather than trusting the information criterion. On daily returns that baseline is a constant — ARIMA(0,0,0) — and the AIC winner on Apple lost to it out of sample (Sharpe −0.20 against 1.16). ML base. You score forecasts with MAE, RMSE and directional accuracy on the same rows against the zero forecast; you build features that are stationary and shifted; you validate walk-forward with an expanding window; and you know, from the gradient-boosting model that did all of this on the S&P 500 and still scored DA 0.494, that the edge machine learning found lies in the cross-section and in richer features, not in a single series’ own past. Structural arbitrage. You test a candidate pair with coint rather than a plain ADF on the residual, separate the formation window from the trading window, re-tie the rope walk-forward, and trade the z-score of a spread with yesterday’s position — the GOOG/S&P 500 book earned a Sharpe of 0.75 with a third of the stock’s drawdown, and NVDA/SPY was correctly rejected. You filter a hidden level or beta with six lines of Kalman recursion, and a hidden regime with one MarkovRegression call whose filter cut the S&P 500 drawdown from −41.4 % to −11.3 %. Volatility. You estimate GARCH(1,1) by maximum likelihood in fifteen lines, in percent; you forecast variance as a decay path and convert it into a VaR that you back-test with Kupiec; you test for the leverage effect and size positions by target over \(\hat\sigma_t\), which gave the S&P 500’s return with half its drawdown.
That list is the positioning statement of the chapter, restated as skills. Time-series models remain important in quantitative trading, but their role has shifted — to volatility modelling, where a variance forecast underlies every risk limit and position size; to structural arbitrage, where cointegration, Kalman filters and regime switching turn a relationship or a hidden state into something tradable; and to serving as the foundation of the machine-learning pipeline, where stationarity, lags, walk-forward validation and baselines decide whether a model is real. Forecasting the mean return of a single series — the job most textbooks open with — has largely been taken over by machine learning on the cross-section, and the honest out-of-sample numbers in Section 6.2 are the reason. The classical models lost the mean-forecasting job and kept the variance, the spread, the regime and the pipeline.
Where this leads. Chapter 7 takes the four inherited disciplines and the cross-sectional turn as given and builds the methods that investment desks use on top of them: learning to rank the cross-section rather than forecast one series; quantile and conformal prediction, which replace a point forecast with an interval that the GARCH residuals of this chapter show is needed; double machine learning; covariance shrinkage and hierarchical risk parity, which generalise the volatility-targeting weight from one asset to a whole book; triple-barrier labels and meta-labelling, where the regime and volatility filters of Sections 5.4–5.6 become the bet-sizing model; and the backtest statistics that decide whether a Sharpe ratio like the pairs book’s 1.09 at a tuned threshold is evidence or noise.
The companion slide deck, Chapter 6 — Time Series Models for Trading and Risk, follows the same six sections with the same data, seeds and printed numbers, adds multiple-choice checkpoints and practice cells for each, and closes with a decision memo, a set of copilot prompts and four discussion questions. Read it alongside this chapter; the two are built to agree.
← Chapter 5 · Contents · Chapter 7: Modern Statistical Learning for Investing →