Chapter 7: Modern Statistical Learning in Practice
Chapter Introduction
Chapters 3 and 5 ended in the same place. A gradient-boosting model, trained with every discipline the earlier chapters taught — stationary features, lags that do not leak, an expanding walk-forward window, a baseline to beat — scored a directional accuracy of 0.494 on one daily series and a rank IC of −0.015 on a cross-section of 213 units. Honest numbers are small. That is not a failure of the method; it is the result, and it is the starting point of this chapter, because the interesting work in applied statistics begins after a prediction has been made and turns out to be weak. Six questions then arise, and they arise in the same form wherever predictions are turned into decisions. Which items belong at the top of an ordered list, when only the order will be used? How wide is the uncertainty around a prediction, and can the width be guaranteed? What does one variable do to an outcome once many others are held fixed, in whatever shape they enter? How should a budget be allocated across many correlated risks when the covariance between them is itself estimated? What should the label be when a decision system has two stages and the event it cares about has a path, not just an end point? And, before any of it is acted on, is the best of everything that was tried more than the best that a search through noise would have produced?
Each of the six sections answers one of those questions with a method that is in daily use far beyond the data it is demonstrated on. Section 7.1, learning to rank, is the machinery behind a search engine’s results page, a recommender’s feed and any triage — admissions, credit applications, résumés — in which a fixed number of slots go to the top of an ordered list and the scores themselves are never shown; it is demonstrated on the §4.5 cross-section, where a top-20 portfolio plays the role of the slots. Section 7.2, quantile and conformal prediction, is how a retailer turns a demand forecast into a safety stock, how a clinician turns a risk model into a band that can be acted on, and how any regulated use of a predictive model attaches a coverage guarantee that an auditor can check; it changes one argument of the §6.2 boosting model, calibrates the result on a held-out year, and watches the guarantee break under a regime change. Section 7.3, double machine learning, is the tool of policy evaluation, marketing uplift and pricing experiments on observational data — any setting where “did the treatment cause the outcome, or did the targeting?” must be answered with many controls of unknown shape; it takes the §4.4 confounder and the Frisch–Waugh–Lovell logic and replaces two linear fits by the §4.3 boosting machine. Section 7.4, covariance shrinkage and cluster-based allocation, applies wherever an estimated covariance is inverted — genomics with thousands of genes and hundreds of samples, sensor networks, budget allocation across correlated risks — and is demonstrated on 21 correlated return series, where the §6.6 sizing rule for one asset becomes an allocation across a whole set. Section 7.5, event-based labels and meta-labelling, is the design of two-stage decision systems — a fraud alert and the model that decides whether to block, a screening test and the model that decides on referral, a lead-scoring rule and the model that decides how much effort each lead deserves — and it begins by asking what the label of the second stage should be when the event has a path; the §6.5 volatility estimate sets the scale of that path. Section 7.6, selection statistics, is the central problem of large-scale A/B experimentation, of model selection over a hyper-parameter grid, and of genome-wide or feature-wide screening, where tens of thousands of hypotheses are tested and the smallest p-value is guaranteed to look impressive; it is demonstrated on 42 rules applied to one daily series, with a bootstrap interval, a deflated performance measure, the probability of overfitting from combinatorial cross-validation, and the Benjamini–Hochberg correction that extends the §3.2 hypothesis test to many hypotheses at once.
Market data is the demonstration vehicle throughout, for three reasons. It is public and plentiful, so every cell runs in your browser on the real thing. It is unforgiving: the signal-to-noise ratio is so low that any leak, any look-ahead and any uncounted trial shows up immediately as a result too good to be true, which makes it the best possible training ground for the discipline. And the results are honest, which is worth saying before you read a single cell. No ranking loss finds an order in five characteristics on 213 units that 71 periods can distinguish from zero. The conformal band’s coverage collapses to 0.18 in the month the distribution moves. The best of 42 rules has a deflated performance score of 0.637, an overfitting probability of 0.533 and no survivor of a false-discovery correction; on a second series it ranks 32nd of 42. The partial effect of one characteristic on the next period’s outcome has a t-statistic of 1.4 once the standard error is clustered by period. What the chapter delivers is not a result; it is the ability to say, with a number, that there is not one — and, where there is something (shrinkage halves the turnover of an allocation at the same risk; a second-stage model cuts a first-stage rule’s worst loss from −46 % to −3.9 %), to say how far the number can be trusted.
Each method is explained from first principles before its cell: why a ranking objective differs from squared error and what the rank IC measures; what the pinball loss minimises and why a held-out quantile of non-conformity scores guarantees coverage under exchangeability; what Neyman orthogonality buys and why a row that trained the nuisance model cannot also testify about the effect; why the eigenvalues of a sample covariance are spread out even when the truth is the identity, and what shrinkage and hierarchical clustering do about it; how three barriers define a label and why a label that looks ten steps ahead needs a ten-step embargo; and what the expected maximum of \(N\) noise trials is, which is the number every selected result must be measured against. The mathematics is kept to what you need to read the output; the references are given in the prose so that you can go further.
The data are the files of the earlier chapters — the §4.5 monthly cross-section of 213 units (xs_panel_monthly.csv), two daily series (sp500.csv, dji.csv) — and one new file of 22 daily series from 2015 to 2024 (etf_daily_prices.csv). Every experiment is walk-forward, every number is before costs, and the cross-section file is survivorship-biased — the §4.5 warning applies to every spread in Section 7.1 and every effect in Section 7.3. Every cell runs in your browser with a fixed seed, and every number in the prose is printed by a cell; the code is written so that a different panel — patients, customers, genes, sensors — can be substituted for the one used here without changing a method. The companion slide deck follows the same six sections with the same data, seeds and numbers, so the two can be read side by side.
Table of Contents
- Learning to Rank
- Quantile and Conformal Prediction: Intervals, Not Points
- Double Machine Learning: Effects with Many Controls
- Covariance Shrinkage and Cluster-Based Allocation (Hierarchical Risk Parity)
- Event-Based Labels and Meta-Labeling: Triple Barriers and Two-Stage Decisions
- Selection Statistics: Deflated Performance, Overfitting Probability, and 42 Rules on Trial
Learning to Rank
Learning to rank is the machinery behind a search engine’s results page, a recommender’s feed, a hospital’s triage queue and a lender’s or a university’s shortlist: a fixed number of slots go to the items at the top of an ordered list, the scores that produced the order are never shown, and a model trained to predict a level — a click probability, a repayment amount, a return — is being trained on something the decision never uses. The demonstration here is the §4.5 cross-section, where you trained a boosting model to minimise squared error on next month’s return and then built a top-20 / bottom-20 portfolio from its predictions; the portfolio played the role of the slots and used only the order. This section takes that observation seriously. You will show that a monotone transform of the predictions leaves the portfolio untouched while multiplying the mean squared error a hundredfold; define the score that matches the decision; then train three models that target the order directly — a rank target, a pairwise logistic loss on pairs of items (the RankNet loss that web search introduced), and the §4.5 trees on ranks — and put all of them through the §4.5 walk-forward protocol without changing it.
The same matrix, one new column
The object is the one from §4.5: at each month-end \(t\) a matrix \(X_t\) of \(N\) stocks by \(K\) characteristics and a vector \(y_t\) of next month’s returns. The five characteristics are last month’s return, 12-1 momentum, 21-day realised volatility, log dollar volume and price as a fraction of the 252-day high, and they are standardised within each month — median and standard deviation across the stocks observed at \(t\), clipped at ±3 — for the reason §4.5 gave: a full-sample z-score would use, for a 2015 row, the mean and spread of nine years that had not yet happened. The cell reproduces that transform and adds one column: the target’s within-month percentile rank, rank(pct=True) inside each month, centred at zero by subtracting one half.
25 335 rows, 119 months, 213 stocks — the §4.5 panel. Compare the two target columns. ret_next has mean 0.013 and standard deviation 0.082, with a range from −0.735 to +0.805: a handful of stock-months in which a name lost three quarters of its value or nearly doubled. A squared-error loss is a sum of squared deviations, and a deviation of 0.8 contributes 64 times as much as a deviation of 0.1; those few months dominate any MSE fit, and the model spends its capacity on explaining them. y_rank runs from −0.495 to +0.500 with standard deviation 0.289, and — this is the point — it has that range and that spread in every month, because a percentile rank is by construction uniform on \([0, 1]\) within each group. The rank target has thrown away the level of returns and kept the order; no month can dominate the fit because every month contributes exactly the same distribution of targets.
Why a top-K portfolio only needs the order
Take any prediction \(\hat y\) and replace it by \(3\tanh(50\hat y)\). The transform is strictly increasing, so it preserves every pairwise comparison, and it is wildly nonlinear, so the values themselves are unrecognisable. What happens to the top-20 / bottom-20 portfolio, and what happens to the mean squared error? The cell uses October 2022, a “prediction” equal to 1 % per unit of momentum z-score, and its transform.
Same twenty names long, same twenty short, the same realised spread of −11.94 % (October 2022 was a month in which momentum lost: the high-momentum names fell hardest) — and an MSE of 1.4686 against 0.0134, 110 times larger. Both the top-20 and the bottom-20 are chosen by sorting the predictions, and any strictly increasing transform preserves a sort, so the two portfolios and their realised spread are identical while the squared error can be anything at all. The MSE measures something the trade never uses. A squared-error model spends its capacity on the level of returns, which §6.2 showed is almost all noise; a ranking model spends it on the order, which is all the portfolio uses. The question the section answers is whether redirecting that capacity finds anything.
Does the rank IC see the transform?
The scoreboard of §4.5 was the rank IC — the Spearman correlation between the prediction and the realised return across the month’s 213 stocks. Spearman’s coefficient is Pearson’s coefficient computed on ranks, so it depends on the predictions only through their order. Before running the cell, decide: is the rank IC, like the portfolio, blind to the monotone transform — and is Pearson’s?
Will spearmanr(p1, y)[0] and spearmanr(p2, y)[0] agree to four decimals for October 2022, and will the two Pearson correlations?
Spearman −0.3316 for both; Pearson moves from −0.3376 to −0.3329. The rank IC records the month’s verdict on momentum — a strongly negative order, the high-momentum names at the bottom of the realised returns — with one number that no monotone transform can touch, exactly as the portfolio cannot be touched. Pearson moves because it sees the values. The rank IC is the score that matches the trade, and it is the first column of every table in this section. Its t-statistic over \(T\) months is \(\bar{IC}/\sigma_{IC}\cdot\sqrt T\), and its standard error is \(\sigma_{IC}/\sqrt T\) — a number you will see is about 0.02 here, and which every mean IC below must be read against.
The scoreboard, and the baseline: least squares on the raw return
The protocol is §4.5’s, unchanged. Test months run from January 2019 — 71 test months — and the model is refitted every 12 months on all rows dated strictly before the block, so 6 refits, the first on 10 212 rows. For every model two scores are computed per month: the rank IC, and the top-20 minus bottom-20 spread, the equal-weight return of the 20 highest-ranked stocks minus the 20 lowest, before costs. The spread’s annualised Sharpe is \(\bar s/\sigma_s\cdot\sqrt{12}\). The first model through the protocol is the baseline the section has to beat: a linear regression on the five z-scores with the MSE loss — the loss of every regression in Chapter 4.
Mean IC +0.0014 with t = 0.06: the linear MSE model ranks no better than a coin. Its extremes nevertheless earn a spread of +8.5 % a year with a Sharpe of 0.34, the pattern §4.5 also found — the tails of a ranking can do a little better than the ranking as a whole, because the tails are where the model is most confident. The last refit’s coefficients are worth reading as economics. A one-standard-deviation move in 12-1 momentum is worth +0.258 % a month, in volatility +0.264 %, in price-to-high −0.224 %; and last month’s return gets −0.004 %, nothing. The short-term reversal effect — last month’s winners tend to give some of it back — is among the best-documented regularities in the cross-section, and the MSE loss cannot see it, because the loss is dominated by the ±80 % months in which nothing about last month’s return mattered.
(a) Change the target: regress on the rank
The simplest way to target the order is to keep the model and change the column it is fitted to. Replacing ret_next by its within-month percentile does not change the model class, and it does not change the features, which are already ranks in spirit (z-scores within the month). What it changes is the weighting of months: every month now contributes the same spread of targets, so the ±80 % months no longer dominate, and the model listens equally to the quiet months in which the order was subtle.
IC −0.0038 (t = −0.17), spread +5.0 % a year, Sharpe 0.24. Not better. With five price-based characteristics on 213 survivors there is little order to learn, whichever target you choose. But note the coefficient signs, which have now changed: last-month return is negative (−0.0038 rank units per z-unit — short-term reversal) and 12-1 momentum is positive (+0.0084). These are the two best-known cross-sectional anomalies, and they appear with the right signs the moment the extreme months stop shouting. The effects are faint, and at this sample size faint is indistinguishable from absent; the point is that the rank target changed what the model listens to, not that it found something.
(b) Learn from pairs: the RankNet loss
A ranking is a statement about pairs. Within month \(t\), stock \(i\) should score above stock \(j\) whenever \(r_i > r_j\); there is nothing more to a ranking than the set of such comparisons. RankNet (Burges et al., 2005), the loss at the origin of modern learning-to-rank in web search, models the probability that \(i\) is preferred to \(j\) as a logistic function of the difference of two scores,
\[P(i \succ j) = \sigma\big(f(x_i) - f(x_j)\big), \qquad \sigma(u) = \frac{1}{1 + e^{-u}},\]
and maximises the log-likelihood of the observed orderings. With a linear scorer \(f(x) = w^\top x\) the difference of scores is \(w^\top(x_i - x_j)\), so the model is a logistic regression, without intercept, on pair differences, with label \(\mathbb 1[r_i > r_j]\). No intercept, because swapping \(i\) and \(j\) negates both the feature difference and the label, and an intercept would break that symmetry. Three design decisions matter. Pairs are formed within a month only — the model never compares a stock in October 2022 with a stock in March 2020, because the trade never does. With 213 stocks a month has \(\binom{213}{2} = 22\,578\) pairs and 119 months give 2.7 million; the cell subsamples 400 random pairs per month with a fixed seed, which gives the same \(w\) to two decimals at a hundredth of the runtime. And the score \(f\) is only ever used to sort — which is all this section has asked of any model.
IC −0.0046 (t = −0.21), spread −1.8 % a year, Sharpe −0.09, from 42 986 pairs at the last refit. The weights say what the rank regression said — reversal (−0.031 on last month’s return) and momentum (+0.026) — and the model is about as good as the other two, which is to say indistinguishable from zero. This deserves a plain statement, because the pairwise loss is the right loss for a ranking problem and it is tempting to expect the right loss to work: a loss decides what a model estimates; it cannot manufacture an order that five characteristics on 213 stocks do not contain.
(c) The §4.5 boosting machine, on both targets
The third approach keeps the §4.5 specification exactly — HistGradientBoostingRegressor with max_depth=2, learning_rate=0.05, max_iter=200, min_samples_leaf=50, the recipe from §4.3 in its histogram-based form — and changes only the target column, once to ret_next (the §4.5 model, which should reproduce the §4.5 number) and once to y_rank.
The §4.5 number returns exactly: IC −0.0150 (t = −1.04) with the MSE loss, spread +7.7 %, Sharpe 0.47. On the rank target the same trees on the same folds give IC +0.0165 (t = +0.98), the best spread of the five at +9.3 % a year, and the best Sharpe, 0.64. The sign of the IC flipped with the target. Read that carefully: a change of target that moves an IC from −0.015 to +0.017 — a difference of 0.03, about one and a half standard errors — is telling you the difference between the two is noise, not that ranks are magic. A reader who saw only the second line would conclude that ranking losses work; a reader who saw only the first would conclude that trees cannot rank. Both lines were produced by one protocol, and the protocol’s verdict is that neither conclusion is available.
Five models, one honest picture
The cell overlays the five cumulative spreads and prints two diagnostics: the standard error of a mean IC over 71 months, and the correlation matrix of the five monthly IC series, which says whether the models are learning the same order or different ones.
The standard error of a mean IC is 0.02, and every one of the five mean ICs — +0.001, −0.004, −0.005, −0.015, +0.017 — lies within one standard error of zero. The five cumulative spreads fan out from the same whipsaw in the spring of 2020 and the visual impression of a winner is an impression of the last two years. The correlation matrix is the more instructive output. The three rank-based models move together: OLS-rank and pairwise correlate 0.92, pairwise and HGB-rank 0.87, OLS-rank and HGB-rank 0.81. The MSE trees are the outlier, correlating 0.31 with OLS, 0.26 with OLS-rank, 0.20 with the pairwise model and 0.23 with the rank trees. So the ranking losses did what they promise — they changed which order the model learns, and three quite different estimators of that order agree with each other — but 71 months of a survivorship-biased file cannot tell a Sharpe of 0.6 from 0. The deliverable is the protocol and the full table, not the best line.
Where this leads: preference models
The pairwise loss you just fitted is a preference model. Its data are statements of the form “\(i\) was better than \(j\) this month” and its score \(f(x)\) is a latent utility that rationalises them — the same structure as a Bradley–Terry model of sports results or the reward model behind a modern language model, where human raters compare pairs of answers. Push the idea further and it becomes a research programme. A ranking loss estimates the order of expected returns directly, which is what a long-short portfolio, a top-decile screen or the sort inside a stochastic discount factor actually needs; squared error estimates a level nobody trades. The same machinery scales from a linear \(w\) to the attention scorer of §4.5 — Kelly, Kuznetsov, Malamud and Xu train their cross-sectional transformer end-to-end on a portfolio Sharpe ratio rather than a squared error, which is another way of saying that they train on the trade’s objective rather than a proxy — and from pairs to listwise losses that weight the top of the order more heavily, because the top-20 matter more than the middle 170. Estimation, inference and portfolio construction can then be done in one step, on one objective, instead of a regression followed by a sort. That is the theme of Prof. Wan’s current research on ranking-loss and preference-based asset-pricing models.
Nothing in this subsection alters the verdict of the previous one. On five characteristics and 213 survivors, no loss finds a reliable order. The loss decides what you estimate; the data decide whether there is anything to find. A more ambitious objective on the same small file would produce a more ambitious-looking number with the same standard error.
How concentrated should the portfolio be?
evaluate builds a top-\(K\) / bottom-\(K\) spread with \(K = 20\). A more concentrated book (\(K = 10\)) trades only the names the model is surest about; a more diversified one (\(K = 40\)) averages over more of the order. The cell re-scores the rank-target boosting predictions at all three values.
\(K = 10\): spread +20.7 % a year, Sharpe 1.10; \(K = 20\): +9.3 %, 0.64; \(K = 40\): +8.9 %, 0.75. The mean IC is the same 0.0165 in every row, because \(K\) changes the portfolio, not the ranking. A Sharpe of 1.1 from the ten most extreme names looks like a discovery. Hold that thought until Section 7.6, where you will learn what the best of three values of \(K\) — or of five losses, or of 42 rules — is worth once the search that found it is counted. Over 71 months none of the three Sharpe ratios is separable from the others, or from zero: the standard error of an annualised Sharpe over \(T\) months is roughly \(\sqrt{12/T} \approx 0.41\).
Nothing. Both the portfolio and the Spearman rank IC depend on the predictions only through their order, and a strictly increasing transform preserves every pairwise comparison. In October 2022 the spread stayed at −11.94 % and the rank IC at −0.3316 while the MSE went from 0.0134 to 1.4686. The MSE measures the level of the predictions, which a top-K portfolio never uses.
No. The standard error of a mean IC over 71 months is about 0.02, so the difference of 0.03 is about one and a half standard errors — noise. What the ranking losses demonstrably changed is which order the model learns: the three rank-based models’ monthly ICs correlate 0.81–0.92 with each other and only 0.20–0.31 with the MSE trees. Whether that order is worth anything, 71 months of a survivorship-biased file cannot say.
Quantile and Conformal Prediction: Intervals, Not Points
The §6.2 boosting model produced one number per day and was worse than predicting zero. Most decisions do not need the number; they need the range. A retailer setting safety stock needs the 95th percentile of next week’s demand, not its mean; a clinician needs a risk band that can be acted on; a regulated use of any predictive model increasingly needs an interval whose coverage can be audited; and a risk desk needs to know how wide tomorrow’s range is for sizing and for stops. The demonstration is daily index returns, where §6.5 showed that the size of tomorrow’s move, unlike its sign, is forecastable — so an interval is the object the data can actually support. This section builds it directly. You will fit three quantile models on the §6.2 features by changing one argument of the §6.2 estimator, discover that their band is a good shape with a wrong size, calibrate it with a split-conformal step that comes with a coverage guarantee, watch the guarantee fail in March 2020 and turn the failure into a regime alarm, and finally use the band’s width as the §6.6 volatility-targeting weight without a GARCH.
The loss that estimates a quantile
A squared-error loss is minimised by the conditional mean. To estimate a conditional quantile you need a loss that is minimised by the quantile, and the one that does it is the pinball (check) loss at level \(\tau \in (0, 1)\):
\[\ell_\tau(y, q) = \begin{cases} \tau\,(y - q) & y \ge q \\ (1-\tau)\,(q - y) & y < q. \end{cases}\]
Under-predicting (the truth lands above \(q\)) costs \(\tau\) per unit of distance; over-predicting costs \(1 - \tau\) per unit. To see which constant \(q\) minimises \(\mathbb E[\ell_\tau(y, q)]\), differentiate with respect to \(q\): the derivative is \(-\tau\,P(y > q) + (1 - \tau)\,P(y < q)\), and setting it to zero gives \(P(y < q) = \tau\). The minimiser is the \(\tau\)-quantile of \(y\). The loss is asymmetric by exactly the odds \(\tau : (1 - \tau)\), which is what pulls the solution away from the centre of the distribution: at \(\tau = 0.95\) a miss below costs nineteen times a miss above, so the fitted value sits where only one observation in twenty lands above it. At \(\tau = 0.5\) the two costs are equal and the minimiser is the median; the pinball loss at 0.5 is half the absolute error. Everything in Chapter 4 about fitting a loss with trees carries over unchanged: HistGradientBoostingRegressor(loss="quantile", quantile=τ) boosts depth-2 trees on the pinball loss instead of the squared error, and three fits — at \(\tau\) = 0.05, 0.5 and 0.95 — give a lower band, a median and an upper band that all depend on the features.
The §6.2 features, split three ways
The features are the eleven from §6.2 — five lagged returns, 5- and 21-day rolling means and standard deviations, 12-1 momentum and the ratio of short to long volatility — every one computed and then shifted, so that the feature on day \(t\) uses returns up to \(t - 1\) only (the §6.2 debug block is the reason). What is new is the split. §6.2 used training and test; this section adds a calibration window between them, a year of data that is never used to fit a tree and whose only job is to measure how wrong the trees are.
753 training days from 5 January 2016 to the end of 2018, 252 calibration days in 2019, and 1 258 test days from 2020 to 2024 — a test period that contains the March 2020 crash, the 2022 bear market and two calm years. The calibration year is a quiet one, which will matter: whatever the band learns about its own errors in 2019, it learns from a year with no crash in it.
Three quantile models
The median band averages 0.053 % — a point forecast as useless as §6.2’s, and for the same reason. The bands are not useless. On the quietest day of the training sample (12 October 2017, the lowest std21) the model says tomorrow’s return lies in [−0.52 %, +0.71 %]; on the stormiest (27 December 2018) it says [−2.31 %, +2.01 %]. The width of the band correlates 0.818 with the 21-day volatility feature. The quantile model has rediscovered GARCH’s message from §6.5 with no parametric model of variance at all: the size of tomorrow’s return is forecastable from the recent size of returns, and the sign is not. What it has that GARCH does not is a band that can also be asymmetric and that can depend on any feature in the matrix — the mean, the momentum, the volatility ratio — in any shape the trees find.
Do the bands keep their promise?
The 90 % band \([\hat q_{0.05}(x), \hat q_{0.95}(x)]\) should contain the realised return on 90 % of days. Where will empirical coverage be lowest — on the training rows, on the calibration year, or on the test years? Think about what each model was fitted to before you run the cell.
Will the 90 % band’s coverage be near 0.90 on all three windows, or will it fall from training to calibration to test — and by how much?
0.895 in training, 0.853 on the calibration year, 0.773 on the test years: one day in four falls outside a band that promised one in ten. The pattern has two causes. The 5 % and 95 % models were fitted to the training rows, so the band is tuned to those rows and covers them at about the nominal rate; on unseen rows the trees have over-fitted the training spread a little, and the band is slightly too narrow even in a calm year like 2019. Then 2020–2024 adds a volatility regime the training set never saw, and the shortfall doubles. A quantile model’s band is a prediction of an interval, subject to every error a prediction can have; it is not a guarantee. The fix does not retrain anything. It measures the miss on data the model never saw, and widens the band by that much.
Split conformal: a guarantee from one held-out quantile
Conformal prediction (Vovk and co-authors; the split-conformal form used here is Romano, Patterson and Candès, 2019, “Conformalized Quantile Regression”) keeps the model’s shape and fixes its size with one number. For each of the \(n\) calibration days compute the non-conformity score — how far outside the band the truth fell, negative if it fell inside:
\[s_i = \max\big(\hat q_{0.05}(x_i) - y_i,\; y_i - \hat q_{0.95}(x_i)\big), \qquad i = 1, \dots, n.\]
Sort the scores and take \(\hat q\) = the \(\lceil (n+1)(1-\alpha) \rceil\)-th smallest. The conformal band on a new day is \([\hat q_{0.05}(x) - \hat q,\; \hat q_{0.95}(x) + \hat q]\): the model’s band, pushed out on both sides by \(\hat q\) (or pulled in, if \(\hat q\) is negative because the raw band was already too wide).
If the calibration and test days are exchangeable — any permutation of their joint distribution is equally likely — then \(P\big(y_{\text{new}} \in \text{band}\big) \ge 1 - \alpha\), in finite samples, for any underlying model, however badly fitted.
The proof is short enough to carry in your head, and it tells you exactly what the guarantee does and does not depend on. The new day’s score \(s_{\text{new}}\) is inside the band exactly when \(s_{\text{new}} \le \hat q\). Under exchangeability, the \(n + 1\) scores \(s_1, \dots, s_n, s_{\text{new}}\) are exchangeable too, so the rank of \(s_{\text{new}}\) among them is uniform on \(\{1, \dots, n+1\}\). The probability that \(s_{\text{new}}\) is at or below the \(k\)-th smallest calibration score is therefore at least \(k/(n+1)\), and with \(k = \lceil (n+1)(1-\alpha) \rceil\) that is at least \(1 - \alpha\). Nothing about the model entered the argument; the model decides where the band is wide and where it is narrow (the shape), and the calibration score decides its size. The whole weight of the guarantee rests on the word exchangeable. Daily returns in 2019 and daily returns in March 2020 are not draws from one urn, and the cells that follow measure exactly how far that assumption breaks.
Calibrate on 2019, test on 2020–2024
With \(n = 252\) and \(\alpha = 0.10\), \(k = \lceil 253 \times 0.9 \rceil = 228\), and the 228-th smallest of the 2019 scores is +0.182 %: the 2019 misses say “widen each side by 0.18 %”, which takes the mean width from 2.58 % to 2.94 %. Coverage on 2020–2024 rises from 0.773 to 0.830 — better, still short of 0.90. By year: 0.727 in 2020, 0.869 in 2021, 0.789 in 2022, 0.892 in 2023, 0.873 in 2024. The guarantee held, approximately, where 2019 resembled the future (2021, 2023, 2024), and failed where it did not (2020, 2022). That is what the proof said would happen: the guarantee is a statement about exchangeable data, and the two years in which it failed are the two years in which the return distribution was wider than anything in the calibration year.
The debug block below is the most common way to get this step wrong, and the output looks innocent.
The cell runs and reports \(n = 753\), \(k = 679\), \(\hat q\) = 0.000 %, and a test coverage of 0.773 — exactly the raw band’s. Three times as many scores produced a correction of zero. The bug is that the training rows are the rows the quantile models were fitted to, and on those rows the 90 % band already covers 89.5 % of days, so the 90th percentile of the training scores sits at the band’s edge and the correction vanishes. The scores of rows a model has seen say nothing about its errors on rows it has not; the guarantee’s proof needs the calibration scores and the new score to be exchangeable, and a score from a row that shaped the model is not exchangeable with one from a row that did not. This is the §4.3 train/test rule in a new costume: the calibration window must be disjoint from the training window, and “more data” is worth nothing if it is the wrong data.
See the band through the crash
The band widens as the std21 feature catches up with the market — a half-width of about ±1.0 % in January and ±2.6 % by 17 March — but the returns of late February and March run ahead of it. Three misses in January, 6 in February, 18 in March, 11 in April, 6 in May, 5 in June. A band built from the features of the last 21 days is a lagging description of a crash: on the day the distribution moves, the band still describes the distribution of three weeks ago, and it takes those three weeks to catch up. Count the misses month by month and the lag becomes something useful.
March 2020: coverage as a regime alarm
January 0.86, February 0.68, March 0.18, April 0.48, May 0.70, June 0.77. In March 2020 the band caught fewer than one day in five: 45 % of days broke through the floor and 36 % through the ceiling — the misses were on both sides, because a crash is a widening, not a shift. The 21-day rolling coverage fell to 0.143 on 30 March. The four worst months of the whole test period are March and April 2020 and June and October 2022, each at 0.62 or below. A band whose coverage collapses is telling you that the distribution has moved, with a lag of a few days, and it does so with no regime model at all — none of the Markov-switching machinery of §6.4, only the count of hits and misses against a promised rate. The coverage series is a regime detector you get for free from a prediction interval, and in Section 7.5 and the chapter’s decision memo it is attached to every live signal as an alarm.
Why the guarantee failed, and what restores it
What failed in March 2020 was not the quantile model, not the formula for \(k\), and not the choice of \(\alpha\). It was the exchangeability assumption: the scores of March 2020 came from a distribution roughly five times wider than 2019’s, and 2019’s scores said nothing about them. The natural repair is to make \(\hat q\) track the recent score distribution — re-calibrate on a trailing window, so that the calibration set is always the most recent year of days the model has not trained on. The cell computes the scores on the test years too (each is known one day after the fact), rolls a 252-day 90 % quantile across the concatenated scores, and shifts it by one day so that today’s band uses only scores that were known at yesterday’s close.
A trailing 252-day calibration lifts coverage to 0.893 — on target on average, with 0.84, 0.94, 0.84, 0.96 and 0.88 by year — but March 2020 is still 0.18: the window learns the new width only after the misses have happened (April 0.67, May 0.90, June 0.91). This is the honest limit of the method, and it should be stated as such. Conformal prediction gives you a coverage that is right on average, and a calibration that adapts restores that average after a regime change within a few weeks; it cannot see a regime before it arrives. What it can do is report the change faster than a quarterly model review — the coverage counter was below 0.5 by the second week of March — and that is the service it provides to a risk desk.
Sizing by width: the §6.6 overlay without GARCH
§6.6 held a fraction \(w_t = \min(\sigma^\ast / \hat\sigma_t, w_{\max})\) of the index, where \(\hat\sigma_t\) was a GARCH forecast known at the close of \(t - 1\). The conformal band’s width is a model-free measure of tomorrow’s range that is also known at that close, and it can play the same role: target a 2 % band, cap the weight at 1.5.
The weight averages 0.76, sits at the 1.5 cap in calm months and falls to 0.37 at its lowest in the spring of 2020. The realised daily standard deviation is 0.796 % against 1.350 % for buy-and-hold. Cumulative return 41.9 % against 59.9 %, Sharpe 0.66 against 0.56, maximum drawdown −21.2 % against −41.4 %. The §6.6 overlay, with a GARCH forecast, did slightly better (Sharpe 0.77, drawdown −22.6 %, and it kept the full 60 % return); the interval width gets most of the way there with no parametric variance model, and with a coverage guarantee and a coverage alarm attached. The mechanism is the one §6.6 explained: return per unit of risk is higher when it is calm, and the drawdowns live in the stormy months, so a rule that holds more when the band is narrow and less when it is wide raises the Sharpe and halves the drawdown without forecasting direction.
Trade coverage for width
The confidence level \(\alpha\) is a choice, and the choice has a price list. The cell re-runs the calibration at \(\alpha\) = 0.20 (an 80 % band), 0.10 and 0.05, holding the quantile models fixed and changing only \(k\) and \(\hat q\).
\(\alpha\) = 0.20: \(\hat q\) = −0.136 % — the band shrinks, because the raw 5/95 band already over-covers 80 % — coverage 0.714, width 2.31 %. \(\alpha\) = 0.10: \(\hat q\) = +0.182 %, coverage 0.830, width 2.94 %. \(\alpha\) = 0.05: \(\hat q\) = +0.388 %, coverage 0.887, width 3.35 %. Each step of coverage costs about half a per cent of width, and the shortfall against the nominal level is about 0.07–0.09 at every \(\alpha\), because the shortfall comes from the regime change, not from the level. That is the price list a risk manager should have before choosing the confidence level of a stop or a limit; the pinball loss and one held-out year produce it in a cell.
With \(\hat q\) the \(\lceil (n+1)(1-\alpha) \rceil\)-th smallest non-conformity score on \(n\) held-out calibration rows, \(P(y_{\text{new}} \in [\hat q_{0.05}(x) - \hat q, \hat q_{0.95}(x) + \hat q]) \ge 1 - \alpha\) for any model, provided calibration and test scores are exchangeable. Scores on training rows are not exchangeable with scores on new rows — the model was shaped by them — so the correction computed on 753 training rows was 0.000 % and left test coverage at the raw band’s 0.773.
The exchangeability assumption: March 2020’s scores came from a distribution about five times wider than 2019’s. A rolling calibration restores coverage on average (0.893 over 2020–24) but only after the misses have occurred (April 0.67, May 0.90). What the coverage counter buys is a regime alarm with no regime model — 45 % of March days below the band and 36 % above, visible within days, faster than any quarterly review.
Double Machine Learning: Effects with Many Controls
§4.4 built a world in which a budgeting tool predicted repayment with a t-statistic of 24 and did nothing, because a hidden trait drove both adoption and repayment — and then showed that if the confounder is measured, putting it in the regression closes the backdoor and recovers the effect. That fix has a condition attached which §4.4 did not test: a control in a regression only does its job if its functional form is right. The question “what does this variable do, once many others are held fixed in whatever shape they enter?” is the question of policy evaluation (did the training programme raise earnings, or did the people who enrolled differ?), of marketing uplift (did the campaign cause the purchase, or did the targeting rule select buyers?), and of pricing experiments on observational data, where the price a customer saw was itself chosen by a model of that customer. This section breaks a linear control with a curved confounder, repairs it with two boosting models and a cross-fitting trick that takes twenty lines, and then asks the same question of a characteristic in the §4.5 panel: does 12-1 momentum move next month’s return once every other characteristic and the sector are held fixed, in whatever shape they enter?
Partialling out: what “controlling for \(X\)” really does
In the regression \(Y = \theta D + X\beta + \varepsilon\), with \(D\) the variable of interest (the “treatment” or the signal) and \(X\) a set of controls, the Frisch–Waugh–Lovell theorem says that the OLS estimate \(\hat\theta\) can be obtained in three steps: regress \(Y\) on \(X\) and keep the residual \(\tilde Y\); regress \(D\) on \(X\) and keep the residual \(\tilde D\); regress \(\tilde Y\) on \(\tilde D\). The slope of that last regression is exactly the multiple-regression coefficient on \(D\),
\[\tilde Y = Y - \mathbb E[Y \mid X], \qquad \tilde D = D - \mathbb E[D \mid X], \qquad \hat\theta = \frac{\sum_i \tilde D_i \tilde Y_i}{\sum_i \tilde D_i^2}.\]
“Controlling for \(X\)” therefore means: remove from \(Y\) and from \(D\) whatever \(X\) explains, and estimate \(\theta\) from what is left. The theorem is exact for OLS, and it exposes OLS’s hidden assumption. OLS estimates \(\mathbb E[Y \mid X]\) and \(\mathbb E[D \mid X]\) as linear functions of \(X\). If the confounder enters through a curve — \(\sin(2x)\), \(x^2\), an interaction — the linear fit removes only the part of the curve that a straight line can approximate, and the rest stays in both residuals. That leftover is correlated between \(\tilde D\) and \(\tilde Y\), because it came from the same confounder, and the residual-on-residual slope absorbs it. The backdoor path of §4.4 is only partly closed, and \(\hat\theta\) picks up the rest — with a small standard error, because nothing in the OLS output knows that the functional form was wrong.
Double machine learning (Chernozhukov, Chetverikov, Demirer, Duflo, Hansen, Newey and Robins, 2018) keeps the FWL recipe exactly and replaces the two linear regressions by any learner — here the §4.3 boosting machine, which can fit a sine curve with depth-2 trees. Two further ingredients, cross-fitting and an orthogonal moment, make the resulting estimate behave like an OLS coefficient even though the nuisance fits are machine-learning models with no standard errors of their own.
Break the linear control
The cell builds a world in which the answer is known. Three covariates; a confounder \(g(X) = 1.5\sin(2x_0) + 1.5\sin(2x_1)\) that drives both the signal \(D\) and the outcome \(Y\); a true effect \(\theta = 0.5\). The naive regression of \(Y\) on \(D\) ignores \(X\); the second regression controls for \(X\) linearly, the way §4.4 controlled for \(U\).
Naive 1.264; with linear controls 1.211 — still two and a half times the truth, with a standard error of 0.023 that says the wrong number is very precise. The diagnostic on the second line explains it: \(D\) correlates 0.73 with the confounder, and a linear fit of \(g\) on \(X\) has an R² of 0.139. The controls closed about a seventh of the backdoor and left the rest to \(\theta\). This is the §4.4 lesson in its harder form. In §4.4 the confounder was unmeasured and no regression could help; here it is measured — all three columns of \(X\) are on the right-hand side — and the regression still fails, because a sine curve is not a line.
The DML recipe: cross-fitting and Neyman orthogonality
The procedure has four steps.
- Split the rows into \(K\) folds.
- For each fold \(k\), fit \(\hat m(X) \approx \mathbb E[D \mid X]\) and \(\hat\ell(X) \approx \mathbb E[Y \mid X]\) on the other \(K - 1\) folds, and compute the residuals \(\tilde D_i = D_i - \hat m(X_i)\), \(\tilde Y_i = Y_i - \hat\ell(X_i)\) on fold \(k\) only.
- \(\hat\theta = \sum_i \tilde D_i \tilde Y_i \big/ \sum_i \tilde D_i^2\) over all rows.
- The standard error comes from the influence function \(\psi_i = \tilde D_i(\tilde Y_i - \hat\theta\tilde D_i)\): \(\widehat{se} = \sqrt{\sum_i \psi_i^2} \big/ \sum_i \tilde D_i^2\).
Step 3 is the moment condition \(\mathbb E[\tilde D(\tilde Y - \theta\tilde D)] = 0\), and the reason it can be trusted with machine-learning residuals is that it is Neyman-orthogonal: its derivative with respect to the nuisance functions \(m\) and \(\ell\), evaluated at the truth, is zero. To see what that buys, write the estimation error of \(\hat\theta\) as an expansion in the nuisance errors. The first-order terms — the ones proportional to \(\hat m - m\) alone or to \(\hat\ell - \ell\) alone — vanish by orthogonality, and what remains is proportional to the product \((\hat m - m)(\hat\ell - \ell)\). A boosting model that estimates \(\mathbb E[D \mid X]\) with an error of order \(n^{-1/4}\) — far too slow for the error itself to be negligible — combines with another of the same quality to give a product of order \(n^{-1/2}\), which is the rate of the OLS coefficient. Two learners that are each moderately good give an estimate that is very good; the naive alternative, regressing \(Y\) on \(D\) and a machine-learned \(\hat\ell(X)\), has a first-order term that does not cancel and inherits the slow rate of the learner.
Step 2 is cross-fitting, and it is there for a reason the next-but-one cell demonstrates. A flexible learner fitted on row \(i\) and then used to compute row \(i\)’s residual will have absorbed part of row \(i\)’s noise into its fit; the in-sample residual is too small and, worse, correlated with the fitted values. Fitting on the other folds means that the residual of row \(i\) never comes from a model that saw row \(i\) — the §4.3 train/test rule, applied not to the final score but to an intermediate quantity inside an estimator.
DML by hand in twenty lines
0.496 ± 0.023, on top of the truth, from two boosting models with depth-2 trees and no tuning. The nuisance models absorbed the sine curves that the linear controls missed. The last number on the line matters for reading every DML output: the residual of \(D\) kept 71 % of its raw spread. That is the part of the signal the confounder does not explain, and it is the only part that can identify \(\theta\) — if the controls explained all of \(D\), there would be no variation left to measure an effect from, and the standard error would be infinite. (KFold with shuffle=True is legal here because the simulated rows are independent; on the stock panel below the folds are still random across rows, and the standard error is corrected for the month structure afterwards.)
Does the split matter?
Cross-fitting introduces a random element — which rows land in which fold — and a method whose answer moved with the seed would be worthless. The cell re-runs with \(K = 2\) folds, so that each nuisance model sees only half the data, under two different seeds.
Will both \(K = 2\) estimates stay within two standard errors (0.046) of the \(K = 5\) answer of 0.496?
0.515 and 0.525 against 0.496: the split moves the estimate by about one standard error, never more. With \(K = 2\) each nuisance model is fitted on 1 000 rows instead of 1 600 and is a little rougher; the cost appears as a slightly higher estimate — the residual confounding of a cruder fit, in the direction of the naive bias — rather than as instability. Five folds is the usual compromise between the quality of the nuisance fits and the runtime; the paper’s recommendation is to repeat the procedure over several random splits and report the median.
Why not fit the nuisance models on the same rows?
Suppose you skip step 2, fit \(\hat m\) and \(\hat\ell\) on all \(n\) rows with a learner flexible enough to memorise, and compute \(\hat\theta\) on the same rows. More data for the nuisance models — will the estimate be unbiased, biased toward zero, or biased away from it? The cell uses unlimited-depth trees with a large learning rate and a minimum leaf of one.
Same rows: 0.377 — a quarter below the truth. The memorising learner left \(D\) with only 31 % of its spread (against 71 % under cross-fitting), because it fitted part of \(D\)’s noise as if it were signal; and what remains is contaminated by the fit that produced it, in a way that pulls the residual-on-residual slope toward zero. The bias is not in the learner — a memorising learner with cross-fitting would give a noisy but centred estimate — it is in letting one row be both teacher and witness. The cross-fitted recipe gave 0.496.
The real question: what does momentum do?
Now the §4.5 panel. \(D\) is the per-month z-score of 12-1 momentum; \(Y\) is next month’s return in per cent; \(X\) is the other four characteristics plus the sector dummies. The question is whether momentum has a partial association with next month’s return once the other characteristics — including the one it is most entangled with — are held fixed in whatever functional form the trees find.
25 335 rows, 14 controls (four characteristics and ten sector dummies), 119 months. Momentum is far from independent of the controls: it correlates 0.51 with price as a fraction of the 252-day high — a stock near its high has usually risen over the past year — and 0.10 with dollar volume. The 52-week-high effect (George and Hwang’s finding that nearness to the high predicts returns) is a documented anomaly in its own right, and any “momentum effect” a naive regression reports is partly that effect wearing a different label. This is exactly the §4.4 confounder, with fourteen of them at once and no reason to believe any of them enters linearly.
Naive, with controls, and DML
One more piece of Chapter 6 discipline enters here. The 213 stocks in one month are not 213 independent observations: they share the market’s move that month, so the errors are correlated within a month, and a standard error that treats the rows as independent overstates precision. The remedy is to cluster by month — sum the influence-function contributions within each month and treat the 119 monthly sums as the independent units. The cell reports all three estimators with clustered standard errors, and for DML the iid standard error as well, so that you can see what clustering costs.
Naive +0.156 % per standard deviation of momentum; with linear controls +0.201 %; DML +0.226 % a month, about 2.7 % a year. Controlling raised the estimate. The 52-week-high effect was masking momentum rather than manufacturing it: stocks near their high have both high momentum and — after the high is controlled for — a slightly lower expected return, so removing the high’s contribution from both \(D\) and \(Y\) leaves a larger partial effect. The controls explain 41 % of momentum’s variance, so 59 % of its spread remains to identify \(\theta\). Then look at the standard errors. Under the iid assumption DML’s standard error is 0.089 and the t-statistic would be 2.5; clustered by month it is 0.164 and t = 1.38. Twenty-five thousand rows are only 119 independent months, and a t-statistic of 1.4 is what 119 months can say about a monthly effect of a quarter of a per cent. The three estimators agree on the sign, disagree by less than half a standard error on the size, and none of them clears 2.
What DML does and does not buy you
| Naive OLS | OLS + controls | DML | |
|---|---|---|---|
| Closes the backdoor through measured \(X\) | no | only if linear | yes, any shape |
| Robust to a badly fitted nuisance model | — | — | yes, to first order |
| Valid standard errors on a panel | with clustering | with clustering | with clustering |
| Closes the backdoor through unmeasured confounders | no | no | no |
| Turns a characteristic into a treatment | no | no | no |
DML answers the question “what is the partial association of momentum with next-month return, holding fourteen controls fixed in whatever functional form they take?”. It does not say what would happen if you made a stock’s momentum higher — there is no such intervention, and a characteristic is not a treatment. And a variable you did not measure — analyst coverage, index membership, news flow — can still drive both momentum and next month’s return; no learner can control for a column that is not in the matrix. The §4.4 hierarchy stands unchanged: regression adjustment (now with flexible controls), then difference-in-differences, then an instrument. DML makes the first rung of that ladder honest about functional form; it does not add a rung.
The Frisch–Waugh–Lovell theorem says OLS controls by regressing out \(\mathbb E[Y \mid X]\) and \(\mathbb E[D \mid X]\) as linear functions; a \(\sin(2x)\) confounder is only 14 % linear, so most of it stayed in both residuals and leaked into \(\theta\). DML keeps the residual-on-residual recipe and estimates the two conditional expectations with boosting (0.496 ± 0.023). The moment is Neyman-orthogonal, so nuisance errors enter only as a product; cross-fitting keeps each row’s residual out of the model that saw it.
The clustered one. The 213 stocks in a month share that month’s market move, so their residuals are correlated and the effective sample is 119 months, not 25 335 rows. Summing the influence-function contributions within each month and treating the monthly sums as independent gives 0.164 and t = 1.38 — the same discipline as the Chapter 6 rule that time-ordered observations are not independent draws.
Covariance Shrinkage and Cluster-Based Allocation (Hierarchical Risk Parity)
Any procedure that inverts an estimated covariance matrix inherits the matrix’s estimation error, amplified: a generalised least-squares fit, a Mahalanobis distance, a discriminant classifier, the Kalman gain of §6.4, and any allocation of a budget across correlated risks. The problem is at its sharpest when the number of variables is not small relative to the number of observations — thousands of genes on hundreds of samples, hundreds of sensors on a few days of readings, dozens of business lines on a few years of results — and the demonstration here is 21 correlated daily return series on one year of data. §6.6 sized a single position by target over forecast volatility; allocating across twenty-one needs the same idea in matrix form, and the classical answer is the minimum-variance allocation: among all weights summing to one, the one with the smallest variance \(w^\top\Sigma w\) has the closed form \(w \propto \Sigma^{-1}\mathbf 1\). Everything fragile about it lives in the inverse. You will measure how ill-conditioned a 21-asset sample covariance is on one year of daily data, watch its optimal weights swing from one year to the next, shrink the matrix with Ledoit–Wolf, and then build a portfolio that never inverts anything — hierarchical risk parity — before putting all four rules through a nine-year walk-forward.
Twenty-one ETFs, one cash proxy removed
2 515 days from January 2015 to the end of 2024, 21 assets: three bond funds (IEF, TLT, LQD), high yield (HYG), the dollar (UUP), gold and silver, oil (USO), the S&P 500, Dow, Nasdaq-100 and Russell 2000, and nine sector funds. Annualised volatility runs from 6.7 % (IEF, 7–10-year Treasuries) to 40.1 % (USO). SHY, the 1–3-year Treasury fund, is excluded on purpose. Its volatility is 1.5 %, and a minimum-variance optimiser offered a 1.5 %-volatility asset will buy 120 % of it and short everything else — a correct answer to an uninteresting question. The 21 that remain are the risky universe, the one a portfolio construction rule has to do something intelligent with.
Why the sample covariance is ill-conditioned
The sample covariance \(S\) of \(N\) assets on \(T\) days is unbiased for \(\Sigma\), and that is where its virtues end. It has \(N(N+1)/2 = 231\) free parameters estimated from \(T = 251\) observations, and its eigenvalues — the variances of the portfolio’s principal directions — are systematically distorted: the largest are too large and the smallest too small, with the spread growing with the ratio \(N/T\). Even when the true covariance is the identity, so that every direction has variance one, the eigenvalues of \(S\) spread across roughly \([(1 - \sqrt{N/T})^2, (1 + \sqrt{N/T})^2]\) — for \(N/T = 21/251\) that is about \([0.50, 1.67]\), a threefold spread that is pure estimation noise. With real, correlated assets, whose true eigenvalues already span two orders of magnitude, the smallest sample eigenvalues are estimated from very few effective observations and come out near zero.
The condition number \(\kappa = \lambda_{\max}/\lambda_{\min}\) measures how much an inversion amplifies error: a relative perturbation of size \(\epsilon\) in \(S\) produces a relative perturbation of up to \(\kappa\epsilon\) in \(S^{-1}\). And the minimum-variance weights live precisely in the directions of the small eigenvalues, because \(w \propto \Sigma^{-1}\mathbf 1\) weights each direction by \(1/\lambda\): the direction with the smallest estimated variance gets the largest weight, and that direction is the one whose variance was estimated worst. The optimiser does not find the lowest-variance portfolio; it finds the portfolio whose variance the sample has most under-estimated. That is why the classic symptoms — enormous long and short positions in near-identical assets, weights that reverse when a year of data is added, realised risk above predicted — are not bugs but the exact consequence of inverting \(S\).
How ill-conditioned is one year of daily data?
Sample condition number 6 373 in 2018 and 5 010 in 2019. Ledoit–Wolf: 415 and 299 — about fifteen times better conditioned — from a shrinkage intensity \(\delta\) of only 0.029 and 0.034. A condition number in the thousands means that a 1 % error in the covariance can become a 60 % error in the inverse; the next cell shows what that does to the weights.
The weights that the inverse produces
Sample minimum-variance on 2018 data: 75 % IEF, −29 % TLT, +10 % SPY, 16 % dollar, 20 % high yield. The long IEF / short TLT pair is the classic signature — two Treasury funds that differ mainly in duration, one bought and the other sold, because the sample found a direction along the pair with a tiny estimated variance. A year later the same rule says −27 % TLT and −22 % SPY: the S&P 500 flipped from long to short on nothing but a new year of data. The total weight change from 2018 to 2019 is 0.99 for the sample optimiser and 0.52 for Ledoit–Wolf, whose SPY weight stays at 0.00 and −0.02, and whose gross leverage is 1.41 against the sample’s 1.97. Both under-predict next year’s volatility — the sample portfolio promised 1.54 % and delivered 2.00 %, LW promised 1.82 % and delivered 2.39 % — which is the fingerprint described above: the optimiser has found the noise in \(S\) and bet on it, and the bet does not pay out of sample. Shrinkage reduces the problem; it does not remove it, because 2019’s covariance was not 2018’s.
Ledoit–Wolf: pull the matrix toward a target
Ledoit and Wolf (2004, “A well-conditioned estimator for large-dimensional covariance matrices”) proposed replacing \(S\) by a convex combination of itself and a structured target,
\[\hat\Sigma_{LW} = (1 - \delta)\, S + \delta\, \mu I, \qquad \mu = \frac{\operatorname{tr}(S)}{N}, \qquad \delta^\ast = \arg\min_\delta \mathbb E\big\lVert \hat\Sigma_{LW} - \Sigma \big\rVert^2.\]
\(S\) is unbiased but noisy; the target \(\mu I\) — every asset at the average variance, zero correlation — is biased but has almost no estimation error, since it depends on the data through one number. The optimal \(\delta^\ast\) trades the two, and the paper’s contribution is a consistent estimator of \(\delta^\ast\) from the data alone: the bias–variance trade-off of §4.3, applied to a matrix. Shrinking toward \(\mu I\) moves every eigenvalue toward their average, so it raises the small eigenvalues and lowers the large ones. The condition number falls from thousands to hundreds while the total variance is untouched; \(\delta = 0.03\) sounds negligible, but it is applied where the matrix is weakest — a small eigenvalue of 0.001 becomes \(0.97 \times 0.001 + 0.03\mu\), which is dominated by the second term. sklearn.covariance.LedoitWolf computes \(\delta^\ast\) and returns both the matrix and the intensity.
The same idea with other targets: shrinkage toward a constant-correlation matrix (Ledoit and Wolf, 2004b, “Honey, I shrunk the sample covariance matrix”), toward a one-factor CAPM covariance, and non-linear shrinkage (Ledoit and Wolf, 2012 onward) that moves each eigenvalue by its own amount rather than all by the same \(\delta\). They exist because \(S^{-1}\) is the problem, not \(S\): for any purpose that does not invert — reporting a correlation, computing the variance of a given portfolio — the sample covariance is fine.
HRP: correlation becomes distance, distance becomes a tree
López de Prado (2016, “Building Diversified Portfolios that Outperform Out of Sample”) drew a different conclusion from the same diagnosis. If the instability lives in \(\Sigma^{-1}\), never compute it. Hierarchical risk parity replaces the inversion with three steps that use only the correlation structure and the individual variances: cluster the assets, order them so that neighbours are alike, and split capital down the tree two clusters at a time, so that estimation error stays local.
The first step needs a distance. Correlation is not one — \(\rho = 1\) means “identical”, which should be distance zero — but \(d_{ij} = \sqrt{\tfrac12(1 - \rho_{ij})}\) is: it is zero for perfectly correlated assets, one for perfectly anti-correlated ones, and it satisfies the triangle inequality, so an agglomerative clustering algorithm can use it. The cell builds a single-linkage tree on that distance (each merge joins the two clusters whose closest members are closest) and reads the dendrogram’s leaves from left to right, which is the quasi-diagonal order: similar assets adjacent, so that the correlation matrix, re-arranged in that order, has its large entries near the diagonal.
Two assets with \(\rho = 0.5\) — what is their distance \(d\), to two decimals? And two with \(\rho = -1\)?
\(d = 0.50\) at \(\rho = 0.5\) and \(1.00\) at \(\rho = -1\). The tree puts the dollar and the two metals first, then the three bond funds, then the defensive sectors (utilities, staples), energy with oil, credit and financials, and finally the equity indices with their cyclical sector funds — a grouping an experienced allocator would have drawn by hand, produced from nothing but 251 days of correlations. Adjacent assets in tree order correlate 0.63 on average against 0.22 in the file’s alphabetical order: the correlation matrix is now nearly block-diagonal.
See the blocks
The right-hand panel shows a red equity block in the bottom-right corner, a smaller bond block, and the blue stripe where bonds meet equities — the negative stock–bond correlation of the 2018 sample, which is what a diversified portfolio is made of. Recursive bisection will walk down this order and, at every split, give more capital to the half with the lower variance.
Recursive bisection
Start with all assets in tree order and weight 1. Split the list in half; compute each half’s variance as the variance of its inverse-variance portfolio (weights proportional to \(1/\sigma^2_i\) within the half, a rule that needs no inverse of anything but a diagonal); give the left half the share \(\alpha = 1 - v_L/(v_L + v_R)\), so that the lower-variance half gets more; multiply the weights in each half by its share; and recurse into each half until every cluster is a single asset.
The weights sum to 1.000, every one is positive, and the largest is 0.308 — UUP, the dollar, which has the lowest volatility in its cluster of dollar-and-metals and therefore wins every split it is part of — followed by LQD and IEF at 0.17, gold at 0.12, high yield at 0.08. No short position, no inverse, no eigenvalue. Twenty lines. The property López de Prado emphasises is that estimation error stays local: a noisy correlation between two energy funds changes how capital is split between them, and nothing else — the weight of Treasuries is decided at a split that never sees that correlation. In the minimum-variance optimiser every entry of \(S\) affects every weight through the inverse; in HRP an entry affects only the splits below the node where the two assets meet.
Nine years walk-forward: which rule wins what?
Each January, estimate on the previous calendar year and hold for the year — the walk-forward of §6.2 with yearly refits. Four rules: equal weight, minimum-variance on the sample covariance, minimum-variance on the Ledoit–Wolf covariance, and HRP. Before running the cell, decide which rule will realise the lowest volatility and which will need the most trading each January. The cell prints the table and the cumulative return of each rule.
Equal weight: volatility 12.6 %, return 8.5 %, Sharpe 0.67, drawdown −34.5 %, no trading. Minimum-variance on the sample matrix: volatility 3.2 % — the lowest, as its objective demands — and Sharpe 0.82, but it re-trades 148 % of the book every January and holds a 102 % position in one asset. Ledoit–Wolf: volatility 3.6 %, Sharpe 1.01, drawdown −7.6 %, turnover 0.70 — the best Sharpe with half the trading and a largest position of 0.47. HRP: volatility 5.4 %, Sharpe 0.84, drawdown −13.9 %, turnover 0.56, long-only, largest weight 0.51. In the Covid year the three risk-based rules earned between 0.4 % and 3.4 % against equal weight’s 6.6 %, at a fraction of its drawdown.
Two readings, one certain and one suggestive. The turnover ordering is certain: it is a property of the estimators, not of the returns, and the sample optimiser’s 148 % a year is the cost of betting on the noise in \(S\) — before transaction costs, which this table does not charge and which would fall hardest on that rule. The Sharpe ordering is suggestive only. Nine years of daily returns give an annualised Sharpe a standard error of about \(1/\sqrt 9 \approx 0.33\); LW’s 1.01 against HRP’s 0.84 and the sample’s 0.82 is half a standard error. What the table does establish is that shrinkage buys the same risk reduction as the sample optimiser with half the trading, and HRP buys most of it with no leverage and no short — which is the property that makes HRP the choice for a mandate that cannot short and shrinkage the choice for one that can.
The condition number is \(\lambda_{\max}/\lambda_{\min}\), the factor by which inversion amplifies estimation error, and the min-variance weights load on the small-eigenvalue directions — the ones estimated worst. Shrinking toward \(\mu I\) moves every eigenvalue toward the average: a 3 % pull is negligible for the large ones and dominant for a near-zero one, so the ratio falls fifteen-fold while the total variance is unchanged. The 2018 → 2019 weight change fell from 0.99 to 0.52.
HRP clusters on \(d = \sqrt{(1-\rho)/2}\), orders the assets quasi-diagonally, then splits capital top-down with each half’s share set by the variance of its inverse-variance portfolio — only diagonals and sub-matrices, never \(\Sigma^{-1}\). UUP wins every split it is part of because it has the lowest volatility in the dollar-and-metals cluster; a noisy correlation elsewhere in the tree cannot touch that weight, which is what “estimation error stays local” means.
Event-Based Labels and Meta-Labeling: Triple Barriers and Two-Stage Decisions
Decision systems with two stages are everywhere: a fraud filter flags a transaction and a second model decides whether to block it; a screening test refers a patient and a second model decides on treatment; a lead-scoring rule selects prospects and a second model decides how much effort each deserves. In every one of them the first stage chooses a direction and the second stage chooses an amount, and the label that trains the second stage must describe an event with a path — the alert was closed within a day, the patient was escalated before the follow-up, the customer churned before the offer — rather than an end point. Every classifier in Chapter 4 needed a label, and every label in this book so far has been of one kind: “the return over the next \(h\) days was positive”. A trader who set a stop-loss would not recognise that label. A position that fell 6 % on day 3 and recovered to +1 % on day 10 is a success by the fixed-horizon label and a stopped-out loss in the trader’s book. This section builds López de Prado’s triple-barrier label on the S&P 500, with barriers scaled by a §6.5-style volatility estimate so that they widen in stormy markets; gives a moving-average rule the job of choosing the side of each trade; and trains a second model — the meta-model — to decide the size, with an embargo in the walk-forward that the label’s look-ahead makes necessary.
What is wrong with “was the 10-day return positive”?
The fixed-horizon label \(\mathbb 1[r_{t \to t+h} > 0]\) looks only at the end point. It ignores the path — and the path is what a risk manager acts on. Two trades with the same end point but different paths are different trades: one was never in trouble, the other was closed at a loss before the recovery it never saw. A model trained on end points learns to predict something no one holding the position with a stop would have experienced, and the mismatch between what the label rewards and what the desk rewards is a source of “signals” that do not survive contact with a risk limit.
The triple-barrier label (López de Prado, 2018, Advances in Financial Machine Learning) describes the trade instead. From entry at the close of day \(t\), three barriers are set: an upper profit-take at \(+m_{pt}\,\hat\sigma_t\), a lower stop-loss at \(-m_{sl}\,\hat\sigma_t\), and a vertical barrier at \(t + h\) — a maximum holding period. The label is +1 if the upper barrier is touched first, −1 if the lower barrier is touched first, and the sign of the return at \(t + h\) if neither horizontal barrier is touched before the vertical one. The multipliers \(m_{pt}\), \(m_{sl}\) and the horizon \(h\) are the trader’s risk parameters, and \(\hat\sigma_t\) is the volatility estimate at \(t\) — here an exponentially weighted standard deviation with a 21-day span, in the spirit of §6.5, known at the close of \(t\) — so that a 2 % stop in a calm month becomes a 5 % stop in March 2020. A barrier set in units of volatility is the same barrier in every regime; a barrier set in per cent is hit constantly in one regime and never in another.
Compute the labels, vectorised
The natural implementation is a loop over entry days and a loop over the following \(h\) days. The vectorised one builds a \(T \times h\) matrix of cumulative log returns from each entry — column \(j\) is the return from \(t\) to \(t + j\) — compares every column against the barriers at once, and takes the first column in which each barrier is crossed.
With barriers at ±2 \(\hat\sigma_t\) and a ten-day horizon, 59.8 % of entries are labelled +1 against 64.1 % for the fixed-horizon label. 76.8 % of entries touch a horizontal barrier within ten days, on average after six; only a quarter of trades run to the vertical barrier. The two labels agree on 88.5 % of days and disagree on 11.5 % — those are the trades where the path and the end point tell different stories, and they are exactly the trades a stop-loss decides. A classifier trained on the fixed-horizon label is being taught the wrong answer on one day in nine. np.where(cond, days, H + 1).min(axis=1) is the idiom worth remembering: it returns the first day on which the condition holds, or \(H + 1\) if it never does, without a loop.
One entry, three barriers
Entering on 24 February 2020 with \(\hat\sigma_t\) = 1.22 %, the barriers sit at ±2.44 %. The stop is hit on day 1 (the upper barrier is never hit — hit_up is 11, the “never” code), so the label is −1. The fixed-horizon label agrees here, but the barrier label closed the trade nine days earlier, before the path fell a further 10 %. The vertical barrier caps how long a bet is allowed to be wrong; the horizontal ones cap how wrong. A label that knows when the trade ended is also a label whose cost of being wrong is bounded — which is what makes the bet-sizing model below meaningful.
The primary signal chooses the side
Meta-labelling (López de Prado, 2018) separates two decisions that a single classifier conflates. The primary model decides the side — long or short — and can be anything: a rule, a fundamental view, a model from another desk. The meta-model is a binary classifier trained on the question “did the primary’s trade pay, in the triple-barrier sense?”, and its output is used to decide the size of the bet, including zero. The division of labour lets a crude but interpretable side model be paired with a statistical size model, and it converts a three-class problem (long, flat, short) into a two-class one in which the classes are far better balanced.
The primary here is a 50/200-day moving-average crossover — one of the 42 rules Section 7.6 puts on trial — long when the fast average is above the slow one and short otherwise. The meta-label is 1 when the side the primary chose has the same sign as the triple-barrier label. Seven features are built for the meta-model, all known at the close of \(t\): the side itself, the volatility estimate, 21- and 63-day momentum, the ratio of 5- to 21-day volatility, the distance of price from its 200-day average, and a 14-day RSI.
2 307 days from October 2015. The primary is long 79 % of the time, and its trade “works” — hits its profit barrier, or ends on the right side of the vertical one — on 55.7 % of days: 60.4 % when it is long, 38.1 % when it is short. That asymmetry is the first thing a meta-model can learn. In a decade in which the index rose, a trend rule’s long trades are right more often than a coin and its short trades are wrong more often than one; a model with side among its features can learn to back the longs and decline the shorts, and anything beyond that — that the primary is only worth backing in some volatility or momentum states — is what the remaining six features are for.
Walk-forward with an embargo
The walk-forward refits each January on all earlier rows, as in §6.2 — with one change forced by the label. The label at day \(t\) is not known until day \(t + 10\), because it depends on the path to the vertical barrier. A model refitted on 1 January 2019 on all rows dated before it would use the labels of the last ten trading days of December 2018, and those labels depend on returns from the first ten trading days of January 2019 — the model’s own test period. The features leak nothing (every one uses data to the close of \(t\)); the labels leak. The fix is to drop the last \(H\) training rows before each refit. That gap is the embargo, and it is needed whenever a label looks ahead: a fixed-horizon label of \(h\) days needs the same \(h\)-day gap, and §4.5 needed none only because a one-month-ahead target at month-end \(t\) is realised at exactly the instant the next month’s features are observed.
1 500 test days, AUC 0.545. The meta-model separates good from bad primary trades a little better than a coin (0.5), which is what seven price features can be expected to deliver after §6.2, and it says “back the trade” (p > 0.5) on 71.3 % of days. On this data the embargo changes the AUC only in the third decimal — the last ten labels of a year are a small fraction of 800 training rows — but its size is not the point. The last training label without the embargo would depend on prices up to 15 January 2019, inside the test year, and a walk-forward that has seen any of its test period has lost the property that made it a walk-forward. The audit question to ask of any label is the one §6.2 asked of any feature: what is the date of the last observation it uses?
From probability to bet size
The meta-model’s probability becomes a position through a bet size \(b_t = \max\big(0,\, 2(p_t - \tfrac12)\big)\): nothing when the model is at or below a coin, a full position at \(p = 1\), and a linear ramp between. The position is \(\text{side}_t \times b_t\), applied to the next day’s return. The cell compares four books: the primary alone, the primary sized by the meta-model, an on/off version that takes the full position whenever \(p > 0.5\), and buy-and-hold.
The primary alone: cumulative 24.6 %, Sharpe 0.20, maximum drawdown −46.1 %. The drawdown is the 2020 whipsaw: the crossover went short in April 2020 after the crash and stayed short through the recovery, and its 2020 Sharpe was −0.68. Meta-labelled and sized: Sharpe 0.77, drawdown −3.9 % — but a cumulative return of only 13.9 %, because the mean bet is 0.16; a model whose probabilities hover just above 0.5 sizes every bet small. The on/off version takes the full position whenever the model leans the primary’s way and earns 63.5 % at Sharpe 0.74 with a −23.8 % drawdown — level with buy-and-hold’s 0.73 at half its drawdown. The year-by-year Sharpe ratios show where the improvement came from: 2020 went from −0.68 to +0.79, because the meta-model declined the shorts; 2022 went from −0.22 to −1.11, because it declined some longs that would have worked. Meta-labelling improved the quality of a signal that was already there — it removed the primary’s worst year — and it did so with a classifier whose AUC is 0.545. It cannot create a signal. A primary rule with no edge would produce a meta-model with nothing to back, and the correct output of a meta-model in that case is a bet size of zero.
A boosting meta-model
Does a more flexible meta-model do better? The cell replaces the logistic regression by HistGradientBoostingClassifier with the §4.3 settings, and reports the same statistics.
Boosting: AUC 0.528, Sharpe 0.14, drawdown −11.3 % — worse than the logistic model’s 0.545 and 0.77. Seven features and about 800 training days per class are not enough for trees to find a stable state dependence; what they find in one year’s training set is not there in the next. The more important reading is the second one: the meta-model’s Sharpe swings by 0.6 with the choice of learner. That swing is the honest error bar on the 0.77 of the previous cell — a Sharpe of 0.77 obtained with one of two reasonable learners, the other giving 0.14, is a number with a standard error at least as large as the difference. Section 7.6 is about counting such choices.
Fixed-horizon: +1, because the end point is positive. Triple-barrier with a stop at −2\(\hat\sigma_t\): −1, because the lower barrier was touched first on day 3 and the trade was closed there. The barrier label describes the path a trader with a stop actually lived through; on the S&P 500 the two labels disagree on 11.5 % of days, and 76.8 % of entries touch a horizontal barrier within ten days.
The labels. A triple-barrier label at day \(t\) depends on the path to \(t + 10\), so the labels of the last ten training days of December depend on returns inside the January test period. Drop the last \(H = 10\) training rows before every refit — the embargo. Features leak nothing (all use data to the close of \(t\)); it is the label’s look-ahead that must be embargoed, by exactly its horizon.
Selection Statistics: Deflated Performance, Overfitting Probability, and 42 Rules on Trial
Every performance number in this course was the result of a search: over features, targets, thresholds, learners, values of \(K\). The search is the part the number never reports, and the problem is the central problem of large-scale A/B experimentation (a thousand variants tested, one winner announced), of model selection (the best cell of a hyper-parameter grid), and of genome-wide or feature-wide screening, where tens of thousands of hypotheses are tested and the smallest p-value is guaranteed to look impressive. The Sharpe ratio is the demonstration statistic, because it is the one whose selection bias has been studied most carefully; every method below transfers to any other score. Section 7.1 tried five losses and three values of \(K\); Section 7.5 tried two learners; §6.3 tuned an entry threshold and found 1.09. This section makes the search explicit. You will backtest 42 technical rules on the S&P 500 — the rules a chart-reader might try in an afternoon — find the best, and put it on trial four ways: a bootstrap confidence interval that respects the autocorrelation of daily P&L; the Deflated Sharpe Ratio, which measures the best against the best that noise would produce; the probability of backtest overfitting, which asks whether the in-sample winner carries anything out of sample; and the Benjamini–Hochberg false-discovery correction across the whole grid, the §3.2 hypothesis test made honest about multiplicity. Then the Dow, with four times the sample, gets to disagree.
The grid: 42 rules a chart-reader might try
Three families. Seventeen moving-average crossovers (fast 5 to 100 days, slow 50 to 250; long when fast is above slow, short otherwise — always in the market). Nine RSI mean-reversion rules (three windows, three pairs of thresholds; long when oversold, short when overbought, flat between). Eight Bollinger breakouts (long above \(k\) standard deviations from a 20- or 50-day mean, short below) and their eight mirror images, the Bollinger mean-reversion rules. That gives 42 daily P&L series over 2 266 days from December 2015, each shifted so that a position decided at the close of \(t - 1\) earns \(r_t\). The mirrors are deliberate. A chart-reader who “tries both directions” has doubled the number of trials, and the statistics below must count them — while remembering that a rule and its mirror are not two independent trials.
The winner is Bollinger mean reversion, 20 days, \(k = 2\), with Sharpe 0.68 — from a rule that is in the market 9.1 % of the time and whose P&L has skew 7.1 and kurtosis 154: a few large days. The median rule earns 0.15, the standard deviation across rules is 0.26, buy-and-hold earned 0.64 over the same days, and 71 % of the rules are positive because the index rose and most of the crossovers are long most of the time. The histogram is the object to look at. It is a distribution of 42 Sharpe ratios with a spread of 0.26, and the red line is its maximum. A grid this size always has a winner. The question is whether 0.68 is more than the winner of 42 coin flips would show — and that is a question with a formula.
Is the best of \(N\) evidence? The expected maximum
Suppose every one of \(N\) rules has a true Sharpe of zero and the estimated Sharpes are independent draws with standard deviation \(\sigma_{SR}\). The expected value of their maximum is not zero; it grows with \(N\). Bailey and López de Prado (2014, “The Deflated Sharpe Ratio”) give the approximation
\[\mathbb E[\max_N SR] \approx \sigma_{SR}\Big[(1-\gamma)\,\Phi^{-1}\!\big(1 - \tfrac1N\big) + \gamma\,\Phi^{-1}\!\big(1 - \tfrac{1}{Ne}\big)\Big], \qquad \gamma = 0.5772,\]
with \(\Phi^{-1}\) the normal quantile function and \(\gamma\) the Euler–Mascheroni constant — the standard extreme-value result for the maximum of \(N\) normals, the same limit theory as the block maxima of §3.5. With \(N = 42\) and \(\sigma_{SR} = 0.26\) (annualised), the bracket is about 2.2 and the expected best is about 0.57. Before any skill, a search over this grid was going to produce a rule with a Sharpe near 0.6. A colleague who reports “the best of my 40 rules has Sharpe 1.1 over ten years” is reporting a number that must be measured against 0.6, not 0, and whether it clears that bar depends on \(N\), on the spread of the trial Sharpes, on the sample length and on the shape of the return distribution. The Deflated Sharpe Ratio combines exactly those.
The debug that produces a Sharpe of 5
Before the four trials, one cell on the most expensive bug in this business. rules_pnl shifts the position by one day. A colleague’s version omits the shift.
The colleague’s best rule is the 20-day Bollinger breakout at \(k = 1.5\) with a Sharpe of 5.34, and the chapter’s winner, the mean-reversion mirror, now scores −3.12. Every number has changed sign and quadrupled. The bug is the missing .shift(1): the position on day \(t\) is computed from the close of day \(t\) — the Bollinger z-score uses \(P_t\) — and then multiplied by \(r_t\), the return that produced \(P_t\). A breakout rule “goes long” on a day whose close was already high, and collects the return that made it high. This is the §6.2 leak in its most flattering form: the position contains the target. A Sharpe of 5 from a daily rule on an index is not a discovery; it is a diagnostic that the P&L was computed with the position and the return on the same day. The fix is one method call, and the audit question is the one from §6.2 and §7.5: what is the date of the last observation the position uses?
Trial 1 — a confidence interval from the stationary bootstrap
§3.2 introduced resampling with replacement as the universal solvent for a statistic with no closed-form standard error, and named the Sharpe ratio as the example. Daily P&L cannot be resampled one day at a time, because Chapter 6 showed that its size clusters — an iid bootstrap would destroy the volatility clustering and understate the interval. The stationary bootstrap of Politis and Romano (1994) resamples in blocks of random length — geometric, with mean 20 days here — starting at random positions and wrapping around the end of the sample, so that autocorrelation and volatility clusters within a block survive while the resampled series remains stationary. The Sharpe is recomputed on each of 300 pseudo-samples.
95 % interval [0.18, 1.11], bootstrap standard error 0.26 (the analytic iid formula \(\sqrt{(1 + SR^2/2)/T}\) gives 0.33), and only 1.7 % of resamples at or below zero. Taken alone, the interval says “significant at 5 %”. But it answers the wrong question. It asks: given that we picked this rule, how uncertain is its Sharpe? It conditions on the pick, and it knows nothing about the 41 rules we did not pick. A confidence interval is a statement about sampling variation in one pre-specified statistic; the statistic here was chosen because it was the largest, and the interval cannot see that.
Trial 2 — the Deflated Sharpe Ratio
Bailey and López de Prado’s Deflated Sharpe Ratio proceeds in two steps. First, compute the Sharpe that a selection would produce by luck alone, \(SR_0\): the expected-maximum formula above, with \(\sigma_{SR}\) estimated as the standard deviation of the \(N\) trial Sharpes. Second, compute the probability that the observed Sharpe exceeds \(SR_0\), allowing for the sampling distribution of a Sharpe ratio under non-normal returns (the Probabilistic Sharpe Ratio of Bailey and López de Prado, 2012):
\[DSR = \Phi\!\left[\frac{(\widehat{SR} - SR_0)\sqrt{T-1}}{\sqrt{1 - \gamma_3 \widehat{SR} + \frac{\gamma_4 - 1}{4}\widehat{SR}^2}}\right],\]
with \(\widehat{SR}\) the per-period (daily, not annualised) Sharpe, \(T\) the number of periods, \(\gamma_3\) the skewness and \(\gamma_4\) the kurtosis of the P&L. The denominator is the standard error of a Sharpe ratio for a return series with those moments: for normal returns (\(\gamma_3 = 0\), \(\gamma_4 = 3\)) it reduces to \(\sqrt{1 + \widehat{SR}^2/2}\), the formula in the previous cell, and fat tails (\(\gamma_4 \gg 3\)) or negative skew widen it, so that the same Sharpe is worth less when it comes from a few big days. A DSR near 1 says the Sharpe survives the number of trials; near 0.5 says the best rule is what noise would produce. Read it as \(1 - p\) for the one-sided test of \(SR > SR_0\). One caution about \(N\): it should be the number of effectively independent trials, and the mirrors and near-duplicate parameter settings in this grid make 42 an over-count — so the DSR below is, if anything, too harsh, and the verdict it delivers survives that.
Against a benchmark of zero the rule looks real: PSR 0.990, in agreement with the bootstrap. Against the benchmark the search deserves — an expected best-of-42 of 0.57 — the Deflated Sharpe Ratio is 0.637: a 36 % chance that a Sharpe this large arises from a grid of 42 worthless rules, before the fat tails are even the main point. The gap between 0.990 and 0.637 is the search. Ten years of the S&P 500 do not distinguish this rule from luck.
Trial 3 — the probability of backtest overfitting (CSCV)
The DSR asks whether the best Sharpe is larger than the best noise would give. Bailey, Borwein, López de Prado and Zhu (2017, “The Probability of Backtest Overfitting”) ask a different question: does the rule that wins in-sample carry anything out of sample? Their combinatorially symmetric cross-validation works on the \(T \times N\) matrix of P&L series. Split the rows into \(S\) blocks of consecutive days. For every way of choosing \(S/2\) of the blocks as in-sample (the remaining \(S/2\) are out-of-sample):
- Pick the rule with the best in-sample Sharpe, \(j^\ast\).
- Find its rank among the \(N\) out-of-sample Sharpe ratios, as a fraction \(\bar\omega \in (0, 1)\), and take the logit \(\lambda = \log\frac{\bar\omega}{1 - \bar\omega}\) — positive when the in-sample winner is above the out-of-sample median, negative when it is below.
- The probability of backtest overfitting is \(PBO = P(\lambda \le 0)\) across the combinations.
\(S = 8\) gives \(\binom{8}{4} = 70\) combinations; the cell draws 45 with a fixed seed. Because the blocks keep the days in order, every combination is a legitimate “train on these periods, test on those” — the walk-forward of §6.2, repeated 45 times over, and symmetric in time: the in-sample and out-of-sample halves are the same size and each block appears in both roles. \(PBO = 0.5\) means the in-sample winner is a coin flip out of sample; a real edge would push it toward 0, because a rule that is genuinely best would be near the top out of sample in most splits.
PBO = 0.533: in 24 of 45 splits the rule that won in-sample was below the median out of sample, the median logit is −0.33, and the in-sample winner finished first out of sample in only 2 splits. This is the signature of a grid with no edge — the best in-sample rule is a different rule each time, and whichever it is carries nothing across the boundary. The argsort(argsort(·)) idiom computes ranks; dividing by \(N + 1\) keeps \(\bar\omega\) strictly inside \((0, 1)\) so that the logit is finite.
Trial 4 — Benjamini–Hochberg across the grid
§3.2 gave each hypothesis its own p-value and warned that running twenty tests on noise should be expected to produce one “significant” result. With 42 tests the tool is a false discovery rate procedure. Benjamini and Hochberg (1995): sort the p-values \(p_{(1)} \le \dots \le p_{(N)}\), find the largest \(k\) such that \(p_{(k)} \le q\,k/N\), and reject the \(k\) smallest. Under independence (or positive dependence) among the tests, the expected fraction of false discoveries among the rejections is at most \(q\). The threshold for the smallest p-value is \(q/N\) — the Bonferroni bar — and it rises linearly for the next ones, which is what makes BH less conservative than Bonferroni when several hypotheses are genuinely false. The p-value of each rule is the two-sided normal p-value of \(t = SR\sqrt T\) against a true Sharpe of zero.
Two rules clear 0.05 on their own p-value — the 20-day Bollinger breakout at \(k = 2\) and its mirror, which share one p-value of 0.0427 because they are the same bet with the sign flipped, and a two-sided test cannot tell them apart. Benjamini–Hochberg needs the best of 42 below \(0.05/42 = 0.0012\); it is 36 times too large. The third-smallest p-value is 0.18. Zero rules survive. The false-discovery correction agrees with the DSR and the PBO, for a reason worth stating: all three are ways of charging the winner for the search, and a rule that cannot pay the charge on one bill cannot pay it on the others.
Does the Dow agree?
Every statistic so far was computed on ten years of one index. dji.csv holds the Dow from 1985 to 2020 — 8 774 days, nearly four times the sample — and the same 42 rules can be run on it unchanged. Three questions: how does the S&P’s winner do on the Dow; does the Dow’s own best rule pass the four trials; and does the rule that won the first half of the Dow’s history keep winning in the second half?
On the Dow the S&P’s winner earns Sharpe −0.00 and ranks 32nd of 42. The Dow’s own best rule — RSI-21 with 30/70 thresholds — has Sharpe 0.23, DSR 0.583, PBO 0.778 and no BH survivor; and the rule that won 1985–2002, a 50/250 crossover with Sharpe 0.25, earned 0.02 afterwards. Thirty-five years, two indices, four statistics: no technical rule in this grid survives the search that found it. This is the honest state of the evidence on simple technical rules on liquid index prices, and it is consistent with the published record — the momentum effects that do survive live in the cross-section of stocks (Section 7.1’s faint reversal and momentum signs) and in longer-horizon time-series momentum across many assets, not in a chart pattern on one index.
Add a rule and re-deflate
One more trial: 63-day time-series momentum, long or short on the sign of the last 63 days’ return. The cell adds it as the 43rd column and recomputes the best rule and its DSR.
The momentum rule earns 0.18 and does not displace the winner; the DSR moves from 0.637 to 0.642 — one more mediocre trial barely changes the benchmark, because \(SR_0\) depends on the number of trials and their spread, and one more rule near the median moves neither. The lesson runs the other way too: 42 was already enough to explain a 0.68. Had the grid held four rules, \(SR_0\) would have been about 0.27 and the same 0.68 would have had a DSR near 0.9. The rule did not change; the count of what was tried did, and the count is what the Sharpe ratio never reports.
Quantopian hosted hundreds of thousands of amateur and professional quants who backtested strategies on a free platform and could be allocated capital if their algorithms looked good. In 2016 its own research team (Wiecki, Campbell, Lent and Stauth, “All That Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms”) took 888 algorithms that users had deployed to paper or live trading and compared each one’s backtest with what it did afterwards, out of sample. The in-sample Sharpe ratio had essentially no power to predict the out-of-sample Sharpe ratio; the ranking of strategies by backtest was close to useless. The one variable that did predict out-of-sample failure was the amount of backtesting the author had done — the more times a user had re-run and tuned a strategy, the worse it performed live. Quantopian’s community platform closed in November 2020, four years after its own paper had measured the problem.
Every re-run is a trial. A backtest reported without the number of trials behind it — and without the DSR, PBO and false-discovery rate that number implies — is a description of the search, not of the strategy.
They answer different questions. The bootstrap conditions on the pick — given this rule, how uncertain is its Sharpe? — and cannot see the 41 rules not picked. The DSR measures the observed 0.68 against the expected best of 42 noise trials, \(SR_0 \approx 0.57\), with a standard error widened by skew 7 and kurtosis 154; the result is a 36 % chance of a Sharpe this large from a worthless grid. PBO 0.533 and zero BH survivors agree with the DSR, and on the Dow the rule ranks 32nd of 42.
Sort the \(N\) p-values; find the largest \(k\) with \(p_{(k)} \le q\,k/N\); reject the \(k\) smallest — this bounds the expected false-discovery fraction at \(q\). With \(N = 42\) and \(q = 0.05\) the smallest p-value must be below 0.0012; 0.043 is 36 times too large, and the third-smallest (0.18) is nowhere near its bar of 0.0036. The two 0.043s are also one bet with the sign flipped, so they are not even two trials.
Chapter Wrap-up
You can now do six things that anyone who turns predictions into decisions has to do after the prediction has turned out to be weak, and you can say with a number how far each of them can be trusted.
Rank. Given any set of items scored for a shortlist, you know that a top-\(K\) selection uses only the order of the scores — a monotone transform left the top-20, bottom-20 and spread untouched while multiplying the MSE 110-fold — and you can train for the order three ways (a within-group rank target, a pairwise logistic loss on pairs of items, the §4.5 trees on ranks) inside the §4.5 walk-forward. You score with the rank IC and read it against its standard error: all five models were within 0.02 of zero, and the rank-based ones agreed with each other (IC correlations 0.81–0.92) about an order that 71 periods cannot confirm. Bound. You can turn the §6.2 boosting model into a band with loss="quantile", whose width tracks the recent scale of the series (correlation 0.82 with std21) while its median stays useless; calibrate it with one held-out quantile of non-conformity scores (coverage 0.773 → 0.830) under a guarantee whose one assumption is exchangeability; and read the collapse of coverage to 0.18 in March 2020 as an alarm that the distribution has moved, with no regime model. The band’s width sized the demonstration series to a Sharpe of 0.66 and a drawdown of −21 % against 0.56 and −41 %. Isolate. You know that “controlling for \(X\)” is residual-on-residual regression, that linear controls closed a seventh of a \(\sin(2x)\) backdoor (1.211 for a truth of 0.5), that two boosting nuisance models with cross-fitting recover it (0.496 ± 0.023), and that skipping the cross-fit drops the estimate to 0.377. On the panel, one characteristic’s partial effect is +0.23 % a month with t = 1.4 once the standard error is clustered by period — and DML fixes functional form, not omitted confounders. Allocate. You can measure a condition number (6 373 on 21 series), shrink it fifteen-fold with \(\delta = 0.03\), build a cluster-based allocation in twenty lines with no inverse, and walk all four rules forward: Sharpe 0.67 / 0.82 / 1.01 / 0.84 for equal, sample, Ledoit–Wolf and HRP with turnover 0 / 1.48 / 0.70 / 0.56 — the turnover ordering certain, the Sharpe ordering within a standard error of 0.33. Label and size. You can build event-based labels with barriers scaled by the recent volatility (76.8 % of entries touch a barrier within ten days; 11.5 % of labels differ from fixed-horizon ones), embargo the walk-forward by the label’s horizon, and train a second-stage model that turned a first-stage rule’s Sharpe of 0.20 and worst loss of −46 % into 0.77 and −3.9 % by declining the trades of its worst year — while a second learner gave 0.14, which is the honest error bar. Judge. You can put any selected result on trial four ways, and you know what each trial sees: the bootstrap interval [0.18, 1.11] conditions on the pick; the deflated performance score 0.637 charges for 42 trials; the overfitting probability 0.533 asks whether the in-sample winner carries anything across a boundary; Benjamini–Hochberg at 0 of 42 controls the false-discovery rate across the grid. A second series agreed: rank 32nd of 42, PBO 0.78, no survivor.
None of the six methods is specific to the data it was demonstrated on. The pairwise ranking loss is the one that orders search results and the one behind preference models in modern language models; the split-conformal calibration is a two-line addition to any regression, in demand forecasting or medical risk, and the coverage counter is an audit that any deployed model can carry; double machine learning is how a treatment effect is estimated from observational data on customers or patients with hundreds of controls; Ledoit–Wolf shrinkage is the default covariance estimator in any field where \(N\) is not small relative to \(T\); the two-stage label-and-size design is the architecture of fraud, screening and lead-scoring systems; and the deflated score, the overfitting probability and the false-discovery correction are what an experimentation platform or a genome-wide screen needs before it announces a winner. The chapter’s results are negative where the earlier chapters’ were, and for the same reason: honest numbers are small, and the protocol is what makes them honest. What the six methods add is the ability to ask better questions of small numbers. Not “what is the outcome?” but “what is the order, and does any loss find it?”; not “what is the point forecast?” but “how wide is the range, with what guarantee, and has the guarantee broken?”; not “is this variable correlated with the outcome?” but “what does it do with fourteen controls held fixed in any shape, and how many independent periods is that estimate worth?”; not “what are the optimal weights?” but “which weights survive the estimation error in the matrix?”; not “was the end point positive?” but “which barrier did the path touch first, and how much should the first stage be backed?”; and, before any of it is acted on, “what is the best of everything I tried worth, once I count what I tried?”.
The message of the course. The tools grew from a pandas Series in Chapter 1 to an attention layer in Chapter 4, a GARCH estimator in Chapter 6 and a deflated performance score here, and the discipline never changed: a fixed seed, a held-out window, a baseline to beat, and a count of everything you tried. Every method in this chapter is an application of that discipline to a new object — a loss, an interval, an effect, a matrix, a label, a selected score — and every failure mode you met was a place where the discipline was skipped: a calibration set that was the training set, a nuisance model that saw its own row, a position that used today’s close, a label that looked into the test year, a winner reported without its 41 losers.
The companion slide deck for this chapter follows the same six sections with the same data, seeds and printed numbers, adds multiple-choice checkpoints and practice cells for each, and closes with a decision memo on whether two of the chapter’s results should be acted on, a set of copilot prompts (make it count the trials; ask for the calibration split and coverage by month; cross-fitting is not optional) and four discussion questions. Read it alongside this chapter; the two are built to agree.