Chapter 7: Modern Statistical Learning in Practice
Chapter Introduction
The boosting examples in §§4.5 and 6.2 produced weak ranking and directional predictions. This chapter asks what the decision actually needs: an order, a range, a partial effect, an allocation, an event label or evidence that a selected result survives its search. Section 7.7 then connects these methods to a source-linked coding reference.
The examples reuse xs_panel_monthly.csv (213 stocks), sp500.csv and dji.csv, and add etf_daily_prices.csv (22 price columns, 21 retained return series). The monthly panel is survivorship-biased. Reported performance is before costs; fixed-weight and signal-times-log-return demonstrations do not establish an executable net portfolio account.
Slides and Book share data, seeds and diagrams across seven sections and 21 parts. Chronological prediction comparisons, random-row DML folds, simulated examples and CSCV each answer different questions; their assumptions remain attached to their results.
How to use the numbered learning path
Follow the numbered parts in each section: concept → prediction → example → explanation → check.
The 3.5-hour class selects examples, group discussion and notebook work. The complete Slides and Book support self-study.
Each setup links to its earlier introduction. Change inputs in the current cell before the operation you test.
Table of Contents
- Learning to Rank
- Quantile and Conformal Prediction
- Double Machine Learning
- Covariance Shrinkage and Cluster Allocation
- Event Labels and Two-stage Decisions
- Strategy Selection and Multiple Testing
- A Revisable Course Reference for Coding
Learning to Rank
| Part | Question |
|---|---|
| 7.1.1 Ranking objectives and evaluation | Does the decision require an accurate level or ordering? |
| 7.1.2 Pointwise and pairwise learners | How can a model learn the desired ordering? |
| 7.1.3 Common-protocol comparison and concentration | Does a changed objective improve the same decision? |
Search results, recommendation slots, triage queues and shortlists often require an ordering. Here the §4.5 top-20 / bottom-20 portfolio uses order with equal weights. Compare raw-return regression, rank targets, pairwise logistic loss and trees on the same walk-forward dates; other decisions can also require calibrated score magnitudes.
Ranking objectives and evaluation
Question: Does the decision require an accurate level or ordering?
Result: Define date-level rank IC and a top-minus-bottom comparison.
The same matrix, one new column
At month-end \(t\), \(X_t\) contains five characteristics: last-month return, 12-1 momentum, realised volatility, log dollar volume and price/252-day-high. ret_next is the following month’s return. Standardise features within the observed month: subtract the median, divide by sample SD, clip at ±3 and fill missing values with zero. This avoids using later months’ feature distributions. The new target is rank(pct=True) − 0.5, approximately centred.
The panel has 25 335 rows, 119 months and 213 distinct stocks; monthly counts vary from 211 to 213. Raw returns have mean 0.013, SD 0.082 and range [−0.735, +0.805]. A squared residual of 0.8 contributes 64 times one of 0.1. Rank targets bound this influence while discarding return magnitude. Without ties, the target grid is \(1/N-0.5,\ldots,0.5\), with mean \(1/(2N)\), not zero. pandas averages ties; 15 months have tied returns. Thus months have similar target scales, not exactly identical distributions. See pandas rank.
Why a top-K portfolio only needs the order
October 2022 uses the illustrative score \(p_1=0.01\times\) momentum z-score and the strictly increasing transform \(p_2=3\tanh(50p_1)\). Predict whether equal-weight top/bottom selection changes when only score magnitudes change.
The selected names and realised spread stay identical; the MSE changes from 0.0134 to 1.4686, about 110-fold. This invariance applies to the equal-weight sort. Magnitude-based exposure, calibrated probabilities and return forecasting can require more than ordering.
Does the rank IC see the transform?
The §4.5 scoreboard uses rank IC: Spearman correlation of predictions with realised returns across the month’s stocks. It equals Pearson correlation of their average ranks. Predict whether a strictly increasing score transform changes this ordering or the Pearson correlation of values.
Will spearmanr(p1, y)[0] and spearmanr(p2, y)[0] agree to four decimals for October 2022, and will the two Pearson correlations?
October’s rank IC is −0.3316 for both scores; Pearson moves from −0.3376 to −0.3329. The monthly diagnostic t-statistic is \(\bar{IC}\sqrt T/s_{IC}\) and its IID standard error is \(s_{IC}/\sqrt T\). These formulas require independent monthly observations for their usual uncertainty interpretation; autocorrelation calls for time-aware inference.
Pointwise and pairwise learners
Question: How can a model learn the desired ordering?
Result: Compare raw-return, rank-target and pairwise losses.
Builds on Ranking objectives and evaluation.
The scoreboard, and the baseline: least squares on the raw return
Use §4.5’s expanding protocol: test months January 2019–November 2024, refit every 12 months, and score each month separately. Training rows precede the block. Their next-month labels must already be observable at the assumed scoring-close refit; the public file has no actual release timestamps. The top-20 minus bottom-20 spread is equal weighted and before costs. Its annualised Sharpe is \(\bar s/s_s\sqrt{12}\); this is a conventional diagnostic, not an execution account.
OLS reports mean IC +0.0014 (t = 0.06), gross spread +8.5%/yr and Sharpe 0.34. Final coefficients are +0.26% per momentum z-unit, +0.26% per volatility z-unit, −0.22% per price/high z-unit and −0.004% per last-return z-unit. A coefficient describes the final fit, not a validated economic effect.
(a) Change the target: regress on the rank
The simplest way to target the order is to keep the model and change the column it is fitted to. Replacing ret_next by its within-month percentile does not change the model class, and it does not change the features, which are already ranks in spirit (z-scores within the month). What it changes is the weighting of months: every month now contributes the same spread of targets, so the ±80 % months no longer dominate, and the model listens equally to the quiet months in which the order was subtle.
Rank-target OLS: IC −0.0038 (t = −0.17), gross spread +5.0%/yr and Sharpe 0.24. Final coefficients are −0.0038 rank units per last-return z-unit and +0.0084 per momentum z-unit. The signs match reversal and momentum hypotheses; this weak evaluation does not confirm those effects.
(b) Learn from pairs: the RankNet loss
A ranking compares items within the same month. RankNet models the probability that item \(i\) is preferred to \(j\) from the difference of their learned scores:
\[P(i \succ j) = \sigma\big(f(x_i) - f(x_j)\big), \qquad \sigma(u) = \frac{1}{1 + e^{-u}},\]
For \(f(x)=w^\top x\), fit a no-intercept logistic regression to \(x_i-x_j\), with label \(\mathbb 1[r_i>r_j]\). LogisticRegression estimates these pair-order probabilities with its default L2 penalty (C=1), so the fit maximises a penalised likelihood. Swapping a non-tied pair negates the margin and reverses the label; an intercept would break that symmetry. At \(N=213\) there are \(\binom{213}{2}=22\,578\) unordered non-self pairs. The code draws 400 ordered pairs with replacement per month, removes self-pairs and fixes the seed; it does not validate equality with a full-pair fit. Return ties are coded 0, so a general extension needs a tie policy.
The final refit uses 42 986 sampled non-self pairs. Pairwise logit reports IC −0.0046 (t = −0.21), gross spread −1.8%/yr and Sharpe −0.09; weights include −0.031 for last return and +0.026 for momentum. The ranking objective does not itself establish forecasting value.
(c) The §4.5 boosting machine, on both targets
Hold the §4.5 histogram-boosting recipe and folds fixed: depth 2, learning rate 0.05, 200 iterations and minimum leaf size 50. Fit once to raw returns and once to rank targets.
HGB-MSE gives IC −0.0150 (t = −1.04), gross spread +7.7%/yr and Sharpe 0.47. HGB-rank gives IC +0.0165 (t = +0.98), +9.3%/yr and Sharpe 0.64. Their paired monthly IC difference is +0.03151 with IID SE 0.01948 (t = 1.62); comparing either model’s standalone SE is not the correct comparison. These diagnostics do not establish superiority and still omit serial dependence and selection adjustment.
Common-protocol comparison and concentration
Question: Does a changed objective improve the same decision?
Result: Compare learners on matched dates and test portfolio concentration.
Builds on Pointwise and pairwise learners.
Five models, one honest picture
The chart sums monthly top-minus-bottom returns; it is an additive gross spread diagnostic, not compounded NAV. The printed IID mean-IC SE and cross-model IC correlations help compare the same test dates.
The average IID SE rounds to 0.02, but HGB-MSE’s |t|=1.04 is slightly above one own SE. OLS-rank/pairwise IC correlation is 0.92, pairwise/HGB-rank 0.87 and OLS-rank/HGB-rank 0.81; MSE-tree correlations with other learners are 0.20–0.31. Different objectives produce different rankings. This gross survivor-panel comparison, with temporal dependence and model search unresolved, does not establish an investment result.
Where this leads: preference models
A score is latent utility in a pairwise preference model, as in Bradley–Terry sports comparisons or human preference learning. The scorer can extend from linear \(w\) to §4.5 attention; listwise objectives can emphasise selected extremes. Portfolio-based objectives, including the Kelly, Kuznetsov, Malamud and Xu cross-sectional-transformer example, connect learning to a decision criterion. Preference-based asset-pricing research can combine estimation, inference and construction; score magnitude still matters for decisions that require calibration or exposure size.
Changing the objective changes what is estimated. This small survivor-panel demonstration does not establish a reliable ordering; more ambitious objectives still require independent evaluation and uncertainty.
How concentrated should the portfolio be?
Hold the rank-tree predictions fixed and evaluate \(K=10,20,40\). Smaller K concentrates on fewer extreme predicted ranks; it does not mean the model is more certain about those names.
The rank IC stays +0.0165 for every K. Choosing the largest of three observed Sharpes adds selection (§7.6). The rough IID annualised-Sharpe SE \(\sqrt{12/71}\approx0.41\) is only a diagnostic; it does not prove the three ratios indistinguishable. Paired, dependence-aware comparison is needed.
Nothing. Both the portfolio and the Spearman rank IC depend on the predictions only through their order, and a strictly increasing transform preserves every pairwise comparison. In October 2022 the spread stayed at −11.94 % and the rank IC at −0.3316 while the MSE went from 0.0134 to 1.4686. The MSE measures the level of the predictions, which a top-K portfolio never uses.
No. Compare the paired monthly IC differences: mean +0.03151, IID SE 0.01948, t=1.62. That does not establish superiority and does not prove the difference is pure noise. Rank-based models’ monthly IC correlations are 0.81–0.92, versus 0.20–0.31 with the MSE trees; their learned order differs. Dependence, search and the survivor universe limit the evidence.
Quantile and Conformal Prediction
| Part | Question |
|---|---|
| 7.2.1 Conditional quantiles and interval checks | How do we predict a range instead of a point? |
| 7.2.2 Conformal calibration | How does held-out calibration adjust an interval? |
| 7.2.3 Coverage under drift and risk applications | What happens when future data differ from calibration? |
An interval supports range-sensitive decisions such as inventory buffers, clinical risk review and exposure sizing. Use the §6.2 daily features to fit quantiles, calibrate on a separate year, assess realised coverage through 2020–24, and explore sizing by interval width. Conditional quantile forecasts and finite-sample marginal coverage are different objects; exchangeability is required for the split-conformal guarantee.
Conditional quantiles and interval checks
Question: How do we predict a range instead of a point?
Result: Fit quantiles and measure interval coverage and width.
The loss that estimates a quantile
Squared-error loss estimates a conditional mean. Pinball (check) loss at \(\tau\in(0,1)\) estimates a conditional quantile:
\[\ell_\tau(y, q) = \begin{cases} \tau\,(y - q) & y \ge q \\ (1-\tau)\,(q - y) & y < q. \end{cases}\]
For continuous \(Y\), differentiating expected loss gives \(-\tau P(Y>q)+(1-\tau)P(Y<q)=0\), hence \(F(q)=\tau\). With atoms the quantile condition is \(F(q-)\le\tau\le F(q)\). At \(\tau=0.95\), under-prediction costs nineteen times over-prediction; at 0.5 the loss is half the absolute error. HistGradientBoostingRegressor(loss="quantile", quantile=τ) uses this objective. Fit levels 0.05, 0.5 and 0.95 for a lower edge, median and upper edge; independently fitted quantiles need not be non-crossing.
The §6.2 features, split three ways
Use §6.2’s eleven features: five lagged returns, 5/21-day rolling means and SDs, 12-1 momentum and short/long volatility ratio. Every feature is shifted, so row \(t\) uses returns through \(t-1\). Add a held-out calibration year between tree fitting and test evaluation.
The dates are 5 January 2016–2018 for fitting, 2019 for calibration, and 2020–24 for evaluation. The quiet calibration year does not establish exchangeability with a later crisis.
Three quantile models
The fitted training median averages 0.053%. Width and lagged std21 correlate 0.818 in training. Flexible quantile bands can be asymmetric and respond to other features, but this association does not validate unseen coverage or prove that direction is unforecastable.
Do the bands keep their promise?
A nominal 90% interval from fifth and ninety-fifth conditional quantiles aims to cover 90% under an accurate predictive distribution. Predict the realised rates on train, calibration and test windows before running the cell.
Will the 90 % band’s coverage be near 0.90 on all three windows, or will it fall from training to calibration to test — and by how much?
The pattern is consistent with model error and distribution change, but the counts alone do not identify either contribution. Held-out scores can calibrate interval size; exchangeability still has to be justified.
Conformal calibration
Question: How does held-out calibration adjust an interval?
Result: A calibration quantile with its exchangeability assumptions stated.
Builds on Conditional quantiles and interval checks.
Split conformal: a guarantee from one held-out quantile
Split conformalized quantile regression (Romano, Patterson and Candès, 2019) adjusts a fitted interval with held-out nonconformity scores. Keep the model fixed after its separate training fit:
\[s_i = \max\big(\hat q_{0.05}(x_i) - y_i,\; y_i - \hat q_{0.95}(x_i)\big), \qquad i = 1, \dots, n.\]
Sort the scores and use rank \(k=\lceil(n+1)(1-\alpha)\rceil\). Then \([\hat q_{0.05}(x)-\hat q,\hat q_{0.95}(x)+\hat q]\) expands for positive correction and shrinks for negative correction. If \(k>n\), define the correction as \(+\infty\); the shown \(n=252,\alpha=0.10\) has \(k\le n\).
With the model held fixed after a disjoint fit and calibration/new scores exchangeable, \(P(y_{new}\in\text{band})\ge1-\alpha\) in finite samples. This is marginal coverage over the calibration and new observation, not coverage conditional on each date or feature value.
For continuous exchangeable scores, the new score’s rank among \(n+1\) positions is uniform. It falls below the calibration threshold in at least \(k/(n+1)\ge1-\alpha\) of those positions. Ties make the coverage bound conservative; they do not give an exactly uniform deterministic rank. The model may be misspecified, but it must not use calibration outcomes during fitting. Daily dependence and changing regimes can break score exchangeability.
Calibrate on 2019, test on 2020–2024
The +0.182 percentage-point correction widens mean test width from 2.58 to 2.94 percentage points. Coverage rises from 0.773 to 0.830. By year it is 0.727, 0.869, 0.789, 0.892 and 0.873. These empirical rates diagnose performance; they do not prove exchangeability or validate a guarantee in an individual year.
The following repair task uses training rows as calibration. Inspect why more scores from the wrong sample cannot support the held-out guarantee.
The incorrect cell reports \(n=753\), \(k=679\), a rounded correction of 0.000% and test coverage 0.773. These fitted training observations are not held-out error scores. Repair all references from trn to cal when computing the calibration bands, targets and scores; the fixed model must not learn from the calibration outcomes. The unchanged test predictions then permit the intended comparison.
See the band through the crash
January–June misses are 3, 6, 18, 11, 6 and 5. Lagged features react after new returns become observable; the band does not anticipate an abrupt shift. Actual lower/upper edges need not be symmetric, so a width is not a ±half-width around zero.
Coverage under drift and risk applications
Question: What happens when future data differ from calibration?
Result: Diagnose time-varying coverage before using width for sizing.
Builds on Conformal calibration.
March 2020: coverage as a regime alarm
January–June rates are 0.86, 0.68, 0.18, 0.48, 0.70 and 0.77. The 21-day rate reaches 0.143 on 30 March. March’s misses occur on both sides; this is consistent with increased dispersion, but it does not rule out other calibration errors or identify a regime. Counts become available after returns arrive, so monitoring is necessarily delayed.
Why the guarantee failed, and what restores it
The ordinary split-conformal guarantee requires exchangeable scores; dependent, changing returns do not automatically satisfy it. A trailing window can adapt the correction empirically. The next cell uses past observed scores only by applying .shift(1); it uses pandas’ interpolated 90% quantile rather than the corrected finite-sample order statistic. This rolling heuristic is not the distinct adaptive-conformal algorithm with a proved long-run coverage result (Gibbs and Candès, 2021).
The realised 2020–24 average is 0.893, with annual rates 0.84, 0.94, 0.84, 0.96 and 0.88. March remains 0.18, then April 0.67, May 0.90 and June 0.91. These are empirical outcomes, not proof of restored coverage. Monitoring can prompt a review without identifying a regime or assuring future calibration.
Sizing by width: the §6.6 overlay without GARCH
§6.6 used \(w_t=\min(\sigma^*/\hat\sigma_t,w_{max})\) from a forecast known before the target return. Replace forecast SD by this fitted interval’s width and target two percentage points, capped at 1.5. Width is a forecast quantity, not model-free risk. The next code multiplies log returns by exposure and prints additive diagnostics; it does not implement exact leveraged NAV.
Mean exposure is 0.76, minimum 0.37, cap 1.5; weighted log-return SD is 0.796% versus 1.350%. The code reports summed weighted log returns 41.9% versus 59.9%, annualised mean/SD ratio 0.66 versus 0.56, and declines in cumulative log-return sums of −21.2 versus −41.4 percentage points. These are not exact cumulative wealth returns or NAV percentage drawdowns. A leveraged account needs simple-return conversion, a wealth recursion beginning at opening NAV, financing and costs. The original §6.6 GARCH diagnostics (0.77, −22.6) use another forecast; this in-sample model comparison does not establish economic superiority or guaranteed profits.
Trade coverage for width
Keep the fitted quantile models fixed and change only the conformal calibration level: \(\alpha=0.20,0.10,0.05\). Measure the change in correction, width and empirical test coverage.
The price is sample-specific: shrinking or widening changes coverage and average width, while drift keeps the test rate below nominal. Choosing a confidence level does not itself fix the calibration assumptions.
With the model fixed after disjoint training, use order statistic \(k=\lceil(n+1)(1-\alpha)\rceil\) on held-out scores, with \(+\infty\) for \(k>n\). Exchangeability of calibration/new scores gives marginal \(P(y_{new}\in\text{band})\ge1-\alpha\); not a per-date or conditional guarantee. Training scores shaped the model and cannot simply replace held-out calibration. Here the training correction rounds to 0.000% and test coverage remains 0.773.
Exchangeability is not justified for dependent, changing returns; empirical March coverage cannot establish that assumption. The trailing-window heuristic observes 0.893 average coverage but does not restore a finite-sample guarantee under arbitrary drift. March’s 45% below and 36% above misses can trigger a calibration review once returns arrive; they do not identify the cause or guarantee a forecastable regime.
Double Machine Learning
| Part | Question |
|---|---|
| 7.3.1 Controls and residualisation | What variation remains after controlling for covariates? |
| 7.3.2 Cross-fitting and orthogonal estimation | How do we reduce bias from flexible nuisance fits? |
| 7.3.3 Empirical interpretation and identification limits | Which causal assumptions still require justification? |
§4.4 separated prediction from causation. Here a measured nonlinear confounder defeats linear adjustment; flexible nuisance fits and cross-fitting address estimation, while identification remains a separate question.
Controls and residualisation
Question: What variation remains after controlling for covariates?
Result: Trace residualised treatment and outcome into an effect estimate.
Partialling out: what “controlling for \(X\)” really does
For \(Y=\theta D+X\beta+\varepsilon\), Frisch–Waugh–Lovell (FWL) is exact sample OLS algebra: include the intercept in \(X\), use \(P_X\) for its linear projection, and regress \((I-P_X)Y\) on \((I-P_X)D\). The resulting coefficient equals the multiple-regression coefficient on \(D\). The population analogue uses conditional means:
\[\tilde Y = Y - \mathbb E[Y \mid X], \qquad \tilde D = D - \mathbb E[D \mid X], \qquad \hat\theta = \frac{\sum_i \tilde D_i \tilde Y_i}{\sum_i \tilde D_i^2}.\]
A linear projection need not equal the conditional mean. A measured curve such as \(\sin(2x)\) can remain in both residuals, biasing the partial slope. The OLS standard error does not diagnose this misspecification.
DML (Chernozhukov et al., 2018) estimates the two conditional means with flexible learners. Cross-fitting avoids using a row to train its own nuisance prediction; orthogonality reduces local sensitivity to nuisance estimation error. Neither removes unmeasured confounding.
Break the linear control
The cell builds a world in which the answer is known. Three covariates; a confounder \(g(X) = 1.5\sin(2x_0) + 1.5\sin(2x_1)\) that drives both the signal \(D\) and the outcome \(Y\); a true effect \(\theta = 0.5\). The naive regression of \(Y\) on \(D\) ignores \(X\); the second regression controls for \(X\) linearly, the way §4.4 controlled for \(U\).
Naive 1.264; linear controls 1.211 versus truth 0.5. The linear fit explains 13.9% of \(g\) variance; this is not a measured fraction of a causal backdoor closed.
Cross-fitting and orthogonal estimation
Question: How do we reduce bias from flexible nuisance fits?
Result: A cross-fitted estimate and an explanation of why the split matters.
Builds on Controls and residualisation.
The DML recipe: cross-fitting and Neyman orthogonality
For held-out row \(i\), \(\tilde D_i=D_i-\hat m_{-k(i)}(X_i)\) and \(\tilde Y_i=Y_i-\hat\ell_{-k(i)}(X_i)\). Combine all folds:
\[\hat\theta=\frac{\sum_i\tilde D_i\tilde Y_i}{\sum_i\tilde D_i^2},\qquad \psi_i=\tilde D_i(\tilde Y_i-\hat\theta\tilde D_i),\qquad \widehat{se}_{iid}=\frac{\sqrt{\sum_i\psi_i^2}}{\sum_i\tilde D_i^2}.\]
Neyman orthogonality means the population moment’s nuisance derivative is zero at the truth. Its second-order remainder includes \(\mathbb E[\delta m(\delta\ell-\theta\delta m)]\), so both product and squared treatment-nuisance errors matter. A sufficient rate condition is \(\|\delta m\|_2(\|\delta\ell\|_2+\|\delta m\|_2)=o(n^{-1/2})\); two errors merely \(O(n^{-1/4})\) do not ensure negligible bias. Independent-row splitting, nonzero residual variation and regularity conditions are also required.
DML by hand in twenty lines
KFold supplies row splits; HistGradientBoostingRegressor is the histogram boosting learner used for the nuisance fits. Both are from scikit-learn; the split is explained in §7.3.2 and boosting in §4.3.
DML retains 71% of raw \(D\) standard deviation. If residual \(D\) is identically zero, the slope is unidentified, not an estimate with an infinite standard error. Shuffled folds suit these independent simulated rows.
Does the split matter?
Compare two two-fold seeds with the existing five-fold estimate. These are split-sensitivity diagnostics for the same simulated sample.
Will both \(K = 2\) estimates stay within two standard errors (0.046) of the \(K = 5\) answer of 0.496?
Both estimates are close on this sample. This does not establish seed stability generally or explain their differences as residual confounding. Repeat splits and report their variation; the DML paper uses median aggregation.
Why not fit the nuisance models on the same rows?
For the seeded unlimited-depth learner below, predict how the same-row estimate compares with truth 0.5. The direction is a property of this example, not a universal overfitting rule.
The same-row learner gives 0.377 here. The comparison also changes learner flexibility, so it does not isolate cross-fitting alone; bias direction is not universal and exact memorisation can make the slope undefined.
Empirical interpretation and identification limits
Question: Which causal assumptions still require justification?
Result: Compare estimates without treating machine learning as identification.
Builds on Cross-fitting and orthogonal estimation.
The real question: what does momentum do?
Now the §4.5 panel. \(D\) is the per-month z-score of 12-1 momentum; \(Y\) is next month’s return in per cent; \(X\) is the other four characteristics plus the sector dummies. The question is whether momentum has a partial association with next month’s return once the other characteristics — including the one it is most entangled with — are held fixed in whatever functional form the trees find.
25 335 stock-months, 14 controls and 119 months. Momentum correlates 0.51 with price relative to the 252-day high and 0.10 with dollar volume; correlation alone does not identify confounding or explain a coefficient change. The input is median-centred, divided by the within-month standard deviation, clipped and missing values set to zero: a coefficient unit is one pre-clipping input-scale unit, not necessarily one realised standard deviation.
Naive, with controls, and DML
Month clustering includes within-month dependence by summing influence contributions before squaring. The displayed formula has no finite-cluster adjustment; independence across months is a further assumption.
Naive +0.156%, linear-adjusted +0.201%, DML +0.226% per input-scale unit. The latter is about 2.7 percentage points under simple ×12 annual scaling, not a realised annual return. The nuisance fit leaves 59% of \(D\) variance (about 77% of its standard deviation). A coefficient increase alone does not attribute the change to one control.
The panel still uses shuffled row folds. Month clustering does not make the 119 months independent or correct cross-fold stock/month dependence; credible inference needs dependence-aware splits and covariance assumptions. Cluster-robust DML treats splitting and inference together.
What DML does and does not buy you
OLS uses a chosen linear control space; DML estimates flexible nuisance functions. Neither guarantees adequate fit or identification. The iid or clustered SE is valid only under its matching sampling and rate conditions.
Read the panel coefficient as a fitted partial association. A causal effect requires a meaningful intervention, consistency, relevant residual variation and justified no-unmeasured-confounding assumptions. DiD and instruments have their own assumptions; they are alternative designs, not an automatic hierarchy of credibility.
OLS projects onto the chosen linear control space; it need not estimate a curved conditional mean. DML uses flexible nuisance fits and an orthogonal residual score. Cross-fitting prevents own-row nuisance training; product and squared nuisance errors still require sufficiently fast convergence. None of this supplies causal identification.
Month clustering includes within-month residual dependence and gives 0.164 in this sample, versus 0.089 under the iid calculation. It does not establish independent months or fix the shuffled row folds. The displayed result is an illustrative partial-association calculation, not certified panel inference.
Covariance Shrinkage and Cluster Allocation
| Part | Question |
|---|---|
| 7.4.1 Covariance instability and shrinkage | Why can small covariance errors create extreme weights? |
| 7.4.2 Hierarchical risk parity | How does a correlation tree determine an allocation? |
| 7.4.3 Walk-forward allocation comparison | Which trade-offs survive a common evaluation period? |
§6.6 sized one position; allocating across correlated assets needs a covariance estimate. With weights summing to one and no short-sale constraint, minimum variance gives \(w=\Sigma^{-1}\mathbf1/(\mathbf1^\top\Sigma^{-1}\mathbf1)\). We compare its sample and shrunk estimates with equal weight and HRP on the same 21-ETF archive.
Covariance instability and shrinkage
Question: Why can small covariance errors create extreme weights?
Result: Diagnose conditioning and explain shrinkage towards a target.
Twenty-one ETFs, one cash proxy removed
2 515 daily observations, 2015–2024, across 21 risky ETFs after excluding SHY, a low-volatility short-Treasury fund rather than literal cash. The universe covers Treasuries, credit, currencies, metals, oil, equity indices and sectors; annualised sample volatility ranges from 6.7% (IEF) to 40.1% (USO). The exclusion is a modelling choice, not a universal allocation rule.
Why the sample covariance is ill-conditioned
For independent, identically distributed observations, the \(T-1\) sample covariance is unbiased entry by entry, yet its eigenvalues and inverse can remain unstable. There are \(N(N+1)/2=231\) covariance entries for 21 assets on 251 days. Under an independent identity-covariance large-dimensional benchmark, the approximate eigenvalue range \([(1-\sqrt{N/T})^2,(1+\sqrt{N/T})^2]\) is about \([0.50,1.66]\); this is not a fitted bound for the ETF archive.
\(\kappa=\lambda_{\max}/\lambda_{\min}\) describes sensitivity of inversion. In an eigenbasis, \(\Sigma^{-1}\mathbf1=\sum_j(v_j^\top\mathbf1)v_j/\lambda_j\): small eigenvalues amplify only directions onto which \(\mathbf1\) projects. Large long/short weights, unstable targets and understated future risk are possible consequences, not inevitable outcomes of every inverse.
How ill-conditioned is one year of daily data?
The archive gives condition numbers 6 373 → 415 in 2018 and 5 010 → 299 in 2019 under Ledoit–Wolf. Intensities are 0.029 and 0.034. These are conditioning diagnostics, not a fixed conversion from 1% covariance error into a stated inverse error.
The weights that the inverse produces
For 2018 targets, predicted versus 2019 realised annualised weighted-log volatility is 1.54% → 2.00% (sample) and 1.82% → 2.39% (LW). Estimation error and a changing covariance can both contribute; these realised gaps do not isolate their causes.
Ledoit–Wolf: pull the matrix toward a target
Ledoit and Wolf (2004, “A well-conditioned estimator for large-dimensional covariance matrices”) proposed replacing \(S\) by a convex combination of itself and a structured target,
\[\hat\Sigma_{LW} = (1 - \delta)\, S + \delta\, \mu I, \qquad \mu = \frac{\operatorname{tr}(S)}{N}, \qquad \delta^\ast = \arg\min_\delta \mathbb E\big\lVert \hat\Sigma_{LW} - \Sigma \big\rVert^2.\]
The target \(\mu I\) estimates one average variance and sets off-diagonals to zero; it reduces complexity but still has sampling error. The intensity is estimated to trade error in the matrix against target bias. Trace is preserved relative to the base covariance. sklearn.covariance.LedoitWolf centres and uses denominator \(T\); np.cov above uses \(T-1\). These scalar conventions do not change unconstrained normalised minimum-variance weights or a condition number. Estimator definition.
Other targets include constant correlation and a one-factor CAPM covariance; nonlinear shrinkage adjusts eigenvalues separately. Sampling error also affects fixed-portfolio variance and correlations; avoiding inversion does not make an estimated matrix exact.
Hierarchical risk parity
Question: How does a correlation tree determine an allocation?
Result: Trace distance, clustering and recursive bisection into weights.
Builds on Covariance instability and shrinkage.
HRP: correlation becomes distance, distance becomes a tree
Hierarchical risk parity uses correlation distance, a single-linkage tree, its leaf order and recursive list bisection. Cluster variance uses the full covariance submatrix: individual variances alone are insufficient. HRP avoids a full inverse, but tree changes and higher-level cluster variances can propagate covariance errors.
\(d_{ij}=\sqrt{(1-\rho_{ij})/2}\) is half the chord distance between standardised return vectors on a common sample. It is zero for perfectly correlated assets and one at perfect anticorrelation; distinct assets can have zero distance. Single-linkage merges clusters through their closest pair; leaves_list supplies the left-to-right order. squareform converts a symmetric distance matrix to the condensed vector used by linkage.
Two assets with \(\rho = 0.5\) — what is their distance \(d\), to two decimals? And two with \(\rho = -1\)?
The original 2018 tree places dollar/metals, bond funds and equity-sector groups near one another. Adjacent mean correlation is 0.63 versus 0.22 in file order; reordering preserves every pair correlation and does not make off-block correlations vanish.
See the blocks
The matrix shows estimated blocks and negative relationships on the same 2018 sample. The next operation bisects its leaf list, which need not follow each dendrogram merge.
Recursive bisection
Start with all assets in tree order and weight 1. Split the list in half; compute each half’s variance as the variance of its inverse-variance portfolio (weights proportional to \(1/\sigma^2_i\) within the half, a rule that needs no inverse of anything but a diagonal); give the left half the share \(\alpha = 1 - v_L/(v_L + v_R)\), so that the lower-variance half gets more; multiply the weights in each half by its share; and recurse into each half until every cluster is a single asset.
The original weights sum to one and are positive: UUP 0.308, LQD and IEF about 0.17, GLD 0.12. Positivity requires well-defined positive cluster variances. A covariance change can affect ancestor splits or the tree, so errors are not guaranteed to stay local.
Walk-forward allocation comparison
Question: Which trade-offs survive a common evaluation period?
Result: Compare risk, return and concentration without selecting on one metric.
Builds on Hierarchical risk parity.
Nine years walk-forward: which rule wins what?
Four rules share the same dates and universe. The code multiplies each daily asset log return by fixed annual target weights: a fixed-weight log-return approximation, not an exact buy-and-hold portfolio accounting model. Predict the observed ordering before running; no rule guarantees lower realised risk.
Before-cost weighted-log statistics favour LW’s observed Sharpe (1.01) over sample minimum variance (0.82) and HRP (0.84). HRP remains long-only in these runs. In 2020, the risk-based approximations sum to 0.4–3.4% log return against 6.6% for equal weight. These sums are not exact compounded portfolio returns.
turnover/yr is mean annual \(\sum_i|w_{new,i}-w_{old,i}|\), not trades from drifted holdings; equal annual targets produce zero without implying no rebalancing. Sample 1.48, LW 0.70 and HRP 0.56 are observations on these windows, not estimator guarantees. Initial funding, drift, trading costs and borrowing are omitted. maxDD reports cumulative-log drawdown and omits opening zero, so it must not be labelled compounded NAV drawdown. An annualised Sharpe SE near \(1/\sqrt9\approx0.33\) is only a rough independence-based heuristic; serial dependence and selection can change uncertainty. The results support a qualified comparison, not a mandate-specific winner.
A large condition number signals sensitivity to perturbations, not a fixed realised error multiplier. Shrinkage moves empirical eigenvalues towards their estimated average; trace preservation uses the matching base denominator. Here condition numbers fall from 6 373 to 415, and annual target change from 0.99 to 0.52.
HRP uses correlation distance, single-linkage leaf order and ordered-list bisection. Cluster variances come from inverse-variance weights and covariance submatrices; no full covariance inverse is needed. UUP receives 0.308 here through its path of splits, but a changed tree or ancestor variance can change that weight.
Event Labels and Two-stage Decisions
| Part | Question |
|---|---|
| 7.5.1 Triple-barrier labels | Which event determines a trade’s outcome? |
| 7.5.2 Primary signals and the meta-model | Should we act on an existing directional signal? |
| 7.5.3 Probability, position size and comparison | How does the second-stage probability affect exposure? |
Two-stage systems separate a proposed action from its acceptance or size: fraud alerts, patient referrals and sales leads use this pattern. An event label should follow the decision’s stated outcome window. Here a daily-close S&P 500 path supplies a volatility-scaled triple-barrier label, a moving-average rule supplies direction, and a later-trained model predicts whether to accept that direction. Training must exclude outcomes unavailable at the refit date.
Triple-barrier labels
Question: Which event determines a trade’s outcome?
Result: Trace upper, lower and time barriers and their label end times.
What is wrong with “was the 10-day return positive”?
A fixed-horizon label \(\mathbb 1[r_{t\to t+h}>0]\) uses the endpoint. A stop-based decision also depends on the intervening path: a trade can cross a loss limit before recovering. Which label is appropriate depends on the decision being modelled; the endpoint label remains appropriate for a fixed-horizon target.
Set three barriers at entry close \(t\): \(+m_{pt}\hat\sigma_t\), \(-m_{sl}\hat\sigma_t\) and \(t+h\). The first observed upper/lower crossing gives +1/−1; if neither occurs, use the return sign at the vertical barrier. Here \(h=10\) trading days and \(\hat\sigma_t\) is the EWM daily log-return SD with span 21, frozen at entry. Scaling by estimated volatility adapts the numerical threshold; it does not ensure equal risk across regimes. Daily closes omit intraday crossings, and neither crossing labels nor nominal stops guarantee executable fills or bounded losses.
Compute the labels, vectorised
fwd is a \(T\times h\) matrix: column \(j\) holds \(\log P_{t+j}-\log P_t\). Compare each path with its entry barriers and take the earliest crossing column. np.column_stack joins these one-dimensional paths as columns; np.where(..., days, H+1).min(axis=1) uses \(H+1\) as the no-touch sentinel.
ok requires all ten future closes and a nonmissing entry volatility, so the label comparison has one common denominator. The reported mean 6.0 days includes vertical exits. The 11.5% disagreement is a target difference; it does not establish that one classifier is universally trained on the wrong answer.
One entry, three barriers
At the original 2σ setting, day 1 first crosses the lower barrier; sentinel 11 means no upper touch. The remaining plotted closes belong to the realised price path, not to a guaranteed held position after exit. Label timing is observed at closes; actual execution needs intraday data, a fill convention and costs.
Primary signals and the meta-model
Question: Should we act on an existing directional signal?
Result: Separate direction from acceptance and respect overlapping label horizons.
Builds on Triple-barrier labels.
The primary signal chooses the side
The primary model chooses direction; the meta-model predicts acceptance or size conditional on that primary choice. This converts the second-stage target to a binary outcome. Class balance depends on the primary signal and sample; a two-stage construction does not guarantee balanced classes or improvement.
The primary is long when its 50-day moving average exceeds its 200-day average, and short otherwise. Seven features use data available by close \(t\): side, EWM volatility, 21-/63-day momentum, 5-/21-day volatility ratio, distance from the 200-day average and 14-day RSI. The symmetric ±2σ barriers make side–label sign agreement a directional-success proxy; unequal profit/stop multipliers would require labels constructed relative to the chosen side. Entries with unavailable future paths remain ineligible for fitting.
There are 2,307 complete-feature entries from 16 October 2015. The primary is long on 79%; sign agreement is 55.7% overall, 60.4% for longs and 38.1% for shorts. These are realised sample rates, including signed vertical exits. A model can use the known side to distinguish states, but those rates alone do not establish future profit.
Walk-forward with an embargo
Refit each January using earlier entries whose outcomes are available before testing. Labels can depend on prices up to \(t+10\); early horizontal touches may end sooner. The code conservatively drops the final \(H=10\) eligible training rows before fitting. This horizon gap prevents training labels from crossing the boundary in this daily example. In a general event dataset, inspect actual label-end timestamps and purge interval overlaps; an additional post-test embargo serves a different purpose. Scale features using training means/SDs only. Overlapping ten-day labels still produce dependent evaluation observations.
AUC 0.545 over 1,500 days describes the sample ordering of realised binary outcomes; 71.3% of predictions exceed 0.5. It does not establish calibrated probabilities, independent-test uncertainty or profitable net execution. Features are known by close \(t\), while applying them to the immediately following close-to-close return assumes execution at that close; operational decisions after the close need a later executable price.
Probability, position size and comparison
Question: How does the second-stage probability affect exposure?
Result: A stated probability-to-size rule checked on later observations.
Builds on Primary signals and the meta-model.
From probability to bet size
The rule \(b_t=\max(0,2(p_t-\tfrac12))\) supplies a heuristic size, not an optimal bet derived from payoffs and costs. Signed exposure is \(\text{side}_t b_t\), marked against the next daily log return. The training label concerns an event lasting up to ten days; the evaluation resets daily and does not implement those stop/profit exits. Calibration and target–payoff alignment therefore need checking.
All four series share 1,500 days and omit trading, borrow and financing costs. stats3 reports sum of weighted log returns, annualised mean/SD and drawdown of cumulative log-P&L. The printed “cum %” and “maxDD” are those diagnostics, not compounded net-NAV return and fractional wealth drawdown. Sized exposure averages 0.16. Year-by-year gross Sharpe rises in 2020 (−0.68 → 0.79) but falls in 2022 (−0.22 → −1.11); full-period improvement does not guarantee improvement in each state.
A boosting meta-model
HistGradientBoostingClassifier is the tree classifier from §4.3. The cell keeps features, training scaler and annual refit dates fixed, then compares AUC and the same gross daily-P&L diagnostics.
Logistic and boosting differ by 0.63 in observed gross Sharpe. This is sensitivity to the learner; it is not a sampling standard error. Selecting the better learner after inspecting these results increases the research search that §7.6 must record.
Fixed-horizon: +1. If the stop lies above −6% and no earlier barrier was touched, the observed day-3 lower crossing gives a triple-barrier label of −1. The label records the specified observed path event; a real stop’s fill and costs may differ. In the original S&P fixture, the labels disagree on 11.5% of eligible entries.
Late training labels may use prices inside the test period. The code drops the last \(H=10\) eligible training rows as a conservative outcome-maturity gap. Early touches can finish sooner; general event validation should inspect actual endpoint timestamps and overlap. Features being known at the earlier close does not make the later outcome available.
Strategy Selection and Multiple Testing
| Part | Question |
|---|---|
| 7.6.1 The search space and winner’s bias | How many alternatives were tried before choosing a winner? |
| 7.6.2 Uncertainty and selection-adjusted statistics | Does the apparent winner survive different evidence checks? |
| 7.6.3 External comparison and research revision | Does the conclusion survive another stated comparison? |
Model, feature, target and threshold choices form a research search. Reporting only its best performance creates winner’s bias. This section retains all 42 technical-rule configurations, then compares conditional bootstrap uncertainty, a Deflated Sharpe benchmark, CSCV rank stability and a BH family calculation. The Dow provides a further stated comparison; these gross index-price demonstrations are not evidence of a deployable net strategy.
The search space and winner’s bias
Question: How many alternatives were tried before choosing a winner?
Result: A complete rule grid and an explanation of selection bias.
The grid: 42 rules a chart-reader might try
rules_pnl shifts the position one observation before multiplying by the daily log return. Both directions count as research configurations; mirrors are dependent alternatives, not independently repeated evidence. All 42 series share 2,266 days from 30 December 2015.
The winner’s annualised Sharpe is 0.676, with 9.1% active days, skew 7.1 and raw kurtosis 154. Across the grid the median is 0.15, SD 0.26 and 71% are positive; buy-and-hold is 0.64 on the same dates. These are gross weighted-log-return diagnostics. Searching guarantees a sample maximum, but does not establish a positive population edge.
Is the best of \(N\) evidence? The expected maximum
Suppose all true trial Sharpes are zero and their estimates are independent Normal draws with common SD \(\sigma_{SR}\). The expected maximum rises with \(N\). The approximation of Bailey and López de Prado (2014) is:
\[\mathbb E[\max_N SR] \approx \sigma_{SR}\Big[(1-\gamma)\,\Phi^{-1}\!\big(1 - \tfrac1N\big) + \gamma\,\Phi^{-1}\!\big(1 - \tfrac{1}{Ne}\big)\Big], \qquad \gamma = 0.5772,\]
\(\Phi^{-1}\) is the Normal quantile function and \(\gamma\) the Euler–Mascheroni constant. With the original annualised SD 0.26 and \(N=42\), the approximation is 0.57. The interactive counterfactual holds the spread fixed while changing the assumed independent count; the actual grid has mirrors and nearby settings, so its effective trial count has not been estimated. Its selected Sharpe must be assessed with that uncertainty, sample length and return moments.
The debug that produces a Sharpe of 5
The debug cell deliberately omits the lag from its 33 SMA/Bollinger rules; its code contains none of the grid’s nine RSI rules. Predict how crediting a close-derived position with the return that produced the same close changes the result.
The buggy 33-rule subset reports BB20 k=1.5 at 5.34 and BBmr20 k=2.0 at −3.12, against the correctly lagged 0.68. Without .shift(1), the position computed from \(P_t\) receives \(r_t=\log(P_t/P_{t-1})\), which was already observed. Add the lag to restore the stated decision clock. This repairs alignment under idealised closing execution; feasible fills and costs still require separate evidence.
Uncertainty and selection-adjusted statistics
Question: Does the apparent winner survive different evidence checks?
Result: Distinguish bootstrap uncertainty, deflation, overfitting probability and FDR.
Builds on The search space and winner’s bias.
Trial 1 — a confidence interval from the stationary bootstrap
Daily dependence makes iid resampling unsuitable unless that assumption is justified. The stationary bootstrap copies runs from random starts, with geometric lengths and wrap-around. Here mean length is 20 and seed 0; 300 complete pseudo-series produce the Sharpe interval. Dependence is retained locally within runs, not perfectly across every horizon. Valid inference requires suitable stationarity, dependence and block-length conditions; the method does not force a smaller or larger interval than an iid approximation.
For this fixed winner, the percentile interval is [0.18,1.11], empirical SD 0.26, and 5/300 draws are nonpositive. The displayed iid analytic SE is 0.33. The bootstrap fraction is not a null-centred test p-value. The calculation keeps best fixed rather than selecting afresh within each pseudo-sample, so it does not correct the 42-rule selection.
Trial 2 — the Deflated Sharpe Ratio
First compute \(SR_0\), the expected maximum under the trial-estimate approximation. Then compare the selected daily Sharpe with that benchmark using its sample length, skewness and raw kurtosis. This is the Probabilistic Sharpe calculation with a search-dependent threshold:
\[DSR = \Phi\!\left[\frac{(\widehat{SR} - SR_0)\sqrt{T-1}}{\sqrt{1 - \gamma_3 \widehat{SR} + \frac{\gamma_4 - 1}{4}\widehat{SR}^2}}\right],\]
Use daily \(\widehat{SR}\), \(T\) daily observations, skewness \(\gamma_3\) and raw kurtosis \(\gamma_4\). For Normal returns the denominator is \(\sqrt{1+\widehat{SR}^2/2}\); its general form reflects sample moments. It is the scale before division by \(\sqrt{T-1}\), not the full SE by itself. The formula uses an asymptotic Normal approximation and does not correct serial correlation here. The effective independent trial count is unknown for this dependent mirror grid; using raw 42 is a stated sensitivity calculation, not a guaranteed conservative bound. \(DSR=\Phi(z)\) near 0.5 means the observed estimate is near its benchmark; it is not a posterior probability of skill. Original method.
PSR is 0.990 against zero; DSR is 0.637 against the reported-grid benchmark 0.57 annualised. The difference comes from the changed benchmark under the stated approximation. It does not assign 36% posterior probability to a worthless grid or establish its true return after costs. Trial dependence, serial correlation and noisy high moments remain relevant limitations.
Trial 3 — the probability of backtest overfitting (CSCV)
CSCV evaluates selection rank stability on the full \(T\times N\) P&L matrix. Split consecutive rows into \(S\) blocks. For each chosen half, use that half to select the best-Sharpe configuration; its complementary half supplies the out-of-sample rank. The logit maps a relative rank to a signed statistic:
- Pick the rule with the best in-sample Sharpe, \(j^\ast\).
- Find its rank among the \(N\) out-of-sample Sharpe ratios, as a fraction \(\bar\omega \in (0, 1)\), and take the logit \(\lambda = \log\frac{\bar\omega}{1 - \bar\omega}\) — positive when the in-sample winner is above the out-of-sample median, negative when it is below.
- The probability of backtest overfitting is \(PBO = P(\lambda \le 0)\) across the combinations.
\(S=8\) gives \(\binom84=70\) possible subsets. The code drops two trailing days, divides 2,264 retained rows equally and samples 45 subsets without replacement (seed 0). Not every sampled subset has its complementary selection subset present, although all 70 would be symmetric as a complete family. In-sample blocks can follow out-of-sample blocks chronologically: this is a symmetric selection diagnostic, not a walk-forward experiment or forecast validation. Dependence and period choice affect its interpretation. Original method.
The sampled result is 24/45 nonpositive logits (0.533), median logit −0.33 and 2/45 OOS-best configurations. Rank is \(1,\ldots,N\) in ascending Sharpe order, so \(\omega=\text{rank}/(N+1)\) avoids infinite logits. Ties need an explicit convention; this code’s nested argsort assigns ordinal positions. These results describe instability of this selection procedure in these samples, not proof that every configuration has zero true edge.
Trial 4 — Benjamini–Hochberg across the grid
For a stated family of \(N\) hypotheses, BH sorts \(p_{(1)}\le\cdots\le p_{(N)}\), chooses the largest \(k\) satisfying \(p_{(k)}\le qk/N\), then rejects all first \(k\). The target is \(E[V/\max(R,1)]\), the expected false fraction among rejections. Nominal control requires valid null p-values and independence or suitable positive regression dependence (PRDS); “positive dependence” alone is too broad. The displayed p-values use a Normal approximation to \(SR\sqrt T\), omitting serial correlation and the sample’s heavy-tail uncertainty. BH cannot repair invalid p-values. SciPy method and assumptions.
No rank crosses the BH line in the original family, so zero hypotheses are rejected at 5%. The mirrored pair has the same two-sided p-value 0.0427. A first-rank threshold is \(q/N\), but the first p-value need not cross it if a later rank qualifies; the labelled four-test counterexample shows this. Dependent index-return tests and their naive p-values do not establish exact 5% FDR control. BH, DSR and CSCV answer distinct questions; disagreement would require investigation rather than a single combined pass/fail rule.
External comparison and research revision
Question: Does the conclusion survive another stated comparison?
Result: An updated search record and a decision that includes null evidence.
Builds on Uncertainty and selection-adjusted statistics.
Does the Dow agree?
Run the same 42 configurations on the 8,774-day Dow file. Distinguish evaluating the frozen S&P winner from selecting a new Dow winner. A second comparison freezes the Dow winner through 2002, then evaluates it from 2003 onward. These index samples overlap in dates and constituents, so the external comparison is not an independent replication.
The frozen S&P winner is nearly zero on Dow and ranks 32/42. Dow’s reselected winner has Sharpe 0.23, DSR 0.583, PBO 0.778 and zero BH rejections under the displayed approximations. The rule selected through 2002 earns 0.25 before and 0.02 afterwards. This grid does not provide convincing transferable evidence under these diagnostics; that finding concerns these rules, data and assumptions, not all technical methods or a claim about the broader literature.
Add a rule and re-deflate
One more trial: 63-day time-series momentum, long or short on the sign of the last 63 days’ return. The cell adds it as the 43rd column and recomputes the best rule and its DSR.
Adding the stated 63-day momentum configuration gives 43 trials. Its Sharpe is 0.18; the winner remains unchanged while DSR becomes 0.642. Count and spread both enter \(SR_0\), so adding a near-median trial can slightly raise DSR by reducing estimated spread; deflation need not move monotonically with raw grid size. Keep every attempted configuration in the search record.
Wiecki, Campbell, Lent and Stauth studied 888 filtered Quantopian algorithms, backtested over 2010–2015 with at least six months after code creation reserved as out-of-sample simulation. Backtest Sharpe had little predictive value (\(R^2 < 0.025\)); risk and portfolio-construction features added information. More backtesting was associated with a larger backtest–out-of-sample discrepancy. This was neither a test of live fills nor evidence that backtesting was the only informative variable. Original study.
Lesson: record the search history and assess genuinely held-out evidence; one selected Sharpe does not describe the full search.
The bootstrap fixes the selected rule and does not repeat the search. DSR compares the selected 0.68 with the 0.57 expected-maximum benchmark under a reported 42-trial approximation, giving 0.637. It is not a posterior probability that the grid is worthless. PBO 24/45, zero BH rejections and Dow rank 32/42 are separate diagnostics with distinct assumptions; together they motivate caution, not a universal impossibility conclusion.
Sort the \(N\) p-values; choose the largest \(k\) with \(p_{(k)}\le qk/N\) and reject all first \(k\). Here no rank crosses, so neither mirrored 0.0427 is rejected. The first-rank threshold is 0.0012, but BH can reject a first p-value that fails its own threshold when a later rank qualifies. Nominal FDR control also requires valid p-values and independence/PRDS; mirrors remain reported configurations but are dependent tests.
A Revisable Course Reference for Coding
| Part | Question |
|---|---|
| 7.7.1 Evidence, provenance and correction | Which source supports the proposed coding pattern? |
| 7.7.2 Retrieve, inspect and detect stale evidence | Is the retrieved example current and eligible? |
| 7.7.3 Adaptation, checks and evaluation | Does the adapted code meet this task’s requirements? |
How can an assistant retrieve a relevant explanation for a new task and stop retrieving an old passage after its source changes?
This reference lab connects the statistical methods to reusable, cited code. It adds no new assignment or grading component.
Evidence, provenance and correction
Question: Which source supports the proposed coding pattern?
Result: Define retrieval, provenance, eligibility and the correction loop.
The case: a plausible answer repeats a teaching bug
A colleague asks for a function that collects a value. Two calls with no list supplied should be independent. A search finds a slide with items=[] in the function definition.
That slide introduces the mutable-default trap. Copying its first code example would reproduce the problem it was written to teach.
The highest similarity is not enough. Read the passage’s status and explanation, retrieve the corrected pattern, then test independent calls and the behaviour of a supplied list. Increasing the number of passages does not turn relevant buggy code into a solution.
RAG: retrieve evidence before generating an answer
Retrieval-augmented generation (RAG) supplies searched passages to the assistant before it adapts code. The collection stays outside the model’s trained parameters.
A reference system must support a correction loop
The index is derived from the source. Correct the original chapter or notebook, review it, rebuild, then retrieve again. Retain older snapshots for audit and refuse retrieval when the active index is stale.
What belongs in the coding reference?
| Material | How to use it |
|---|---|
| Current canonical Slides and Book | Default reference, with headings and code explanations |
| Current project notebooks | Retain cell locations and assumptions; ignore saved outputs as proof |
| Intentional bugs and exercise starters | Label as teaching cautions; exclude from solution retrieval by default |
| Older material and narration | Historical context, explicitly requested |
| Instructor answers | Private index and explicit instructor access only |
| Data, credentials, student submissions, generated duplicates | Outside the coding-reference corpus |
A PDF’s extracted text does not capture all diagrams or scanned pages. Check the original page when the visual matters. “All course material” means an organised inventory with these boundaries, not a claim that every file is a verified answer.
A chunk is an explanation with its provenance
Each unit carries its file and location, source hash and index revision, status and audience, and validation (NOT_TASK_TESTED until checked for this task).
Retrieve, inspect and detect stale evidence
Question: Is the retrieved example current and eligible?
Result: Inspect cited chunks and reject an outdated index.
Builds on Evidence, provenance and correction.
Load a small, public reference collection
The browser lab uses six curated teaching cards, not the instructor’s private index. Two describe a retired or unfinished default-list example; four are current references.
read_json loads one record per card into a DataFrame. status tells us whether a card is eligible; revision identifies its version. The source column points back to the course explanation.
First retrieve by words; then inspect the evidence
TF–IDF gives a word more weight when it is frequent in a passage but uncommon across the collection. Cosine similarity compares the directions of the resulting vectors. Here, larger means more shared weighted vocabulary; it is not a probability of correctness.
fit_transform learns the vocabulary and encodes cards; transform uses the fitted vocabulary for the query. ravel flattens the one-row score matrix. Sorting alone retains ineligible cards.
Filter eligibility before offering a reference
query selects current material; head(3) limits the evidence; iloc[0] selects its first row.
The full local tool combines semantic embeddings with keyword search. Embeddings help match different wording; keyword search helps preserve API names such as merge and validate. It filters the candidate set before ranking and returns source-linked evidence. The browser baseline filters an already-computed full ranking, which has the same eligible ordering here.
Interactive illustration: correct, rebuild, retrieve
Predict first: select “Source corrected; not rebuilt”. Why must retrieval stop? Compare successful and failed rebuilds; identify the returned revision and the task checks still needed.
Detect an edit before using an old index
The result is False. This cell demonstrates change detection, not a full rebuild. The local tool hashes actual source files, checks new and deleted files, rebuilds passages and vectors into a new revision, then switches the active pointer. A failed rebuild keeps the previous snapshot and refuses stale retrieval.
Adaptation, checks and evaluation
Question: Does the adapted code meet this task’s requirements?
Result: Task-specific checks and a measured evaluation of the reference workflow.
Builds on Retrieve, inspect and detect stale evidence.
Adapt the retrieved pattern and test the requirement
None marks an omitted list. If the requirement says never mutate an input, adapt the supplied-list branch and add that test.
Your turn: prevent an unfinished example entering the answer
The slide starter keeps all six cards and fails its eligibility check. The completed version keeps only the four current references:
After your check passes, change the task to “explain the mutable-default bug”. Would you retrieve a teaching caution for explanation? Explain why that still would not make its buggy code a solution.
Use the full local tool on another project
The course implementation indexes canonical chapters, project notebooks and separately labelled older/private sources. It runs locally with a cached embedding model and SQLite; it does not need an API credential. A connected cloud assistant may still send selected retrieved context to its provider.
python KnowledgeBase/course_rag.py search "pandas merge validate duplicate customer_id values"
python KnowledgeBase/course_rag.py status
# After correcting and checking the original course material:
python KnowledgeBase/course_rag.py syncThese are local terminal commands, run from the instructor’s course repository using its configured Python. The public notebook below provides a portable, smaller implementation; students do not need the private repository or its instructor sources.
Download the notebook lab. It implements retrieval, a stale-index guard, correction and rebuilding on public cards. Browser cells need no local installation; the notebook uses pandas and scikit-learn.
What to ask a coding assistant
Retrieve current course guidance for this task and cite the file and location. State the data schema, prediction time and assumptions. Adapt the method, explain the changes, and test normal, missing and boundary cases. If the course evidence is insufficient or contradictory, say so and check current official documentation.
An MCP connection makes search_course available as a tool; it does not guarantee that every assistant turn invokes it. Ask for retrieval explicitly and inspect the citations. Existing chat answers do not update automatically when a course file changes.
How would we know it reduces coding mistakes?
Also test edits and deletions, status filtering and access controls. Bug examples and private answers must remain ineligible for ordinary solution retrieval.
A working index and a good retrieval score alone do not establish fewer coding errors. RAG can retrieve an incorrect source, omit a condition or encourage an unsuitable adaptation. CodeRAG-Bench studies these retrieval and generation difficulties for coding tasks.
What you discovered — revisable reference
- Search course explanations before adapting statistical code; keep the source and its qualifications together.
- Distinguish relevance, eligibility and correctness. A teaching bug can be highly relevant.
- Correct the original, review it, rebuild the index and retrieve again. Quarantine unresolved errors.
- Test the adaptation on the real task and consult current documentation when needed.
Further reading: Sentence Transformers semantic search, SQLite FTS5, and Codex MCP connections.
The companion slides apply the chapter’s evidence in a decision memo and group discussion.
Chapter Wrap-up
Match the output to the decision: ordering to shortlists; calibrated ranges to uncertainty; partial effects to an identified question; covariance and cluster structure to allocation; mature event outcomes to two-stage decisions; and search-aware statistics to selected results. A useful coding reference preserves each method’s sources, assumptions and task checks.
Keep the units and clocks attached: rank IC is measured by month; conformal coverage requires its stated calibration assumptions; cross-fitting does not remove omitted confounding or time dependence; target-weight change is not executed turnover; sign agreement is not a stopped-trade payoff; and bootstrap intervals do not account for selecting their input. Weak or null evidence remains part of the result.
Use the same data, seed and comparison protocol in both formats. Record every trial, preserve a held-out evaluation, and test each adaptation from a complete cell.